MEAP VERSION 6

ABOUT THIS MEAP

You can download the most up-to-date version of your electronic books from your Manning Account at account.manning.com. For customer support write to support@manning.com.

© Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.

https://livebook.manning.com/#!/book/pyspark-in-action/discussion

WELCOME

Thank you for purchasing the MEAP for PySpark in Action: Python data analysis at scale. It is a lot of fun (and work!) and I hope you’ll enjoy reading it as much as I am enjoying writing the book.

My journey with PySpark is pretty typical: the company I used to work for migrated their data infrastructure to a data lake and realized along the way that their usual warehouse-type jobs didn’t work so well anymore. I spent most of my first months there figuring out how to make PySpark work for my colleagues and myself, starting from zero. This book is very influenced by the questions I got from my colleagues and students (and sometimes myself). I’ve found that combining practical experience through real examples with a little bit of theory brings not only proficiency in using PySpark, but also how to build better data programs. This book walks the line between the two by explaining important theoretical concepts without being too laborious.

This book covers a wide range of subjects, since PySpark is itself a very versatile platform. I divided the book into three parts.

- Part 1: Walk teaches how PySpark works and how to get started and perform basic data manipulation.

- Part 2: Jog builds on the material contained in Part 1 and goes through more advanced subjects. It covers more exotic operations and data formats and explains more what goes on under the hood.

- Part 3: Run tackles the cooler stuff: building machine learning models at scale, squeezing more performance out of your cluster, and adding functionality to PySpark.

To have the best time possible with the book, you should be at least comfortable using Python. It isn’t enough to have learned another language and transfer your knowledge into Python. I cover more niche corners of the language when appropriate, but you’ll need to do some research on your own if you are new to Python.

Furthermore, this book covers how PySpark can interact with other data manipulation frameworks (such as Pandas), and those specific sections assume basic knowledge of Pandas.

Finally, for some subjects in Part 3, such as machine learning, having prior exposure will help you breeze through. It’s hard to strike a balance between “not enough explanation” and “too much explanation”; I do my best to make the right choices.

Your feedback is key in making this book its best version possible. I welcome your comments and thoughts in the liveBook discussion forum.

Thank you again for your interest and in purchasing the MEAP!

—Jonathan Rioux

BRIEF TABLE OF CONTENTS

Part 1: Walk

1   Introduction

2   Your first data program in PySpark

3   Submitting and scaling your first PySpark program

4   Analyzing tabular data with pyspark.sql

5   The data frame through a new lens: joining and grouping

Part 2: Jog

6   Multi-dimentional data frames: using PySpark with JSON data

7   Bilingual PySpark: blending Python and SQL code

8   Extending PySpark with Python: RDD and user-defined-functions

9   Faster big data processing: a primer

Part 3: Run

10   A foray into machine learning: logistic regression with PySpark

11   Simplifying your experiments with machine learning pipelines

12   Machine learning for unstructured data

13   PySpark for graphs: GraphFrames

14   Testing PySpark code

15   Going full circle: structuring end-to-end PySpark code

Appendixes:

A   Solutions to the exercices

B   Installing PySpark locally

C   Using PySpark with a cloud provider

D   Python essentials

E   PySpark data types

F   Efficiently using PySpark’s API documentation

1

INTRODUCTION

In this chapter, you will learn:

- What is PySpark

- Why PySpark is a useful tool for analytics

- The versatility of the Spark platform and its limitations

- PySpark’s way of processing data

According to pretty much every news outlet, data is everything, everywhere. It’s the new oil, the new electricity, the new gold, plutonium, even bacon! We call it powerful, intangible, precious, dangerous. I prefer calling it useful in capable hands. After all, for a computer, any piece of data is a collection of zeroes and ones, and it is our responsibility, as users, to make sense of how it translates to something useful.

Just like oil, electricity, gold, plutonium and bacon (especially bacon!), our appetite for data is growing. So much, in fact, that computers aren’t following. Data is growing in size and complexity, yet consumer hardware has been stalling a little. RAM is hovering for most laptops at around 8 to 16 GB, and SSD are getting prohibitively expensive past a few terabytes. Is the solution for the burgeoning data analyst to triple-mortgage his life to afford top of the line hardware to tackle Big Data problems?

Introducing Spark, and its companion PySpark, the unsung heroes of large-scale analytical workloads. They take a few pages of the supercomputer playbook — powerful, but manageable, compute units meshed in a network of machines — and bring it to the masses. Add on top a powerful set of data structures ready for any work you’re willing to throw at them, and you have a tool that will grow (pun intended) with you.

This book is great introduction to data manipulation and analysis using PySpark. It tries to cover just enough theory to get you comfortable, while giving enough opportunities to practice. Each chapter except this one contains a few exercices to anchor what you just learned. The exercises are all solved and explained in Appendix A.

1.1  WHAT IS PYSPARK?

What’s in a name? Actually, quite a lot. Just by separating PySpark in two, one can already deduce that this will be related to Spark and Python. And it would be right!

At the core, PySpark can be summarized as being the Python API to Spark. While this is an accurate definition, it doesn’t give much unless you know the meaning of Python and Spark. If we were in a video game, I certainly wouldn’t win any prize for being the most useful NPC. Let’s continue our quest to understand what is PySpark by first answering What is Spark?.

1.1.1  You saw it coming: What is Spark?

Spark, according to their authors, is a unified analytics engine for large-scale data processing. This is a very accurate definition, if a little dry.

Digging a little deeper, we can compare Spark to an analytics factory. The raw material — here, data — comes in, and data, insights, visualizations, models, you name it! comes out.

Just like a factory will often gain more capacity by increasing its footprint, Spark can process an increasingly vast amount of data by scaling out instead of scaling up. This means that, instead of buying thousand of dollars of RAM to accommodate your data set, you’ll rely instead of multiple computers, splitting the job between them. In a world where two modest computers are less costly than one large one, it means that scaling out is less expensive than up, keeping more money in your pockets.

The problem with computers is that they crash or behave unpredictably once in a while. If instead of one, you have a hundred, the chance that at least one of them go down is now much higher. [1] Spark goes therefore through a lot of hoops to manage, scale, and babysit those poor little sometimes unstable computers so you can focus on what you want, which is to work with data.

This is, in fact, one of the weird thing about Spark: it’s a good tool because of what you can do with it, but especially because of what you don’t have to do with it. Spark provides a powerful API [2] that makes it look like you’re working with a cohesive, non-distributed source of data, while working hard in the background to optimize your program to use all the power available. You therefore don’t have to be an expert at the arcane art of distributed computing: you just need to be familiar with the language you’ll use to build your program. This leads us to…

1.1.2  PySpark = Spark + Python

PySpark provides an entry point to Python in the computational model of Spark. Spark itself is coded in Scala, a language very powerful if a little hard to grasp. In order to meet users where they are, Spark also provides an API in Java, Python and R. The authors did a great job at providing a coherent interface between language while preserving the idiosyncrasies of the language where appropriate. Your PySpark program will therefore by quite easy to read by a Scala/Spark programmer, but also to a fellow Python programmer who hasn’t jumped into the deep end (yet).

Python is a dynamic, general purpose language, available on many platforms and for a variety of tasks. Its versatility and expressiveness makes it an especially good fit for PySpark. The language is one of the most popular for a variety of domains, and currently is a major force in data analysis and science. The syntax is easy to learn and read, and the amount of library available means that you’ll often find one (or more!) who’s just the right fit for your problem.

1.1.3  Why PySpark?

There are no shortage of libraries and framework to work with data. Why should one spend their time learning PySpark specifically?

PySpark packs a lot of advantages for modern data workloads. It sits at the intersection of fast, expressive and versatile. Let’s explore those three themes one by one.

PYSPARK IS FAST

If you search "Big Data" in a search engine, there is a very good chance that Hadoop will come within the first few results. There is a very good reason to this: Hadoop popularized the famous MapReduce framework that Google pioneered in 2004 and is now a staple in Data Lakes and Big Data Warehouses everywhere.

Spark was created a few years later, sitting on Hadoop’s incredible legacy. With an aggressive query optimizer, a judicious usage of RAM and some other improvements we’ll touch on in the next chapters, Spark can run up to 100x faster than plain Hadoop. Because of the integration between the two frameworks, you can easily switch your Hadoop workflow to Spark and gain the performance boost without changing your hardware.

PYSPARK IS EXPRESSIVE

Beyond the choice of the Python language, one of the most popular and easy-to-learn language, PySpark’s API has been designed from the ground up to be easy to understand. Most programs read as a descriptive list of the transformations you need to apply to the data, which makes them easy to reason about. For those familiar with functional programming languages, PySpark code is conceptually closer to the "pipe" abstraction rather than pandas, the most popular in-memory DataFrame library.

You will obviously see many examples through this book. As I was writing those examples, I was pleased about how close to my initial (pen and paper) reasoning the code ended up looking. After understanding the fundamentals of the framework, I’m confident you’ll be in the same situation.

PYSPARK IS VERSATILE

There are two components to this versatility. First, there is the availability of the framework. Second, there is the diversified ecosystem surrounding Spark.

PySpark is everywhere. All three major cloud providers have a managed Hadoop/Spark cluster as part of their offering, which means you have a fully provisioned cluster at a click of a few buttons. You can also easily install Spark on your own computer to nail down your program before scaling on a more powerful cluster. Appendix B covers how to get your own local Spark running, while Appendix C will walk through the current main cloud offerings.

PySpark is open-source. Unlike some other analytical software, you aren’t tied to a single company. You can inspect the source code if you’re curious, and even contribute if you have an idea for a new functionality or find a bug. It also gives a low barrier to adoption: download, learn, profit!

Finally, Spark’s eco-system doesn’t stop at PySpark. There is also an API for Scala, Java, R, as well as a state-of-the-art SQL layer. This makes it easy to write a polyglot program in Spark. A Java software engineer can tackle the ETL pipeline in Spark using Java, while a data scientist can build a model using PySpark.

WHERE PYSPARK FELL SHORT

It would be awesome if PySpark was The Answer to every data problem. Unfortunately, there are some caveats. None of them are a deal-breakers, but they are to be considered when you’re selecting a framework for your next project.

PySpark isn’t the right choice if you’re dealing with small data sets. Managing a distributed cluster comes with some overhead, and if you’re just using a single node, you’re paying the price but aren’t using the benefits. As an example, a PySpark shell will take a few seconds to launch: this is often more than enough time to process data that fits within your RAM.

PySpark also has a disadvantage when it comes to the Java and Scala API. Since Spark is at the core a Scala program, Python code have to be translated to and from JVM [3] instructions. While more recent versions have been bridging that gap pretty well, pure Python translation, which happens mainly when you’re defining your own User Defined Functions (UDF), will perform slower. We will cover UDF and some ways to mitigate the performance problem in Chapter 8.

Finally, while programming PySpark can feel easy and straightforward, managing a cluster can be a little arcane. Spark is a pretty complicated piece of software, and while the code base matured remarkably over the past few years, the days where scaling a 100-machine cluster and manage it as easily as a single node are far ahead. We will cover some of the developer-facing configuration and problems in the Chapter about performance, but for hairier problems, do what I do: befriend your dev ops.

1.1.4  Your very own factory: how PySpark works

In this section, I will explain how Spark processes a program. It can be a little odd to present the workings and underpinnings of a system that we claimed, a few paragraphs ago, hides that complexity. We still think that having a working knowledge of how Spark is set up, how it manages data and how it optimizes queries is very important. With this, you will be able to reason with the system, improve your code and figure out quicker when it doesn’t perform the way you want.

If we’re keeping the factory analogy, we can imagine that the cluster of computer where Spark is sitting on is the building.

Figure 1.1. A totally relatable data factory, outside and in.

If we look at Figure 1.1, we can see two different way to interpret a data factory. On the left, we see how it looks like from the outside: a cohesive unit where projects come in and results comes out. This is what it will appear to you most of the time. Under the hood, it looks more like on the right: you have some workbenches where some workers are assigned to. The workbenches are like the computers in our Spark cluster: there is a fixed amount of them, and adding or removing some is easy but needs to be planned in advance. The workers are called executors in Spark’s literature: they are the one performing the actual work on the machines.

One of the little workers looks spiffier than the other. That top hat definitely makes him stand out of the crowd. In our data factory, he’s the manager of the work floor. In Spark terms, we call this the master. In the spirit of the open work-space, he shares one of the workbenches with his fellow employees. The role of the master is crucial to the efficient execution of your program, so 1.1.6 is dedicated to this.

1.1.5  Some physical planning with the cluster manager

Upon reception of the task, which is called a driver program in the Spark world, the factory starts running. This doesn’t mean that we get straight to processing! Before that, the cluster need to plan the capacity it will allocate for your program. The entity, or program, taking care of this is aptly called the cluster manager. In our factory, this cluster manager will look at the workbenches with available space and secure as many as necessary, then start hiring workers to fill the capacity. In Spark, it will look at the machines with available computing resources and secure what’s necessary, before launching the required number of executors across them.

Note

Spark provides its own cluster manager, but can also play well with other ones when working in conjunction with Hadoop or another Big Data platform. We will definitely discuss the intricacies of managing the cluster manager (pun intended) in the chapter about performance, but in the meantime, if you read about YARN or Mesos in the wild, know that they are two of the most popular nowadays.

Any directions about capacity (machines and executors) are encoded in a SparkContext object which represents the connection to our Spark cluster. Since our instructions didn’t mention any specific capacity, the cluster manager will allocate the default capacity prescribed by our Spark installation.

We’re off to a great start! We have a task to accomplish, and the capacity to accomplish it. What’s next? Let’s get working!

1.1.6  A factory made efficient through a lazy manager

Just like in a large-scale factory, you don’t go to each employee and give them a list of tasks. No, here, you’ll provide your list of steps to the manager and let them deal with it. In Spark, the manager/master takes your instructions (carefully written in Python code), translate them in Spark steps and then process them across the worker. The master will also manage which worker (more on them in a bits) has which slice of the data, and make sure that you don’t lose some bits in the process.

Your manager/master has all the qualities a good manager has: smart, cautious and lazy. Wait, what? You read me right. Laziness in a programming context — and one could argue in the real world too — can actually be very good thing. Every instruction you’re providing in Spark can be classified in two categories: transformations and actions. Actions are what many programming languages would consider IO. Actions includes, but are not limited to:

- Printing information on the screen

- Writing data to a hard drive or cloud bucket

In Spark, we’ll see those instructions most often via the show and write methods, as well as other calling those two in their body.

Transformations are pretty much everything else. Some examples of transformation are:

- Adding a column to a table

- Performing an aggregation according to certain keys

- Computing summary statistics on a data set

- Training a Machine Learning model on some data

Why the distinction, you might ask? When thinking about computation over data, you, as the developer, are only concerned about the computation leading to an action. You’ll always interact with the results of an action, because this is something you can see. Spark, with his lazy computation model, will take this to the extreme and will avoid performing data work until an action triggers the computation chain. Before that, the master will store (or cache) your instructions. This way of dealing with computation has many benefits when dealing with large scale data.

First, storing instructions in memory takes much less space than storing intermediate data results. If you are performing many operations on a data set and are materializing the data each step of the way, you’ll blow your storage much faster although you don’t need the intermediate results. We can all argue that less waste is better.

Second, by having the full list of tasks to be performed available, the master can optimize the work between executors much more efficiently. It can use information available at run-time, such as the node where specific parts of the data are located. It can also re-order and eliminate useless transformations if necessary.

Finally, during interactive development, you don’t have to submit a huge block of commands and wait for the computation to happen. Instead, you can iteratively build your chain of transformation, one at the time, and when you’re ready to launch the computation (like during your coffee break), you can add an action and let Spark work its magic.

Lazy computation is a fundamental aspect of Spark’s operating model and part of the reason it’s so fast. Most programming languages, including Python, R and Java, are eagerly evaluated. This means that they process instructions as soon as they receive them. If you have never worked with a lazy language before, it can look a little foreign and intimidating. If this is the case, don’t worry: we’ll weave practical explanations and implications of that laziness during the code examples when relevant. You’ll be a lazy pro in no time!

Note

Reading data, although clearly being I/O, is considered a transformation by Spark. This is due to the fact that reading data doesn’t perform any visible work to the user. You therefore won’t read data until you need to display or write it somewhere.

What’s a manager without competent employees? Once the task, with its action, has been received, the master starts allocating data to what Spark calls executors. Executors are processes that run computations and store data for the application. Those executors sit on what’s called a worker node, which is the actual computer. In our factory analogy, an executor would be an employee performing the work, while the worker node would be a workbench where many employees/executors can work. If we recall Figure 1.1, our master wears a top hat and sits with his employees/workers at one of the workbenches.

That concludes our factory tour. Let’s summarize our typical PySpark program.

We first encode our instructions in Python code, forming a driver program.

When submitting our program (or launching a PySpark shell), the cluster manager allocates resources for us to use. Those will stay constant for the duration of the program.

The master ingests your code and translate it into Spark instructions. Those instructions are either transformations or actions.

Once the master reaches an action, it optimizes the whole computation chain and splits the work between executors. Executors are processes performing the actual data work and they reside on machines labelled worked nodes.

That’s it! As we can see, the overall process is quite simple, but it’s obvious that Spark hides a lot of the complexity arising from efficient distributed processing. For a developer, this means shorter and clearer code, and a faster development cycle.

[1] It can be a fun probability exercise to compute by how much, but I will try to keep the math stuff at a minimum.

[2] Application Programming Interface, which is basically the set of functions, classes and variables provided for you to interact with

[3] Java Virtual Machine, which is like an emulator running on your computer. Both Java and Scala targets the JVM.

1.2  WHAT WILL YOU LEARN IN THIS BOOK?

This book will use PySpark to solve a variety of tasks a data analyst, engineer or scientist will encounter during his day to day life. We will therefore

- read and write data from (and to) a variety of sources and formats;

- deal with messy data with PySpark’s data manipulation functionality;

- discover new data sets and perform exploratory data analysis;

- build data pipelines that transform, summarize and get insights from data in an automated fashion;

- test, profile and improve your code;

- troubleshoot common PySpark errors, how to recover from them and avoid them in the first place.

After covering those fundamentals, we’ll also tackle different tasks that aren’t as frequent, but are interesting and an excellent way to showcase the power and versatility of PySpark.

- We’ll perform Network Analysis using PySpark’s own graph representation

- We’ll build Machine Learning models, from simple throwaway experiments to Deep Learning goodness

- We’ll extend PySpark’s functionality using user defined functions, and learn how to work with other languages

We are trying to cater to many potential readers, but are focusing on people with little to no exposure to Spark and/or PySpark. More seasoned practitioners might find useful analogies for when they need to explain difficult concepts and maybe learn a thing or two!

The book focuses on Spark version 2.4, which is currently the most recent available. Users on older Spark versions will be able to go through most of the code in the book, but we definitely recommend using at least Spark 2.0+.

We’re assuming some basic Python knowledge: some useful concepts are outlined in Appendix D. If you feel for a more in-depth introduction to Python, I recommend The Quick Python Book, by Naomi Ceder (Manning, 2018).

1.3  WHAT DO I NEED TO GET STARTED?

In order to get started, the only thing absolutely necessary is a working installation of Spark. It can be either on your computer (Appendix B) or using a cloud provider (Appendix C). Most examples in the book are doable using a local installation of Spark, but some will require more horsepower and will be identified as such.

A code editor will also be very useful for writing, reading and editing scripts as you go through the examples and craft your own programs. A Python-aware editor, such as PyCharm, is a nice-to-have but is in no way necessary. Just make sure it saves your code without any formatting: don’t use Microsoft Word to write your programs!

The book’s code examples are available on GitHub, so Git will be a useful piece of software to have. If you don’t know git, or don’t have it handy, GitHub provides a way to download all the book’s code in a Zip file. Make sure you check regularly for updates!

Finally, I recommend that you have an analog way of drafting your code and schema. I am a compulsive note-taker and doodler, and even if my drawing are very basic and crude, I find that working through a new piece of software via drawings helps in clarifying my thoughts. This means less code re-writing, and a happier programmer! Nothing spiffy, some scrap paper and a pencil will do wonders.

1.4  SUMMARY

- PySpark is the Python API for Spark, a distributed framework for large-scale data analysis. It provides the expressiveness and dynamism of the Python programming language to Spark.

- PySpark provides a full-stack analytics workbench. It has an API for data manipulation, graph analysis, streaming data as well as machine learning.

- Spark is fast: it owes its speed to a judicious usage of the RAM available and an aggressive and lazy query optimizer.

- Spark provides bindings for Python, Scala, Java, and R. You can also use SQL for data manipulation.

- Spark uses a master which processes the instructions and orchestrates the work. The executors receive the instructions from the master and perform the work.

- All instructions in PySpark are either transformations or actions. Spark being lazy, only actions will trigger the computation of a chain of instructions.

2

YOUR FIRST DATA PROGRAM IN PYSPARK

This chapter covers:

- Launching and using the pyspark shell for interactive development

- Reading and ingesting data into a data frame

- Exploring data using the DataFrame structure

- Selecting columns using the select() method

- Filtering columns using the where() method

- Applying simple functions to your columns to modify the data they contain

- Reshaping singly-nested data into distinct records using explode()

Data-driven applications, no matter how complex, all boils down to what I like to call three meta-steps, which are easy to distinguish in a program.

1. We start by ingesting or reading the data we wish to work with.

2. We transform the data, either via a few simple instructions or a very complex machine learning model

3. We then export the resulting data, either into a file to be fed into an app or by summarizing our findings into a visualization.

The next two chapters will introduce a basic workflow with PySpark via the creation of a simple ETL (Extract, Transform and Load, which is a more business-speak way of saying Ingest, Transform and Export). We will spend most of our time at the pyspark shell, interactively building our program one step at a time. Just like normal Python development, using the shell or REPL (I’ll use the terms interchangeably) provides rapid feedback and quick progression. Once we are comfortable with the results, we will wrap our program so we can submit it in batch mode.

Data manipulation is the most basic and important aspect of any data-driven program and PySpark puts a lot of focus on this. It serves as the foundation of any reporting, any machine learning or data science exercise we wish to perform. This section will give you the tools to not only use PySpark to manipulate data at scale, but also how to think in terms of data transformation. We obviously can’t cover every function provided in PySpark, but I’ll provide a good explanation of the ones we use. I’ll also introduce how to use the shell as a friendly reminder for those cases when you forget how something works.

Since this is your first end-to-end program in PySpark, we’ll get our feet wet with a simple problem to solve: counting the most popular word being used in the English language. Now, since collecting all the material ever produced in the English language would be a massive undertaking, we’ll start with a very small sample: Pride and Prejudice by Jane Austen. We’ll make our program work with this small sample and then scale it to ingest a larger corpus of text.

Since this is our first program, and I need to introduce many new concepts, this Chapter will focus on the data manipulation part of the program. Chapter 3 will cover the final computation as well as wrapping our program and then scaling it.

Tip

The book repository, containing the code and data used for the examples and exercises, is available at  github.com/jonesberg/PySparkInAction.

2.1  SETTING UP THE PYSPARK SHELL

PySpark provides a REPL (Read, eval, print loop) for interactive development. Python and other programming language families, such as Lisp, also provides one, so there is a good chance that you already worked with one in the past. It speeds up your development process by giving instantaneous feedback the moment you submit an instruction, instead of forcing you to compile your program and submit it as one big monolithic block. I’ll even say that using a REPL is even more useful in PySpark, since every operation can take a fair amount of time. Having a program crash mid-way is always frustrating, but it’s even worse when you’ve been running a data intensive job for a few hours.

For this chapter (and the rest of the book), I assume that you have access to a working installation of Spark, either locally or in the cloud. If you want to perform the installation yourself, Appendix B contains step-by-step instructions for Linux, OsX and Windows. If you can’t install it on your computer, or prefer not to, Appendix C provides a few cloud-powered options as well as additional instructions to upload your data and make it visible to Spark.

Once everything is set up, you can launch the pyspark shell by inputting pyspark into your terminal. You should see an ASCII-art version of the Spark logo, as well as some useful information. Listing 2.1 shows what happens on my local machine.

Listing 2.1. Launching pyspark on a local machine

$ pyspark

Python 3.7.3 (default, Mar 27 2019, 16:54:48)

Type 'copyright', 'credits' or 'license' for more information

IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.

19/09/07 12:16:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ❶

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ❷

Welcome to

____              __

/ __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  '_/

/__ / .__/\_,_/_/ /_/\_\   version 2.4.3 ❸

/_/

Using Python version 3.7.3 (default, Mar 27 2019 16:54:48) ❹

SparkSession available as 'spark'. ❺

In [1]: ❻

❶

When using PySpark locally, you most often won’t have a full Hadoop cluster pre-configured. For learning purposes, this is perfectly fine.

❷

Spark is indicating the level of details it’ll provide to you. We will see how to configure this in  2.1.2.

❸

We are using Spark version 2.4.3

❹

PySpark is using the Python available on your path.

❺

The  pyspark shell provides an entry point for you through the variable spark. More on this in 2.1.1.

❻

The REPL is now ready for your input!

Note

I highly recommend you using IPython when using PySpark in interactive mode. IPython is a better front-end to the Python shell containing many useful functionalities, such as friendlier copy-and-paste and syntax highlighting. The installation instructions in Appendix B includes configuring PySpark to use the IPython shell.

While all the information provided in Listing 2.1 is useful, two elements are worth expanding on: the SparkSession entry point and the log level.

2.1.1  The SparkSession entry-point

In 2.1 we saw that, upon launching the PySpark shell creates a spark variable that refers to SparkSession entry point. I will discuss about this entry point in this section as it provides the functionality for us to read data into PySpark [4].

In Chapter 1, we spoke briefly about the Spark entry point called SparkContext. SparkSession is a super-set of that. It wraps the SparkContext and provides functionality for interacting with data in a distributed fashion.Just to prove our point, see how easy it is to get to the SparkContext from our SparkSession object: just call the sparkContext attribute from spark.

$ spark.sparkContext

# <SparkContext master=local[*] appName=PySparkShell>

The SparkSession object is a recent addition to the PySpark API, making its way in version 2.0. This is due to the API evolving in a way that makes more room for the faster, more versatile data frame as the main data structure over the lower level RDD. Before that time, you had to use another object (called the SQLContext) in order to use the data frame. It’s much easier to have everything under a single umbrella.

This book will focus mostly on the data frame as our main data structure. I’ll discuss about the RDD in Chapter 8, when we discuss about lower-level PySpark programming and how to embed our own Python functions in our programs.

Reading older PySpark code

While this book shows modern PySpark programming, we are not living in a vacuum. If you go on the web, you might face older PySpark code that uses the former SparkContext/ sqlContext combo. You’ll also see the sc variable mapped to the SparkContext entry-point. With that we know about SparkSession and SparkContext, we can reason about old PySpark code by using the following variable assignments.

sc = spark.sparkContext

sqlContext = spark

You’ll see traces of SQLContext in the API documentation for backwards compatibility. I recommend avoiding using this as the new SparkSession approach is cleaner, simpler and more future-proof.

2.1.2  Configuring how chatty spark is: the log level

Monitoring your PySpark jobs is an important part of developing a robust program. PySpark provides many levels of logging, from nothing at all to a full description of everything happening on the cluster. By default, the pyspark shell defaults on WARN, that can be a little chatty when we’re learning. Fortunately, we can change the settings for your session by using the code in Listing 2.2.

Listing 2.2. Deciding on how chatty you want PySpark to be.

spark.sparkContext.setLogLevel(KEYWORD)

Table 2.1 lists the available keywords you can pass to setLogLevel. Each subsequent keyword contains all the previous ones, with the obvious exception of OFF that doesn’t show anything.

Table 2.1. log level keywords

Keyword

Signification

OFF

No logging at all (not recommended).

FATAL

Only fatal errors

ERROR

My personal favorite, will show  FATAL as well as other useful (but recoverable) errors.

WARN

Add warnings (and there is quite a lot of them).

INFO

Will give you runtime information

DEBUG

Will provide debug information on your jobs.

TRACE

Will trace your jobs (more verbose debug logs). Can be quite pedagogic, but very annoying.

ALL

Everything that PySpark can spit, it will spit.

Note

When using the pyspark shell, anything chattier than WARN might appear when you’re typing a command, which makes it quite hard to input commands into the shell. You’re welcome to play with the log levels as you please, but we won’t show any output unless it’s valuable for the task at hand.

Setting the log level to  ALL is a very good way to annoy oblivious co-workers if they don’t lock their computers. You haven’t heard it from me.

You now have the REPL fired-up and ready for your input.

[4] It does actually a whole lot more, but we will cover other aspects in Chapter 7.

2.2  MAPPING OUR PROGRAM

In the Chapter introduction, we introduced our problem statement: what are the most popular words in the English language? Before even hammering code in the REPL, we can start by mapping the major steps our program will need to perform.

1. Read: Read the input data (we’re assuming a plain text file)

2. Token: Tokenize each word

3. Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

Visually, a simplified flow of our program would look like figure 2.1

Figure 2.1. A simplified flow of our program, illustrating the 5 steps.

Our goal is quite lofty: the English language produced through history an unfathomable amount of written material. Since we are learning, we’ll start with a relatively small source, get our program working, and then scale it to accommodate a larger body of text. For this, I chose to use Jane Austen’s Pride and Prejudice, since it’s already in plain text and freely available.

Data analysis and Pareto’s principle

Pareto’s principle, also known commonly as the 80/20 rules, is often summarized as 20% of the efforts will yield 80% of the results. In data analysis, we can consider that 20% to be analysis, visualization, machine learning models, anything that provides tangible value to the recipient.

The remainder is what I call invisible work: ingesting the data, cleaning it, figuring its meaning and shaping it into a usable form. If you look at your simple steps, Step 1 to 3 can be considered invisible work: we’re ingesting data and getting it ready for the counting process. Step 4 and 5 are really the visible ones that are answering our question (one could argue that only Step 5 is performing visible work, but let’s not split hairs here). Steps 1 to 3 are there because the data requires processing to be usable for our problem. They aren’t core to our problem, but we can’t do without them.

When building your own project, this will be the part that will be the most time consuming and you might be tempted (or pressured!) to skimp on it. Always keep in mind that the data you ingest and process is the raw material of your programs, and that feeding it garbage will yield, well, garbage.

2.3  READING AND INGESTING DATA INTO A DATA FRAME

The first step of our program is to ingest the data in a structure we can perform work in. PySpark provides two main structures for performing data manipulation:

1. The Resilient Distributed Dataset (or RDD)

2. The data frame

The RDD can be seen like a distributed collection of objects. I personally visualize this as a bag that you give orders to. You pass orders to the RDD through regular Python functions over the items in the bag.

The data frame is a stricter version of the RDD: conceptually, you can think of it like a table, where each cell can contain one value. The data frame makes heavy usage of the concept of columns where you perform operation on columns instead of on records, like in the RDD. Figure 2.2 provides a visual summary of the two structures.

If you’ve used SQL in the past, you’ll find that the data frame implementation takes a lot of inspiration from SQL. The module name for data organization and manipulation is even named pyspark.sql! Furthermore, Chapter 7 will teach you how to mix PySpark and SQL code within the same program.

Figure 2.2. A RDD vs a data frame. In the RDD, each record is processed independently. With the data frame, we work with its columns, performing functions on them.

Some language convention

Since this book will talk about data frames more than anything else, I prefer using the non-capitalized nomenclature, i.e. "data frame". I find this to be more readable than using capital letters or even DataFrame without a space.

When referring to the PySpark object directly, I’ll use DataFrame but with a fixed-width font. This will help differentiate between data frame the concept and DataFrame the object.

This book will focus on the data frame implementation as it is more modern and performs faster for all but the most esoteric tasks. Chapter 8 will discuss about trade-offs between the RDD and the data frame. Don’t worry: once you’re learned the data frame, it’ll be a breeze the learn the RDD.

Reading data into a data frame is done through the DataFrameReader object, which we can access through spark.read. The code in listing 2.3 displays the object, as well as the methods it exposes. We recognize a few file formats: csv stands for comma separated values (which we’ll use as early as Chapter 4), json for JavaScript Object Notation (a popular data exchange format) and text is plain text.

Listing 2.3. The DataFrameReader object

In [3]: spark.read

Out[3]: <pyspark.sql.readwriter.DataFrameReader at 0x115be1b00>

In [4]: dir(spark.read)

Out[4]: [<some content removed>, _spark', 'csv', 'format', 'jdbc', 'json',

'load', 'option', 'options', 'orc', 'parquet', 'schema', 'table', 'text']

PySpark reads your data

PySpark provides many readers to accommodate the different ways you can process data. Under the hood, spark.read.csv() will map to spark.read.format('csv').load() and you may encounter this form in the wild. I usually prefer using the direct csv method as it provides a handy reminder of the different parameters the reader can take.

orc and parquet are also data format especially well suited for big data processing. ORC (which stands for Optimized Row Columnar) and Parquet are competing data format which serves pretty much the same purpose. Both are open-sourced and now part of the Apache project, just like Spark.

PySpark defaults to using parquet when reading and writing files, and we’ll use this format to store our results through the book. I’ll provide a longer discussion about the usage, advantages and trade-offs of using Parquet or ORC as a data format in Chapter 6.

Let’s read our data file. I am assuming that you launched PySpark at the root of this book’s repository. Depending on your case, you might need to change the path where the file is located.

Listing 2.4. "Reading" our Jane Austen novel in record time

book = spark.read.text("./data/ch02/1342-0.txt")

book

# DataFrame[value: string]

We get a data frame, as expected! If you input your data frame, conveniently named book, into the shell, you see that PySpark doesn’t actually output any data to the screen. Instead, it prints the schema, which is the name of the columns and their type. In PySpark’s world, each column has a type: it represents how the value is represented by Spark’s engine. By having the type attached to each column, you can know instantly what operations you can do on a the data. With this information, you won’t inadvertently try to add an integer to a string: PySpark won’t let you add 1 to "blue". Here, we have one column, named value, composed of a string. A quick graphical representation of our data frame would look like figure 2.3. Besides being a helpful reminder of the content of the data frame, types are integral to how Spark processes data quickly and accurately. We will explore the subject extensively in Chapter 5.

Figure 2.3. A high-level schema of a our book data frame, containing a value string column. We can see the name of the column, its type, and a small snippet of the data.

If you want to see the schema in a more readable way, you can use the handy method printSchema(), illustrated in Listing 2.5. This will print a tree-like version of the data frame’s schema. It is probably the method I use the most when developing interactively!

Listing 2.5. Printing the schema of our data frame

book.printSchema()

# root

#  |-- value: string (nullable = true)

Same information, displayed in a friendlier way.

Speeding up your learning by using the shell

This doesn’t just apply to PySpark, but using the functionality of the shell can often save a lot of searching into the documentation. I am a big fan of using dir() on an object when I don’t remember the exact method I want to apply, like I did in listing 2.3.

PySpark’s source code is very well documented, if you’re unsure about the proper usage of a function, class or method, you can print the doc attribute or, for those using IPython, use a trailing question mark (or two, if you want more details).

Listing 2.6. Using PySpark’s documentation directly in the REPL

In [*]: print(spark.read.__doc__)

Interface used to load a :class:`DataFrame` from external storage systems

(e.g. file systems, key-value stores, etc). Use :func:`spark.read`

to access this.

.. versionadded:: 1.4

In [*]: spark.read?

Type:        property

String form: <property object at 0x1159a0958>

Docstring:

Returns a :class:`DataFrameReader` that can be used to read data

in as a :class:`DataFrame`.

:return: :class:`DataFrameReader`

.. versionadded:: 2.0

2.4  EXPLORING DATA IN THE DATAFRAME STRUCTURE

One of the key advantages of using the REPL for interactive development is that you can peek at your work as you’re performing it. Now that our data is loaded into a data frame, we can start looking at how PySpark structured our text.

In 2.3, we saw that the default behaviour of imputing a data frame in the shell is to provide the schema or column information of the object. While very useful, sometimes we want to take a peek of the data.

Enter the show() method.

2.4.1  Peeking under the hood: the show() method

The most fundamental operation of any data processing library or program is displaying the data it contains. In the case of PySpark, it becomes even more important, since we’ll definitely be working with data that goes beyond a screenful. Still, sometimes, you just want to see the data, raw, without any complications.

The show() method displays a sample of the data back to you. Nothing more, nothing less. With printSchema(), it will become one of your best friend to perform data exploration and validation. By default, it will show 20 rows and truncate long rows. The code in listing 2.7 shows the default behaviour of the method, applied to our book data frame. For text data, the length limitation is limiting (pun intended). Fortunately, show() provides some options to display just what you need.

Listing 2.7. Showing a little data using the .show() method.

book.show()

# +--------------------+

# |               value|

# +--------------------+

# |The Project Guten...|

# |                    |

# |This eBook is for...|

# |almost no restric...|

# |re-use it under t...|

# |with this eBook o...|

# |                    |

# |                    |

# |Title: Pride and ...|

# |                    |

# | Author: Jane Austen|

# |                    |

# |Posting Date: Aug...|

# |Release Date: Jun...|

# |Last Updated: Mar...|

# |                    |

# |   Language: English|

# |                    |

# |Character set enc...|

# |                    |

# +--------------------+

# only showing top 20 rows

The show() method takes three optional parameters.

- n can be set to any positive integer, and will display that number of rows.

- truncate, if set to true, will truncate the columns to display only 20 characters. Set to False to display the whole length, or any positive integer to truncate to a specific number of characters.

- vertical takes a Boolean value and, when set to True, will display each record as a small table. Try it!

The code in listing 2.8 shows a couple options, stating with showing 10 records and truncating then at 50 characters. We can see more of the text now!

Listing 2.8. Showing less length, more width with the show() method parameters

book.show(10, truncate=50)

# +--------------------------------------------------+

# |                                             value|

# +--------------------------------------------------+

# |The Project Gutenberg EBook of Pride and Prejud...|

# |                                                  |

# |This eBook is for the use of anyone anywhere at...|

# |almost no restrictions whatsoever.  You may cop...|

# |re-use it under the terms of the Project Gutenb...|

# |    with this eBook or online at www.gutenberg.org|

# |                                                  |

# |                                                  |

# |                        Title: Pride and Prejudice|

# |                                                  |

# +--------------------------------------------------+

# only showing top 10 rows

With the show() and printSchema() methods under your belt, you’re now fully ready to experiment with your data.

Non-lazy Spark?

If you are coming from an other data frame implementation, such as Pandas or R data.frame, you might find it odd to see the structure of the data frame instead of a summary of the data when calling the variable. The show() method might appear as a nuisance to you.

If we take a step back and think about PySpark’s use-cases, it makes a lot of sense. show() is an action, since it perform the visible work of printing data on the screen. As savvy PySpark programmers, we want to avoid to accidentally trigger the chain of computations, so the Spark developers made show() explicit. When building a complicated chain of transformations, triggering its execution is a lot more annoying and time-consuming than having to type the show() method when you’re ready.

That being said, there are some moments, especially when learning, when you want your data frames to be evaluated after each transformation (which we call eager evaluation). Since Spark 2.4.0, you can configure the SparkSession object to support printing to screen. We will cover how to create a SparkSession object in greater details in Chapter 3, but if you want to use eager evaluation in the shell, you can paste the following code in your shell.

from pyspark.sql import SparkSession

spark = (SparkSession.builder

.config("spark.sql.repl.eagerEval.enabled", "True")

.getOrCreate())

All the examples in the book assume that the data frames are evaluated lazily, but this option can be useful if you’re demonstrating Spark. Use it as you see fit, but remember that Spark owe a lot of its performance to its lazy evaluation. You’ll be leaving some extra horsepower on the table!

Our data is ingested and we’ve been able to see the two important aspects of our data frame:

- its structure, via the printSchema() method;

- a subset of the data it contains, via the show() method.

We can now start the real work: performing transformations on the data frame to accomplish our goal. Let’s take some time to review our 5 steps we outlined at the beginning of the chapter.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. Token: Tokenize each word

3. Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

Our next step will be to tokenize or separate each word so we can clean and count them.

2.5  MOVING FROM A SENTENCE TO A LIST OF WORDS

When ingesting our selected text into a data frame, PySpark created one record for each line of text, and provided a value column of type String. In order to tokenize each word, we need to split each string into a list of distinct words.

I’ll start by providing the code in one fell swoop, and then we’ll break down each step one at a time. You can see it in all its glory in listing 2.9.

Listing 2.9. Splitting our lines of text into arrays or words

from pyspark.sql.functions import split

lines = book.select(split(book.value, " ").alias("line"))

lines.show(5)

# +--------------------+

# |               value|

# +--------------------+

# |[The, Project, Gu...|

# |                  []|

# |[This, eBook, is,...|

# |[almost, no, rest...|

# |[re-use, it, unde...|

# +--------------------+

# only showing top 5 rows

In a single line of code (I don’t count the import or the show() which is only being used to display the result), we’ve done quite a lot. The remainder of this section will introduce basic column operations and explain how we can build our tokenization step as a one-liner. More specifically, we’ll learn about

- The select() method and its canonical usage, which is selecting data.

- The alias() method to rename transformed columns

- Importing column functions from pyspark.sql.functions and using them.

2.5.1  Selecting specific columns using select()

This section will introduce the most basic functionality of select(), which is to select one or more columns from your data frame. It’s a conceptually very simple method, but provides the foundation for many additional operations on your data.

In PySpark’s world, a data frame is made out of Column objects, and you perform transformations on them. The most basic transformation is the identity, where you return exactly what was provided to you. If you’ve used SQL in the past, you might think that this sounds like a "SELECT" statement, and you’d be right! You also get a free pass: the method name is also conveniently named select().

We’ll go over a quick example: selecting the only columns of our book data frame. Since we already know the expected output, we can focus on the gymnastics fo the select() method. Listing 2.10 provides the code performing that very useful task.

Listing 2.10. The simplest select statement ever, provided by PySpark

book.select(book.value)

PySpark provides for each column in its data frame a dot notation that refers to the column. This is definitely the simplest way to select a column, as long as the name doesn’t contain any funny characters: PySpark will accept $!@# as a column name, but you won’t be able to use the dot notation for this column.

PySpark provides more than one way to select columns. I displayed the three most common in listing 2.11.

Listing 2.11. Selecting the value column from the book data frame, three ways

from pyspark.sql.functions import col

book.select(book.value)

book.select(book["value"])

book.select(col("value"))

The first one is our old trusty dot notation that we got acquainted with a few paragraphs ago.

The second one uses brackets instead of the dot to name the column. It addresses the $!@# problem since you pass the name of the column as a string.

The last one uses the col function from the pyspark.sql.functions module. The main difference here is that you don’t specify that the column comes from the book data frame. This will become very useful when working with more complex data pipelines in Part 2 of the book. I’ll use the col object as much as I can since I consider its usage to be more idiomatic and it’ll prepare us for more complex use-cases.

There is theoretically a fourth way to select a column, which is by passing the column name as a simple string. In this case, you’d just have to write book.select('value').

For simple select statements (and other methods that I’ll cover later), it can be a viable option. That being said, it’s not as flexible as the other options and the moment that your code requires column transformations, like in 2.5.2, you’ll have to use another option. Future-proof your code by picking up one of the three from the start.

2.5.2  Transforming columns: splitting a string into a list of words

We just saw a very simple way to select a column in PySpark. We will now build on this foundation by selecting a transformation of a column instead. This provides a powerful and flexible way to express our transformations, and as you’ll see, this pattern will be frequently used when manipulating data.

PySpark provides a split() function in the pyspark.sql.functions module for splitting a longer string into a list of shorter strings. The most popular use-case for this function is to split a sentence into words. The split() function takes two parameters.

1. A column object containing strings

2. A Java regular expression delimiter to split the strings against.

Since we want to split words, we won’t over-complicate our regular expression and just use the space character to split. Listing 2.12 shows the results of our code.

Listing 2.12. Splitting our lines of text into lists of words

from pyspark.sql.functions import col, split

lines = book.select(split(col("value"), " "))

lines

# DataFrame[split(value,  ): array<string>]

lines.printSchema()

# root

#  |-- split(value,  ): array (nullable = true)

#  |    |-- element: string (containsNull = true)

lines.show(5)

# +--------------------+

# |     split(value,  )|

# +--------------------+

# |[The, Project, Gu...|

# |                  []|

# |[This, eBook, is,...|

# |[almost, no, rest...|

# |[re-use, it, unde...|

# +--------------------+

# only showing top 5 rows

The split functions transformed our string column into an array column, containing one or more string elements. This is what we were expecting: even before looking at the data, seeing that the structure behaves according to plan is a good way to sanity-check our code.

Looking at the 5 rows we’ve printed, we can see that our values are now separated by a comma and wrapped in square brackets, which is how PySpark visually represents an array. The second record is empty, so we just see [], an empty array.

PySpark’s built-in functions for data manipulations are extremely useful and you should definitely spend a little bit of time going over the API documentation to see what’s available there. If you don’t find exactly what you’re after, Chapter 6 will cover how you can create your own function over Column objects.

Advanced topic: PySpark’s architecture and the JVM heritage

If you’re like me, you might be interested to see how PySpark builds its core pyspark.sql.functions functions. If you look at the source code for split(), you might be in for a disappointment.

since(1.5)

@ignore_unicode_prefix

def split(str, pattern):

"""

Splits str around pattern (pattern is a regular expression).

.. note:: pattern is a string represent the regular expression.

>>> df = spark.createDataFrame([('ab12cd',)], ['s',])

>>> df.select(split(df.s, '[0-9]+').alias('s')).collect()

[Row(s=[u'ab', u'cd'])]

"""

sc = SparkContext._active_spark_context

return Column(sc._jvm.functions.split(_to_java_column(str), pattern))

It effectively refers to the split function of the sc_jvm.functions object. This has to do with how the data frame was built. PySpark’s uses a translation layer to call JVM functions for its core functions. This makes PySpark faster, since you’re not transforming your Python code into JVM one all the time: it’s already done for you. It also makes porting PySpark to another platform a little easier: if you can call the JVM functions directly, you don’t have to re-implement everything.

This is one of the trade-offs of standing on the shoulders of the Spark giant. This also explains why PySpark uses JVM-base regular expressions instead of the Python ones in its built-in functions. Part 3 will expand on this subject greatly, but in the meantime, don’t be surprised if you explore PySpark’s source code!

PySpark renamed our column in a very weird way: split(value, ) isn’t what I’d consider an awesome name for our column. Just like the infomercials say, there must be a better way!.

2.5.3  Renaming columns: alias and withColumnRenamed

When performing transformation on your columns, PySpark will give a default name to the resulting column. In our case, we were blessed by the split(value, ) name after splitting our value column using a space as the delimiter. While accurate, it’s definitely not programmer friendly.

There is an implicit assumption that you’ll want to rename the resulting column yourself, using the alias() method. It’s usage isn’t very complicated: when applied to a column, it takes a single parameter, and returns the column it was applied to, with the new name. A simple demonstration is provided in listing 2.13.

Listing 2.13. Our data frame before and after the aliasing

book.select(split(col("value"), " ")).printSchema()

# root

#  |-- split(value,  ): array (nullable = true) ❶

#  |    |-- element: string (containsNull = true)

book.select(split(col("value"), " ").alias("line")).printSchema()

# root

#  |-- line: array (nullable = true) ❷

#  |    |-- element: string (containsNull = true)

❶

Our new column is called  split(value, ), which isn’t really pretty

❷

We aliased our column to the name  line. Much better!

alias() provides a clean and explicit way to name your columns after you’ve performed work on it. On the other hand, it’s not the only renaming player in town. Another equally valid way to do so is by using the .withColumnRenamed() method on the data frame. It takes two parameters: the current name of the column and the wanted name of the column. Since we’re already performing work on the column with split, chaining alias makes a lot more sense than using another method. Listing 2.14 shows you the two different approaches.

When writing your own code, choosing between those two options is pretty easy:

- when you’re using a method where you’re specifying which columns you want to appear (like select in our case here, but the next chapters will have many other examples), use alias.

- if you just want to rename a column without changing the rest of the data frame, use .withColumnRenamed.

Listing 2.14. Renaming a column via alias on the column and withColumnRenamed on the DataFrame

# This looks a lot cleaner

lines = book.select(split(book.value, " ").alias("line"))

# This is messier, and you have to remember the name PySpark assigns automatically

lines = book.select(split(book.value, " "))

lines = lines.withColumnRenamed("split(value,  )", "line")

This section introduced a new set of PySpark fundamentals: we learned how to select not only plain columns, but also column transformations. We also learned how to explicitly name the resulting columns, avoiding PySpark’s predictable but jarring naming convention. We can then move forward with the remainder of the operations. If we look at our 5 steps, we’re halfway done with step 2: we have a list of words, but we need for each token or word to be its own records.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. [IN PROGRESS] Token: Tokenize each word

3. Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

2.6  RESHAPING YOUR DATA: EXPLODING A LIST INTO ROWS

When working with data, a key element in data preparation is making sure that it "fits the mold": this means making sure that the structure containing the data is logical and appropriate for the work at hand. At the moment, each record of our data frame contains multiple words into an array of strings. It would be better to have one record for each word.

Enter the explode function. When applied to a column containing a container-like data structure (such as an array), it’ll take each element and give it its own row. This is much more easier explained visually than using words, so figure 2.4 explains the process.

Figure 2.4. Exploding a data frame of array[String] into a data frame of String. Each element of each array becomes its own record.

The code follows the same structure as split, and you can see the results in listing 2.15. We now have a data frame containing at most one word per row. We are almost there!

Before continuing our data processing journey, we can take a step back and look at a sample of the data. Just by looking at the 15 rows returned, we can see that Prejudice, has a comma and that the cell between Austen and This contains the empty string. That gives us a good blueprint of the next steps that needs to be performed before we start analyzing word frequency.

Listing 2.15. Exploding a column of arrays into rows of elements

from pyspark.sql.functions import explode, col

words = lines.select(explode(col("line")).alias("word"))

words.show(15)

# +----------+

# |      word|

# +----------+

# |       The|

# |   Project|

# | Gutenberg|

# |     EBook|

# |        of|

# |     Pride|

# |       and|

# |Prejudice,|

# |        by|

# |      Jane|

# |    Austen|

# |          |

# |      This|

# |     eBook|

# |        is|

# +----------+

# only showing top 15 rows

Looking back at our 5 steps, we can now conclude step 2, and our words are tokenized. Let’s attack the third one, where we’ll be cleaning our words to simplify the counting.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. [DONE] Token: Tokenize each word

3. Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

2.7  WORKING WITH WORDS: CHANGING CASE AND REMOVING PUNCTUATION

So far, with split and explode, our pattern has been the following: find the relevant function in pyspark.sql.functions, apply it, profit! This section will use the same winning formula to normalize the case of our words and remove punctuation, so we’ll walk a little faster.

Listing 2:16 contains the source code to lower the case of all the words in the data frame. The code should look very familiar: we select a column transformed by lower, a PySpark function lowering the case of the data inside the column passed as a parameter. We then alias the resulting column to word to avoid PySpark’s default nomenclature. Illustrated, it could look approximately like figure 2.5.

Listing 2.16. Lower the case of the words in the data frame

from pyspark.sql.functions import lower

words_lower = words.select(lower(col("word")).alias("word_lower"))

words_lower.show()

# +-----------+

# | word_lower|

# +-----------+

# |        the|

# |    project|

# |  gutenberg|

# |      ebook|

# |         of|

# |      pride|

# |        and|

# | prejudice,|

# |         by|

# |       jane|

# |     austen|

# |           |

# |       this|

# |      ebook|

# |         is|

# |        for|

# |        the|

# |        use|

# |         of|

# |     anyone|

# +-----------+

# only showing top 20 rows

Figure 2.5. Applying lower to a column and aliasing the resulting column: a creative explanation

Removing punctuation and other non-useful characters can be a little trickier. We won’t improvise a full NLP (Natural Language Processing, which is an amazingly field of data analysis/science that focuses on text. We’ll cover it briefly in Chapter 9!) library here, relying on the functionality PySpark provides in its data manipulation toolbox. In the spirit of keeping this exercise simple, we’ll keep the first contiguous group of letters as the word, including apostrophes, and remove the rest. It will effectively remove punctuation, quotation marks and other symbols, at the expense of being less robust with more exotic word construction. listing 2.17 shows the code in all its splendour.

Regular expressions for the rest of us

PySpark uses regular expressions in two functions we used so far: regexp_extract() and split(). You do not have to be a regexp expert to work with PySpark (I certainly am not). Through the book, each time that I’ll use a non-trivial regular expression, I’ll provide a plain English definition so you can follow along.

If you are interested in building your own, the RegExr (regexr.com/) website is really useful, as well as the Regular Expression Cookbook, by Steven Levithan and Jan Goyvaerts (O’Reilly, 2012).

Listing 2.17. Using regexp_extract to keep what looks like a word

from pyspark.sql.functions import regexp_extract

words_clean = words_lower.select(

regexp_extract(col("word_lower"), "[a-z]*", 0).alias("word")  ❶

)

words_clean.show()

# +---------+

# |     word|

# +---------+

# |      the|

# |  project|

# |gutenberg|

# |    ebook|

# |       of|

# |    pride|

# |      and|

# |prejudice|

# |       by|

# |     jane|

# |   austen|

# |         |

# |     this|

# |    ebook|

# |       is|

# |      for|

# |      the|

# |      use|

# |       of|

# |   anyone|

# +---------+

# only showing top 20 rows

❶

We only match for multiple lower-case characters (between  a and z). The star (*) will match for 0 or more occurrences.

Our data frame of words looks pretty regular by now, with the exception of the empty cell between austen and this. We will solve this with a judicious usage of filtering.

2.8  FILTERING ROWS

An important data manipulation operation is to be able to filter records according to a certain predicate. In our case, blank cells shouldn’t be considered: they’re not words! Conceptually, we should be able to provide a test to perform on each records: if it returns true, we keep the record. False? You’re out!

PySpark provides not one, but two identical methods to perform this task. You can either use .filter() or its alias .where(). This duplication is to ease the transition for users coming from other data processing engines or libraries: some use one, some the other. PySpark provides both, so no arguments possible! I personally prefer where() because it’s one character less and my w key is less used than f, but you might have other motives. If we look at listing 2.18, we can see that columns can be compared to values using the usual Python comparison operators. In this case, we’re using the "not equal", or !=.

Listing 2.18. Filtering rows in your data frame, using where or filter.

words_nonull = words_clean.where(col("word") != "")

words_nonull.show()

# +---------+

# |     word|

# +---------+

# |      the|

# |  project|

# |gutenberg|

# |    ebook|

# |       of|

# |    pride|

# |      and|

# |prejudice|

# |       by|

# |     jane|

# |   austen|

# |     this| <-- See, the blank cell is gone!

# |    ebook|

# |       is|

# |      for|

# |      the|

# |      use|

# |       of|

# |   anyone|

# | anywhere|

# +---------+

# only showing top 20 rows

We could have tried to filter earlier in our program. It’s a trade-off to consider: if we filtered too early, our filtering clause would have been comically complex for no good reason. Since PySpark caches all the transformations until an action is triggered, we can focus on the readability of our code and let Spark optimize our intent, like we saw in Chapter 1. We’ll see in Chapter 3 how you can transform PySpark code so it almost reads like a series of written instructions and take advantage of the lazy evaluation.

This seems like a good time to take a break and reflect on what we accomplished so far. If we look at our 5 steps, we’re 60% of the way there. Our cleaning step took care of non-letter characters and filtered the empty records. We’re ready for counting and displaying the results of our analysis.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. [DONE] Token: Tokenize each word

3. [DONE] Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

In terms of PySpark operations, we covered a huge amount of ground in the data manipulation space. You can now select not only columns but transformations of columns, renaming them as you please after the fact. We learned how to break nested structures, such as arrays, into single records. We finally learn how to filter records using simple tests.

We can now rest. The next chapter will cover the end of our program. We will also be looking at bringing our code in one single file, moving away from the REPL into batch mode. We’ll explore options to simplify and increase the readability of our program, and then finish by scaling it to larger corpus of texts.

2.9  SUMMARY

- Almost all PySpark programs will revolve around 3 major steps: reading, transforming and exporting data.

- PySpark provides a REPL (read, eval, print loop) via the pyspark shell where you can experiment interactively with data.

- A PySpark’s data frame is a collection of columns. You operate on the structure using chained transformations. PySpark will optimize the transformations and perform the work only when you submit an action, such as show(). This is one of the pillars of PySpark’s performance.

- PySpark’s repertoire of functions that operate on columns are located in pyspark.sql.functions.

- You can select columns or transformed columns via the select() statement.

- You can filter columns using the where() or filter() methods and providing a test that will return True or False, only the records returning True will be kept.

- PySpark can have columns of nested values, like arrays of elements. In order to extract the elements into distinct records, you need to use the explode() method.

2.10  EXERCISES

Exercise 2.1

Rewrite the following code snippet, removing the withColumnRenamed method. Which version is clearer and easier to read?

from pyspark.sql.functions import col, length

# The `length` function returns the number of characters in a string column.

ex21 = (

spark.read.text("./data/Ch02/1342-0.txt")

.select(length(col("value")))

.withColumnRenamed("length(value)", "number_of_char")

)

Exercice 2.2

The following code blocks gives an error. What is the problem and how can you solve it?

from pyspark.sql.functions import col, greatest

ex22.printSchema()

# root

#  |-- key: string (containsNull = true)

#  |-- value1: long (containsNull = true)

#  |-- value2: long (containsNull = true)

# `greatest` will return the greatest value of the list of column names,

# skipping null value

# The following statement will return an error

ex22.select(

greatest(col("value1"), col("value2")).alias("maximum_value")

).select(

"key", "max_value"

)

Exercise 2.3

Let’s take our words_nonull data frame, available in listing 2.18. You can use the code in the repository (code/Ch02/end_of_chapter.py) into your REPL to get the data frame loaded.

a) Remove all of the occurrences of the word "is"

b) (Challenge) Using the length function explained in exercise 2.1, keep only the words with more than 3 characters.

Exercise 2.4

The where clause takes a Boolean expression over one or many column to filter the data frame (see 2.8). Beyond the usual Boolean operators (>, <, ==, ⇐, >=, !=), PySpark provides other functions returning Boolean columns in the pyspark.sql.functions module.

A good example is the isin() function, which takes a list of values as a parameter, and will return only the records where the value in the column equals a member of the list.

Let’s say you want to remove the words is, not, the and if from your list of words, using a single where() method on the words_nonull data frame (see exercise 2.3). Write the code to do so.

Exercise 2.5

One of your friends come to you with the following code. They have no idea why it doesn’t work. Can you diagnose the problem, explain why it is an error and provide a fix?

from pyspark.sql.functions import col, split

book = spark.read.text("./data/ch02/1342-0.txt")

book = book.printSchema()

lines = book.select(split(book.value, " ").alias("line"))

words = lines.select(explode(col("line")).alias("word"))

3

SUBMITTING AND SCALING YOUR FIRST PYSPARK PROGRAM

This chapter covers:

- Summarizing data using groupby and a simple aggregate function

- Ordering results for display

- Writing data from a data frame

- Using spark-submit to launch your program in batch mode

- Simplify the writing of your PySpark using method chaining

- Scaling your program almost for free!

Chapter 2 dealt with all the data preparation work for our word frequency program. We read the input data, tokenized each word and cleaned our records to only keep lower-case words. If we bring out our outline, we only have Step 4 and 5 to complete.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. [DONE] Token: Tokenize each word

3. [DONE] Clean: Remove any punctuation and/or tokens that aren’t words.

4. Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

After tackling those two last steps, we’ll look at packaging our code in a single file to be able to submit it to Spark without having to launch a shell. We’ll take a look at our completed program and look at simplifying our program by removing intermediate variables. We’ll finish with scaling our program to accommodate more data sources.

3.1  GROUPING RECORDS: COUNTING WORD FREQUENCIES

If you take our data frame in the same shape as it was left at the end of Chapter 2 (hint: look at code/Ch02/end_of_chapter.py if you want to catch up), there is not much to be done. Having a data frame containing one single word per record, we just have to count the word occurrences and take the top contenders. This section will show how to count records using the GroupedData object and perform an aggregation function (here counting the items) on each group. A more general blueprint for grouping and aggregating data will be touched upon in Chapter 4, but we’ll see the basics in this Chapter.

The easiest way to count record occurrence is to use the groupby method, passing the columns we wish to group on as a parameter. The code in listing 3.1 shows that the returned value is a GroupedData object, not a DataFrame. I call this GroupedData object a transitional object: PySpark grouped our data frame on the word column, waiting for instructions on how to summarize the information contained in each group. Once we apply the count() method, we get back a data frame containing the grouping column word, as well as count column containing the number of occurrences for each word. A visual interpretation of how a DataFrame morphes into a GroupedData object is on display in figure 3.1

Listing 3.1. Counting word frequencies using groupby() and count()

groups = words_nonull.groupby(col("word"))

groups

# <pyspark.sql.group.GroupedData at 0x10ed23da0>

results = words_nonull.groupby(col("word")).count()

results

# DataFrame[word: string, count: bigint]

results.show()

# +-------------+-----+

# |         word|count|

# +-------------+-----+

# |       online|    4|

# |         some|  203|

# |        still|   72|

# |          few|   72|

# |         hope|  122|

# |        those|   60|

# |     cautious|    4|

# |       lady's|    8|

# |    imitation|    1|

# |          art|    3|

# |      solaced|    1|

# |       poetry|    2|

# |    arguments|    5|

# | premeditated|    1|

# |      elevate|    1|

# |       doubts|    2|

# |    destitute|    1|

# |    solemnity|    5|

# |gratification|    1|

# |    connected|   14|

# +-------------+-----+

# only showing top 20 rows

Figure 3.1. A schematic representation of our groups object. Each small box represents a record.

Peeking at the results data frame in listing 3.1, we see that the results are in no specific order. As a matter of fact, I’d be very surprised if you had the exact same order of words as me! This has to do with how PySpark manages data: in Chapter 1, we learned that PySpark distributes the data across multiple nodes. When performing a grouping function, such as groupby, each worker performs the work on its assigned data. groupby and count are transformations, so PySpark will queue them lazily until we request an action. When we pass the show method to our results data frame, it triggers the chain of computation that we see in figure 3.2.

Figure 3.2. A distributed group by on our words_nonull data frame. The work is performed in a distributed fashion until we need to assemble the results in a cohesive display, via show().

Because of the distributed and lazy nature of PySpark, it makes sense to not care about the ordering of records until explicitly mentioned. Since we wish to see the top words on display, let’s put a little order in our data frame and, by the same occasion, complete the last step of our program.

3.2  ORDERING THE RESULTS ON THE SCREEN USING ORDERBY

In 3.1, we explained why PySpark doesn’t necessarily maintain order of records when performing transformations. If we look at our 5 step blueprint, the last step is to return the top N records, for different values of N. We already know how to show a specific number of records, so this section will focus on ordering the records in a data frame.

1. [DONE] Read: Read the input data (we’re assuming a plain text file)

2. [DONE] Token: Tokenize each word

3. [DONE] Clean: Remove any punctuation and/or tokens that aren’t words.

4. [DONE] Count: Count the frequency of each word present in the text

5. Answer: Return the top 10 (or 20, 50, 100)

The method PySpark provides for ordering records in a data frame is orderBy. Without further ado, let’s take a look at listing 3.2, which performs and shows the results of this computation.

Listing 3.2. Displaying the top 10 words in Jane’s Austen Pride and Prejudice

results.orderBy("count", ascending=False).show(10)

# +----+-----+

# |word|count|

# +----+-----+

# | the| 4480|

# |  to| 4218|

# |  of| 3711|

# | and| 3504|

# | her| 2199|

# |   a| 1982|

# |  in| 1909|

# | was| 1838|

# |   i| 1749|

# | she| 1668|

# +----+-----+

# only showing top 10 rows

The list is very much unsurprising: even though we can’t argue with Ms. Austen’s vocabulary, she isn’t immune to the fact that the English language needs pronouns and other very common words. In natural language processing, those words are called stop words and would be removed. As far as we are concerned, we solved our original query and can rest easy. Should you want to get the top 20, top 50, or even top 1,000, it’s easily done by changing the parameter to show().

PySpark’s method naming convention zoo

If you have a very good sense of details, you might have noticed that we used groupby (lowercase), but orderBy (lowerCamelCase, where you capitalize each word but the first). This seems like an odd design choice.

groupby is in fact an alias for groupBy, just like where is an alias of filter. My guess is that the PySpark developers found that a lot of typing mistakes were avoided by accepting the two cases. orderBy didn’t have that luxury, for a reason that escape my understanding, so we need to be mindful. You can see the output of IPython’s auto-complete for those two methods in figure 3.3.

Figure 3.3. PySpark’s camelcase vs camelCase

Part of this incoherence is due to Spark’s heritage. Scala prefers camelCase for methods. On the other hand, we saw regexp_extract, which uses Python’s preferred snake_case (words separated by an underscore) in Chapter 2. There is no magic secret here: you’ll have to be mindful about the different case conventions at play in PySpark.

Showing results on the screen is great for quick assessment, but most of the time, you’ll want them to have some sort of longevity. It’s much better to save those results into a file, so we’ll be able to re-use those results without having to compute everything each time.

3.3  WRITING DATA FROM A DATA FRAME

Having the data on the screen is great for interactive development, but you’ll often want to export your results. PySpark treats writing data a little differently than most data processing libraries, since it can scale to immense volumes of data. For this, we’ll start by naively write our results in a CSV file, and see how PySpark performs the job.

Listing 3.3 shows the code and results. A data frame exposes the write method, which we can specialize for CSV by chaining the csv method. This is very consistent with the read method we saw in Chapter 2. If we look at the results, we can see that PySpark didn’t create a results.csv file. Instead, it created a directory of the same name, and put 201 files inside the directory (200 CSVs + 1 _SUCCESS file).

Listing 3.3. Writing our results in multiple CSV files, one per partition

results.write.csv('./results.csv')

# The following command is run using the shell.

# In IPython, you can use the bang pattern (! ls -l)

# to get the same results without leaving the console.

#`ls -l` is a Unix command listing the content of a directory.

# On windows, you can use `dir` instead

$ ls -l

# [...]

# -rw-r--r--@   1 jonathan_rioux  247087069  724726 Jul 30 17:30 1342-0.txt

# drwxr-xr-x  404 jonathan_rioux  247087069   12928 Aug  4 13:31 results.csv ❶

$ ls -l results.csv/

# [...]

# -rw-r--r--  1 jonathan_rioux  247087069    0 Aug  4 13:31 _SUCCESS ❷

# -rw-r--r--  1 jonathan_rioux  247087069  468 Aug  4 13:31 part-00000-615b75e4-ebf5-44a0-b337-405fccd11d0c-c000.csv

# [...]

# -rw-r--r--  1 jonathan_rioux  247087069  353 Aug  4 13:31 part-00199-615b75e4-ebf5-44a0-b337-405fccd11d0c-c000.csv ❸

❶

The results are written in a directory called  results.csv

❷

The  _SUCCESS file means the operation was successful

❸

We have  part-00000 to part-00199, which means our results are split across 200 files

There it is, ladies and gentleman! The first moment where we have to care about PySpark’s distributed nature.

Just like PySpark will distribute the transformation work across multiple workers, it’ll do the same for writing data. While it might look like a nuisance for our simple program, it is tremendously useful when working in distributed environments. When you have a large cluster of nodes, having many smaller files makes it easy to logically distribute reading and writing the data, making it way faster than having a single massive file.

By default, PySpark will give you 1 file per partition. This means that our program, as run on my machine, yields 200 partitions at the end. This isn’t the best for portability. In order to reduce the number to partitions, we can apply the coalesce method with the desired number of partitions. Listing 3.4 shows the difference of using coalesce(1) on our data frame before writing to disk. We still get a directory, but there is a single CSV file inside of it. Mission accomplished!

Listing 3.4. Writing our results under a single partition

results.coalesce(1).write.csv('./results_single_partition.csv')

$ ls -l

# [...]

# -rw-r--r--@   1 jonathan_rioux  247087069  724726 Jul 30 17:30 1342-0.txt

# drwxr-xr-x  404 jonathan_rioux  247087069   12928 Aug  4 13:31 results.csv

# drwxr-xr-x    6 jonathan_rioux  247087069     192 Aug  4 13:43 results_single_partition.csv

$ ls -l results_single_partition.csv/

# [...]

# -rw-r--r--  1 jonathan_rioux  247087069      0 Aug  4 13:43 _SUCCESS

# -rw-r--r--  1 jonathan_rioux  247087069  70993 Aug  4 13:43 part-00000-f8c4c13e-a4ee-4900-ac76-de3d56e5f091-c000.csv

Note

You might have realized that we’re not ordering the file before writing it. Since our data here is pretty small, we could have written the words by decreasing order of frequency. If you have a very large data set, this operation will be quite expensive. Furthermore, since reading is a potentially distributed operation, what guarantees that it’ll get read the exact same way? Never assume that your data frame will keep the same ordering of records unless you explicitly ask via  orderBy().

Our workflow has been pretty interactive so far. We write one or two lines of text before showing the result to the terminal. As we get more and more confident with operating on the data frame’s structure, those shows will become fewer.

Now that we’ve performed all the necessary steps interactively, let’s look in putting our program in a single file and looking at refactoring opportunities.

3.4  PUTTING IT ALL TOGETHER: COUNTING

Interactive development is fantastic for rapid iteration of our code. When developing programs, it’s great to experiment and validate our thoughts through rapid code inputs into a shell. When the experimentation is over, it’s good to bring our program into a cohesive body of code.

The pyspark shell allows to go back in history using the directional arrows of your keyboard, just like a regular python REPL. To make things a bit easier, I am providing the step by step program in listing 3.5. This section is dedicated to streamline and make our code terser and more readable.

Listing 3.5. Our first PySpark program, dubbed "Counting Jane Austen"

from pyspark.sql.functions import col, explode, lower, regexp_extract, split

book = spark.read.text("./data/ch02/1342-0.txt")

lines = book.select(split(book.value, " ").alias("line"))

words = lines.select(explode(col("line")).alias("word"))

words_lower = words.select(lower(col("word")).alias("word"))

words_clean = words_lower.select(

regexp_extract(col("word"), "[a-z']*", 0).alias("word")

)

words_nonull = words_clean.where(col("word") != "")

results = words_nonull.groupby(col("word")).count()

results.orderBy("count", ascending=False).show(10)

results.coalesce(1).write.csv("./results_single_partition.csv")

This program runs perfectly id you paste its entirety in the pyspark shell. With everything in the same file, we can look at making our code friendlier and easier for future you to come back at it.

3.4.1  Simplifying your dependencies with PySpark’s import conventions

This program uses five distinct functions from the pyspark.sql.functions modules. We should probably replace this with a qualified import, which is Python’s way to import a module by assigning a keyword to it. While there is no hard rule, the common wisdom is to use F to refer to PySpark’s functions. Listing 3.6 shows the before and after.

Listing 3.6. Simplifying our PySpark functions import

# Before

from pyspark.sql.functions import col, explode, lower, regexp_extract, split

# After

import pyspark.sql.functions as F

Since col, explode, lower, regexp_extract and split are all in the pyspark.sql.functions, we can import the whole module. Since the new import statement imports the entirety of the pyspark.sql.functions module, we assign the keyword (or key-letter) F. The PySpark community seems to have implicitly settled on using F for pyspark.sql.functions and I encourage you to do the same. It’ll make your programs consistent, and since many functions in the module share their name with Pandas or Python built-in functions, you’ll avoid name clashes.

Warning

It can be very tempting to do a start import like  from pyspark.sql.functions import *. Do not fall into that trap! It’ll make it hard for your readers which functions comes from PySpark and which comes from regular Python. In Chapter 8, when we’ll use user defined functions (UDF), this separation will become even more important. Good coding hygiene rules!

That was easy enough. Let’s look at how we can simplify our program flow by using one of my favourite aspect of PySpark, its chaining abilities.

3.4.2  Simplifying our program via method chaining

If we look at the transformation methods we applied on our data frames (select(), where(), groupBy() and count()), they all have something in common: they take a structure as a parameter — the data frame or GroupedData in the case of count() — and return a structure. There is no concept of in-place modification in PySpark: all transformations can be seen as a pipe that ingests a structure and returns a modified structure. This section will look at what is probably my favourite aspect of PySpark: method chaining.

Our program uses intermediate variables quite a lot: everytime we perform a transformation, we assigned the result to a new variable. This is very useful when using the shell as we keep state of our transformation and can peek at our work at the end of every step. On the other hand, once our program works, this multiplication of variables is not as useful and can clutter our program visually.

In PySpark, every transformation returns an object, which is why we need to assign a variable to the result. This means that PySpark doesn’t perform modifications in place. For instance, the following code block by itself wouldn’t do anything because we don’t assign the result to a variable or perform an action to display or save our results.

results.orderBy("word").count()

We can avoid intermediate variables by chaining the results of one method to the next. Since each transformation returns a data frame (or a GroupedData, when we perform the groupby() method, we can directly append the next method without assigning the result to a variable. This means that we can eschew all but one variable assignment. The code in listing 3.7 shows the before and after. Note that we also added the F prefix to our functions, to respect the import convention we outlined in 3.4.1.

Listing 3.7. Removing intermediate variables by chaining transformation methods

# Before

book = spark.read.text("./data/ch02/1342-0.txt")

lines = book.select(split(book.value, " ").alias("line"))

words = lines.select(explode(col("line")).alias("word"))

words_lower = words.select(lower(col("word")).alias("word"))

words_clean = words_lower.select(

regexp_extract(col("word"), "[a-z']*", 0).alias("word")

)

words_nonull = words_clean.where(col("word") != "")

results = words_nonull.groupby("word").count()

# After

results = (

spark.read.text("./data/ch02/1342-0.txt")

.select(F.split(F.col("value"), " ").alias("line"))

.select(F.explode(F.col("line")).alias("word"))

.select(F.lower(F.col("word")).alias("word"))

.select(F.regexp_extract(F.col("word"), "[a-z']*", 0).alias("word"))

.where(F.col("word") != "")

.groupby("word")

.count()

)

It’s like night and day: the "after" is much terser and readable, and we’re able to easily follow the list of steps. Visually, we can also see the difference in Figure 3.4.

Figure 3.4. Method chaining eliminates the need for intermediate variables

I am not saying that intermediate variables are absolutely evil and to be avoided. They can hinder your code readability, so you have to make sure they serve a purpose. A lot of burgeoning PySpark developers take the habit of always writing on top of the same variable. If you see yourself doing something like listing 3.8, chain your methods like in listing 3.7. You’ll get the same result, and prettier code.

Listing 3.8. If you write over the same variable over and over again, consider chaining your methods.

# Don't do that

df = spark.read.text("./data/ch02/1342-0.txt")

df = df.select(F.split(F.col("value"), " ").alias("line"))

df = ...

Make your life easier by using Python’s parentheses

If you look at the "after" code of listing 3.7, you’ll notice that I start my right side of the equal sign with an opening parenthesis (spark = ( […]). This is a trick I use when I need to chain methods in Python. If you don’t wrap your result into a pair of parentheses, you’ll need to add a \ character at the end of each line, which adds visual noise to your program. PySpark code is especially prone to line breaks when you use method chaining.

results = spark\

.read.text('./data/ch02/1342-0.txt')\

...

As a lazy alternative, I am a big fan of using Black as a Python code formatting tool (black.readthedocs.io/). It removes a lot of the guesswork of having your code logically laid-out and consistent. Since we read code more than we write code, readability matters.

Since we are performing two actions on results (displaying the top 10 words on the screen and writing the data frame to a csv file), we have to use a variable. If you only have 1 action to perform on your data frame, you can channel your inner code golfer [5] by not using any variable name. Most of the time, I prefer lumping my transformations together and keep the action visually separate, like we are doing now.

Our program is looking much more polished now. The last step will be to add the PySpark’s plumbing to prepare it for batch mode.

[5] writing a program using the lowest possible number of characters.

3.5  YOUR FIRST NON-INTERACTIVE PROGRAM: USING SPARK-SUBMIT

When we launched the pyspark shell, we saw that the spark variable was mapped to our SparkSession entry point, already configured for interactive work. When using batch submit, this isn’t the case. In this section, I teach how to create your own entry point and submit the code in batch mode. You will then be able to submit this program (and any properly coded PySpark program).

Before entering in the how?, let’s see what happens if we submit our program as is. In listing 3.9 , we can see that PySpark replies immediately with a NameError, saying that spark isn’t defined.

Listing 3.9. Launching a PySpark program without spark defined.

$ spark-submit word_count.py

[...]

Traceback (most recent call last):

File "/Users/jonathan_rioux/Dropbox/PySparkBook/code/Ch02/word_count.py", line 3, in <module>

results = (spark

NameError: name 'spark' is not defined

[...]

Unlike the pyspark command, Spark provides a single launcher for its programs in batch mode, called spark-submit. The simplest way to submit a program is to provide the program name as the first parameter. As our programs grows in complexity, I will teach through Part 2 and 3 how to augment the spark-submit with other parameters.

3.5.1  Creating your own SparkSession

In Chapter 1, we learned that our main point of action is through an entry point which in our case is a SparkSession object. This section covers how to create a simple bare-bone entry point so our program can run smoothly.

PySpark provides a builder pattern using the object SparkSession.builder. For those familiar with object-oriented programming, a builder pattern provides a set of methods to create a highly configurable object without having multiple constructors. In this chapter, we will only look at the happiest case, but the SparkSession builder pattern will become increasingly useful in Part 3 as we look into performance tuning and adding dependencies to our jobs.

In listing 3.10 , we start the builder pattern, and then then chain a configuration parameter which defined the application name. This isn’t absolutely necessary, but when monitoring your jobs (see Chapter 9), having a unique and well thought-out job name will make it easier to know what’s what. We finish the builder pattern with the .getOrCreate() method to create our SparkSession.

Note

You can’t have two  SparkSession objects in your program working at the same time. This is why the getOrCreate() method is called like this. If you were to create a new entry point in the pyspark shell, you’d get all kinds of funny errors. By using the getOrCreate() method, your program will work both in interactive and batch mode.

Listing 3.10. Creating our own simple SparkSession

from pyspark.sql import SparkSession

spark = (SparkSession.builder

.appName("Counting word occurences from a book.")

.getOrCreate())

3.6  USING SPARK-SUBMIT TO LAUNCH YOUR PROGRAM IN BATCH MODE

We saw at the beginning of 3.5 how to submit a program using spark-submit. Let’s try again, in listing 3.11, with our properly configured entry point. The full code is available on the book’s repository, under code/Ch02/word_count_submit.py.

Listing 3.11. Submitting our job for real this time

$ spark-submit ./code/Ch02/word_count_submit.py

# [...]

# +----+-----+

# |word|count|

# +----+-----+

# | the| 4480|

# |  to| 4218|

# |  of| 3711|

# | and| 3504|

# | her| 2199|

# |   a| 1982|

# |  in| 1909|

# | was| 1838|

# |   i| 1749|

# | she| 1668|

# +----+-----+

# only showing top 10 rows

# [...]

Tip

You get a deluge of "INFO" messages? Don’t forget that you have control over this: use  spark.sparkContext.setLogLevel("WARN") right after your spark definition. If your local configuration has INFO as a default, you’ll still get a slew of messages until it catches this line, but it won’t obscure your results.

With this, we’re done! Our program successfully ingests the book, transforms it into a cleaned list of word frequencies and then exports it two ways: as a top-10 list on the screen and as a CSV file.

If we look at our process, we applied one transformation interactively at the time, show()-ing the process after each one. This will often be your modus operandi when working with a new data file. Once you’re confident about a block of code, you can remove the intermediate variables. PySpark gives you out of the box a productive environment to explore large data sets interactively and provides an expressive and terse vocabulary to manipulate data. It’s also easy to go from interactive development to batch deployment: you just have to define your SparkSession and you’re good to go.

3.7  WHAT DIDN’T HAPPEN IN THIS CHAPTER

Chapter 2 and 3 were pretty dense with information. We learned how to read text data, process it to answer and question, display the results on the screen and write them to a CSV file. On the other hand, there are many elements we left out on purpose. Let’s have a quick look at what we didn’t do in this Chapter.

With the exception of coalescing the data frame in order to write it into a single file, we didn’t care much for the distributing of the data. We saw in Chapter 1 that PySpark distributes data across multiple workers nodes, but our code didn’t pay much attention to this. Not having to constantly think about partitions, data locality and fault tolerance made our data discovery process much faster.

We didn’t spend much time configuring PySpark. Beside providing a name for our application, no additional configuration was inputted in our SparkSession. It’s not to say we’ll never touch this, but we can start with a bare-bone configuration and tweak as we go. Chapter 6 will expand into the subject.

Finally, we didn’t care much about the order of operations. We made a point to describe our transformations as logically as they appear to us, and we’re letting Spark’s optimize this into efficient processing steps. We could potentially re-order some and get the same output, but our program reads well, is easy to reason about and works well.

This echoes the statement I made in Chapter 1: PySpark is remarkable by not only what it provides, but also what it can abstract over. You can write your code as a sequence of transformations that will get you to your destination most of the time. For those cases where you want more finely-tuned performance or more control about the physical layout of your data, we’ll see in Part 3 that PySpark won’t hold you back.

3.8  SCALING UP OUR WORD FREQUENCY PROGRAM

That example wasn’t big data. I’ll be the first to say it.

Teaching big data processing has a catch 22. While I really want to show the power of PySpark to work with massive data sets, I don’t want you to purchase a cluster or rack up a massive cloud bill. It’s easier to show the ropes using a smaller set of data, knowing that we can scale using the same code.

Let’s take our word counting example: how can we scale this to a larger corpus of text? Let’s download more files from Project Gutenberg and place them in the same directory.

$ ls -l data/Ch02

[...]

-rw-r--r--@ 1 jonathan_rioux  247087069   173595 Aug  4 15:03 11-0.txt

-rw-r--r--@ 1 jonathan_rioux  247087069   724726 Jul 30 17:30 1342-0.txt

-rw-r--r--@ 1 jonathan_rioux  247087069   607788 Aug  4 15:03 1661-0.txt

-rw-r--r--@ 1 jonathan_rioux  247087069  1276201 Aug  4 15:03 2701-0.txt

-rw-r--r--@ 1 jonathan_rioux  247087069  1076254 Aug  4 15:03 30254-0.txt

-rw-r--r--@ 1 jonathan_rioux  247087069   450783 Aug  4 15:03 84-0.txt

While this is not enough to claim "We’re doing Big Data(tm)", it’ll be enough to explain the general concept. If you really want to scale, you can use Appendix C to provision a powerful cluster on the cloud, download more books or other text files and run the same program for a few dollars.

We will modify our word_count_submit.py in a very subtle way. Where we .read.text(), we’ll just change the path to account for all files in the directory. Listing 3.12, “Scaling our word count program” shows the before and after: we are just changing the 1342-0.txt to a *.txt, which is called a glob pattern. This will select all the .txt files in the directory.

Listing 3.12. Scaling our word count program

# Before

results = (spark

.read.text('./data/ch02/1342-0.txt')

# After

results = (spark

.read.text('./data/ch02/*.txt')

Note

You can also just pass the name of the directory if you want PySpark to ingest all the files within the directory.

The results of running the program over all the files in the directory are available in Listing 3.13, “Results of scaling our program to multiple files”.

Listing 3.13. Results of scaling our program to multiple files

$ spark-submit ./code/Ch02/word_count_submit.py

+----+-----+

|word|count|

+----+-----+

| the|38895|

| and|23919|

|  of|21199|

|  to|20526|

|   a|14464|

|   i|13973|

|  in|12777|

|that| 9623|

|  it| 9099|

| was| 8920|

+----+-----+

only showing top 10 rows

With this, you can confidently say that you are able to scale a simple data analysis program, using PySpark. You can use the general formula we’ve outlined here and modify some of the parameters and methods to fit your use-case. Chapter 3 will dig a little deeper into some interesting and common data transformations, building on what we’ve learned here.

3.9  SUMMARY

- You can group records using the groupby method, passing the column names you want to group against as a parameter. This returns a GroupedData object that waits for an aggregation method to return the results of a computation over the groups, such as the count() of records.

- PySpark’s repertoire of functions that operate on columns are located in pyspark.sql.functions. The unofficial but well respected convention is to qualify this import in your program using the F keyword.

- When writing a data frame to a file, PySpark will create a directory and put one file per partition. If you want to write a single file, use the coaslesce(1) method.

- In order to prepare your program to work in batch mode via spark-submit, you need to create a SparkSession. PySpark provides a builder pattern in the pyspark.sql module.

- If your program needs to scale across multiple files within the same directory, you can use a glob pattern to select many files at once. PySpark will collect them in a single data frame.

3.10  EXERCISES

For this exercise, you’ll need the word_count_submit.py program we worked on this Chapter. You can pick it from the book’s code repository (Code/Ch03/word_count_submit.py)

3.10.1  Exercise 3.1

a) Modifying the word_count_submit.py program, return the number of distinct words in Jane Austen’s Pride and Prejudice. (Hint, results contains 1 record for each unique word…)

b) (Challenge) Wrap your program in a function that takes a file name as a parameter. It should return the number of distinct words.

3.10.2  Exercise 3.2

Taking word_count_submit.py, modify the script to return a sample of 20 words that appear only once in Jane Austen’s Pride and Prejudice.

3.10.3  Exercise 3.3

a) Using the substr function (refer to PySpark’s API or the pyspark shell help if needed), return the top 5 most popular first letters (keep only the first letter of each word).

b) Compute the number of words starting with a consonant or a vowel. (Hint: the isin() function might be useful)

3.10.4  Exercise 3.4

Let’s say you want to get both the count() and sum() of a GroupedData object. Why doesn’t this code work? Map the inputs and outputs of each method.

my_data_frame.groupby("my_column").count().sum()

Multiple aggregate function application will be covered in Chapter 4.

4

ANALYZING TABULAR DATA WITH PYSPARK.SQL

This chapter covers:

- Reading delimited data into a PySpark data frame

- Understanding how PySpark represents tabular data in a data frame

- Ingesting and exploring tabular or relational data

- Selecting, manipulating, renaming and deleting columns in a data frame

- Summarizing data frames for quick exploration

So far, in chapters 2 and 3, we’ve dealt with textual data, which is unstructured. Through a chain of transformations, we extracted some information to get the most common words in the text. This chapter will go a little deeper into data manipulation using structured data, which is data that follow a set format. More specifically, we will work with tabular data, which follows the classical rows and columns layout. Just like the two previous chapters, we’ll take a data set and answer a simple question by exploring and processing the data.

We’ll use some public Canadian television schedule data to identify and measure the proportion of commercials over the total programming. The data used is typical of what you see from mainstream relational databases. This chapter and the next builds heavily on chapter 2 and 3, and add additional methods and information to use the data frame as a tabular data container. We perform some data exploration, assembly, and cleaning, using the pyspark.sql data manipulation module, and we finish by answering some questions hidden in our data set. The exercises will at the end of the chapter give you the opportunity to craft your own data manipulation code.

The initialization part of our Spark program (relevant imports and SparkSession creation) is provided in listing 4.1.

Listing 4.1. Relevant imports and scaffolding for this chapter

import os

import numpy as np

from pyspark.sql import SparkSession

import pyspark.sql.functions as F

spark = SparkSession.builder.getOrCreate()

4.1  WHAT IS TABULAR DATA?

Tabular data is data that we can typically represent in a 2-dimensional table. You have rows and columns containing a single (or simple) value. A good example would be your grocery list: you may have one column for the item you wish to purchase, one for the quantity, and one for the expected price. Figure 4.1 provides an example of a small grocery list. We have the three columns mentioned, as well as four rows, each representing an entry in our grocery list.

Figure 4.1. My grocery list represented as tabular data. Each row represents an item, and each column represents an attribute.

The easiest analogy we can make for tabular data is the spreadsheet format: the interface provides you with a large number of rows and columns where you can input and perform computation on data. SQL databases, even if they use a different vocabulary, can also be thought of as tables made up of rows and columns. Tabular data is an extremely common data format, and because it’s so popular and easy to reason about, it makes for a perfect first dive into PySpark’s data manipulation API.

PySpark’s data frame structure maps very naturally to tabular data. In chapter 2, I explain that PySpark operates either on the whole data frame structure (via methods such as select() and groupby()) of on Column objects (for instance when using a function like split()). The data frame is column-major, so its API focuses on manipulating the columns to transform the data. Because of this, we can simplify how we reason about data transformations by thinking about what operations to do and which columns will be impacted.

Note

The resilient distributed dataset, briefly introduced in chapter 1, is a good example of a structure that is  row-major. Instead of thinking about columns, you are thinking about items with attributes in which you apply functions. It’s an alternative way of thinking about your data, and chapter 8 contains a lot more information about where it can be useful.

4.1.1  How does PySpark represent tabular data?

In chapter 2, our data frame always contained a single column, up until the very end where we counted the occurrence of each word. In other words, we took unstructured data (a body of text), performed some transformations, and created a two-column table containing the information we wanted. Tabular data is in a way an extension of this, where we have more than one column to work with.

Let’s take my very healthy grocery list as an example, and load it into PySpark. To make things simple, we’ll encode our grocery list into a list of lists. PySpark has multiple ways to import tabular data, but the two most popular are the list of lists and the pandas data frame. I cover briefly how to work with Pandas in chapter 8 and appendix D contains more information about how to use pandas. Considering the size of our grocery list, it would be a little overkill to import a library just for loading 4 records, so I kept it in a list of lists.

Listing 4.2. Creating a data frame out of our grocery list

my_grocery_list = [

["Banana", 2, 1.74],

["Apple", 4, 2.04],

["Carrot", 1, 1.09],

["Cake", 1, 10.99],

]  ❶

df_grocery_list = spark.createDataFrame(my_grocery_list, ["Item", "Quantity", "Price"])

df_grocery_list.printSchema()

# root

#  |-- Item: string (nullable = true)   ❷

#  |-- Quantity: long (nullable = true) ❷

#  |-- Price: double (nullable = true)  ❷

❶

My grocery list is encoded in a list of lists.

❷

PySpark automatically inferred the type of each field from the information Python had about each value.

We can easily create a data frame from data in our program with the spark.createDataFrame function, as listing 4.2 shows. Our first parameter is the data itself. You can either provide a list of items (here a list of lists), a pandas data frame or a Resilient Distributed Dataset, which I cover in chapter 9. The second parameter is the schema of the data frame. chapter 6 covers the automatic and manual schema definitions in greater depth. In the meantime, passing a list of column names will make PySpark happy, while it infers the types (string, long, and double, respectively) of our columns. Visually, the data frame will look like figure 4.2, although much more simplified. The master node knows about the structure of the data frame, but the actual data is represented on the worker nodes. Each column maps to data stored somewhere on our cluster, managed by PySpark. We operate on the abstract structure and let the master delegate the work efficiently.

Figure 4.2. Each column of our data frame maps to some place on our worker nodes.

PySpark gladly represented our tabular data using our column definitions. This means that all the functions we learned so far apply to our tabular data. By having one flexible structure for many data representations — we’ve done text and tabular so far — PySpark makes it easy to move from one domain to another. It removes the need to learn yet another set of functions and a whole new abstraction for our data.

4.2  PYSPARK FOR ANALYZING AND PROCESSING TABULAR DATA

My grocery list was fun, but the potential for analysis work is pretty limited. We’ll get our hands on a larger data set, explore it, and ask a few introductory questions that we might find interesting. This process is called exploratory data analysis (or EDA) and is usually the first step data analysts and scientists undertake when placed in front of new data. Our goal is to get familiar with the data discovery functions and methods as well as performing some basic data assembly. Being familiar with those steps will remove the awkwardness of working with data you won’t see transforming before your eyes. Until we can process visually millions of records per second, this chapter will show you a blueprint you can re-use when facing new data frames.

Graphical exploratory data analysis?

A lot of the EDA work you’ll see in the wild incorporates charts and/or tables. Does it mean that PySpark has the option to do the same?

We saw in chapter 2 how to pretty print a data frame so we can view the content at a glance. This still applies for summarizing information and displaying it on the screen. If you want to export the table in an easy to process format (to incorporate in a report, for instance), you can use spark.write.csv, making sure you coalesce the data frame in a single file. (See chapter 3 for a refresher on coalesce().) By its very nature, table summaries won’t be very huge so you won’t risk running out of memory.

PySpark doesn’t provide any charting capabilities and doesn’t play with other charting libraries (like matplotlib, seaborn, altair, or plot.ly). Taking a step back, it makes a lot of sense: PySpark distributes your data over many computers. It doesn’t make much sense to distribute a chart creation. The usual solution will be to transform your data using PySpark, use the toPandas() method to transform your PySpark data frame into a pandas data frame, and then use your favourite charting library. When using charts, I provide the code I used to generate them, but instead of explaining the process each time, I provide explanations — as well as a primer in using pandas with PySpark — in Appendix D.

For this exercise, we’ll use some open data from the Government of Canada, more specifically the CRTC (Canadian Radio-television and Telecommunications Commission). Every broadcaster is mandated to provide a complete log of the programs, commercials and all, showcased to the Canadian public. This gives us a lot of potential questions to answer, but we’ll select one specific one: what are the channels with the most and least proportion of commercials?

You can download the file on the Canada Open Data portal (open.canada.ca/data/en/dataset/800106c1-0b08-401e-8be2-ac45d62e662e), selecting the BroadcastLogs_2018_Q3_M8 file. The file is a whopping 994MB to download, which might be a little too large for some people. The book’s repository contains a sample of the data under the data/Ch04 directory, which you can use in lieu of the original file. You’ll also need to download the "Data Dictionary" in DOC form, as well as the "Reference Tables" zip file, unzipping them into a "ReferenceTables" directory in data/Ch04. Once again, the examples are assuming that the data is downloaded under data/Ch04 and that PySpark is launched from src/Ch04/.

4.3  READING DELIMITED DATA IN PYSPARK

This section is dedicated to ingesting delimited data in a PySpark data frame, so we can start manipulating it. I will cover how to use a specialized reader object for delimited data, the most common parameters to set, and how to identify frequent patterns when looking at tabular data.

Delimited data is a very common, popular, and tricky way of sharing data. In a nutshell, the data is sitting verbatim in a file, separated by two types of delimiters. A visual sample of delimited data is depicted in figure 4.3.

1. The first one is a row delimiter. The row delimiter splits the file in logical records. There is one and only one record between delimiters.

2. The second one is a field delimiter. Each record is made up of an identical number of fields and the field delimiter tells where one field starts and ends.

Figure 4.3. A sample of our data, highlighting the field delimiter (|) and row delimiter (\n)

The newline character (\n when depicted explicitly) is the de-facto record delimiter. It naturally breaks down the file into visual lines, where one record starts at the beginning of the line and ends, well, at the end. The comma character , is the most frequent field delimiter. It’s so prevalent that most people call delimited data files "CSV", which stands for "Comma Separated Values".

CSV files are easy to produce and have a loose set of rules to follow to be considered usable. Because of this, PySpark provides a whopping 25 optional parameters when ingesting a CSV. Compare this to the two for reading text data. In listing 4.3 , I use three configuration parameters. This is enough to parse our data into a data frame.

Listing 4.3. Reading our broadcasting information

DIRECTORY = "../../data/Ch04"

logs = spark.read.csv(

os.path.join(DIRECTORY, "BroadcastLogs_2018_Q3_M8.CSV"),  ❶

sep="|",  ❷

header=True,  ❸

inferSchema=True,  ❹

)

❶

We specify the file path where our data resides first

❷

Our file uses a vertical bar, so we pass  | as a parameter to sep

❸

header takes a boolean. When true, the first row of your file is parsed as the column names.

❹

inferSchema takes a boolean as well. When true, it’ll pre-parse the data to infer the type of the column.

The next section expands on the most important parameters when reading CSV data and provides more detailed explanations behind the code in listing 4.3.

4.3.1  Customizing the SparkReader object to read CSV data files

This section focuses on how we can specialize the SparkReader object to read delimited data and what are the most popular configuration parameters to accommodate the various declinations of CSV.

Reading delimited data can be a dicey business. Because of how flexible and human editable the format is, a CSV reader needs to provide many options to cover the many use-cases possible. There is also a risk that the file is malformed, in which case you will need to treat it as text and gingerly infer the fields manually. I will stay on the happy path and cover the most popular scenario: a single file, properly delimited.

THE PATH TO THE FILE YOU WANT TO READ AS THE ONLY MANDATORY PARAMETER

Just like when reading text, the only truly mandatory parameter is the path, which contains the file or files path. As we saw in chapter 2, you can use a glob pattern to read multiple files inside a given directory, as long as they have the same structure. You can also explicitly pass a list of file paths if you want specific files to be read.

PASSING AN EXPLICIT FIELD DELIMITER WITH THE SEP PARAMETER

The most common variation you’ll encounter when ingesting and producing CSV file is selecting the right delimiter. The comma is the most popular, but it suffers from being a popular character in text, which means you need a way to differentiate which commas are part of the text and which oner are delimiters. Our file use the vertical bar character, an apt choice: it’s easily reachable on the keyboard yet infrequent in text.

Note

In French, we use the comma for separating numbers between their integral part and their decimal one (e.g.  1.02 → 1,02). This is pretty awful when in a CSV file, so most French CSV will use the semicolon (;) as a field delimiter. This is one more example of why you need to be vigilant when using CSV.

When reading CSV data, PySpark will default to using the comma character as a field delimiter. You can set the optional parameter sep (for separator) to the single character you want to use as a field delimiter.

QUOTING TEXT TO AVOID MISTAKING A CHARACTER FOR A DELIMITER

When working with CSV that use the comma as a delimiter, it’s common practice to quote the text fields to make sure any comma in the text is not mistaken as a field separator. The CSV reader object provides an optional quote parameter that defaults to the double-quote character ". Since I am not passing an explicit value to quote, we are keeping the default value. This way, we can have a field with the value "Three | Trois", whereas we would consider this to be two fields without the quotation character.

If we don’t want to use any character as a quote, we need to pass explicitly the empty string to quote.

USING THE FIRST ROW AS THE COLUMN NAMES

The header optional parameter takes a Boolean flag. If set to true, it’ll use the first row of your file (or files, if you’re ingesting many) and use it to set your column names.

You also can pass a list of strings as the schema optional parameter if you wish to explicitly name your columns. If you don’t fill any of those two, your data frame will have _c* for column names, where the star is replaced with increasing integers (_c0, _c1, …).

INFERRING WHAT A COLUMN TYPE WHILE READING THE DATA

PySpark has a schema discovering capacity. You turn it on by setting inferSchema to True (by default, this is turned off). This optional parameter forces PySpark to go over the ingested data: one time to set the types of each column, one time to ingest the data. This makes the ingestion quite a bit longer but avoids us to write the schema by hand (I go down to this level of detail in chapter 6). Let the machine do the work!

We are lucky enough that the Government of Canada is a good steward of data, and provides us with clean, properly formatted files. In the wild, malformed CSV files are legion and you will run into some errors when trying to ingest some of them. Furthermore, if your data is large, you often won’t get the chance to inspect each row one by one to fix mistakes. Chapter 9 covers some strategies to ease the pain and also shows you some ways to share your data with the schema included.

Our data frame schema, displayed on listing 4.4, is coherent with the documentation we’ve downloaded. The column names are properly displayed and the types make sense. That’s plenty enough to get started with some exploration.

Listing 4.4. The schema of our logs data frame

logs.printSchema()

# root

#  |-- BroadcastLogID: integer (nullable = true)

#  |-- LogServiceID: integer (nullable = true)

#  |-- LogDate: timestamp (nullable = true)

#  |-- SequenceNO: integer (nullable = true)

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  |-- CategoryID: integer (nullable = true)

#  |-- ClosedCaptionID: integer (nullable = true)

#  |-- CountryOfOriginID: integer (nullable = true)

#  |-- DubDramaCreditID: integer (nullable = true)

#  |-- EthnicProgramID: integer (nullable = true)

#  |-- ProductionSourceID: integer (nullable = true)

#  |-- ProgramClassID: integer (nullable = true)

#  |-- FilmClassificationID: integer (nullable = true)

#  |-- ExhibitionID: integer (nullable = true)

#  |-- Duration: string (nullable = true)

#  |-- EndTime: string (nullable = true)

#  |-- LogEntryDate: timestamp (nullable = true)

#  |-- ProductionNO: string (nullable = true)

#  |-- ProgramTitle: string (nullable = true)

#  |-- StartTime: string (nullable = true)

#  |-- Subtitle: string (nullable = true)

#  |-- NetworkAffiliationID: integer (nullable = true)

#  |-- SpecialAttentionID: integer (nullable = true)

#  |-- BroadcastOriginPointID: integer (nullable = true)

#  |-- CompositionID: integer (nullable = true)

#  |-- Producer1: string (nullable = true)

#  |-- Producer2: string (nullable = true)

#  |-- Language1: integer (nullable = true)

#  |-- Language2: integer (nullable = true)

Exercise 4.1

Lets take the following file, called sample.csv.

Item,Quantity,Price

$Banana, organic$,1,0.99

Pear,7,1.24

$Cake, chocolate$,1,14.50

Complete the following code to ingest the file successfully.

sample = spark.read.csv([...],

sep=[...],

header=[...],

quote=[...],

inferSchema=[...]

)

Exercise 4.2

Re-read the data in a logs_raw data frame, taking inspiration from the code in listing 4.3, this time without passing any optional parameters. Print the first 5 rows of data, as well as the schema. What are the differences in terms of data and schema between logs and logs_raw?

4.3.2  Exploring the shape of our data universe

When working with tabular data, especially if it comes from a SQL data warehouse, you’ll often find that the data set is split between tables. In our case, our logs table contains a majority of fields suffixed by ID; those IDs are listed in other tables and we have to link them to get the legend of those IDs. This section introduces briefly what a star schema is, why they are so frequently encountered, and how we can represent them visually to work with them.

Our data universe (the set of tables we are working with) follows a very common pattern in relational databases: a center table containing a bunch of IDs (or keys) and some ancillary tables around containing a legend between each key and its value. This is called a star schema since it visually looks like a star. Star schemas are common in the relational database world because of normalization, a process used to avoid duplicating pieces of data and improve data integrity. Data normalization is illustrated in figure 4.4, where our centre table logs contain IDs that maps to the auxiliary tables around called link tables. In the case of the CD_Category link table, it contains many fields (such as Category_CD and English_description) that are made available to logs when you link the two tables with the Category_ID key.

Figure 4.4. The logs table "ID" columns map to other tables, like the CD_category table which links the Category_ID field.

In Spark’s universe, we often prefer working with a single table instead of linking a multitude of tables to get the data. We call them denormalized tables, or colloquially fat tables. We will start by assessing the data available to us directly in the logs table before plumping our table, a topic I cover in chapter 5. By looking at the logs table, its content, and the data documentation, we will avoid linking tables that contain data with no real value for our analysis.

The right structure for the right work

Normalization, denormalization, what gives? Isn’t this a book about data analysis?

While this book isn’t about data architecture, it’s important to understand, at least a little bit, how data might be structured so we can work with it. Normalized data has many advantages when you’re working with relational information (such as our broadcast tables). Besides being easier to maintain, data normalization reduces the probability of getting anomalies or illogical records in your data.

When dealing with analytics, a single table containing all the data is best. However, having to link the data by hand can be tedious, especially when working with dozens or even hundreds of link tables. Fortunately, data warehouses don’t change their structure very often. If you’re faced with a complex star schema one day, befriend one of the database managers. There is a very good chance that they’ll provide you with the information to denormalize the tables, most often in SQL, and chapter 7 will show you how you can adapt the code into PySpark with a minimum of efforts.

4.4  THE BASICS OF DATA MANIPULATION: DIAGNOSING OUR CENTRE TABLE

It is common practice to explore and summarize the data when you first get acquainted with it. It’s just like a first date with your data: you want a good overview, not agonize on the details. [6] This section shows the most common manipulations done on a data frame in greater detail. I show how you can select, delete, rename, re-order, and create columns so you can customize how a data frame is shown. I also cover summarizing a data frame, so you can have a quick diagnostic overview of the data inside your structure. No flowers required.

4.4.1  Knowing what we want: selecting columns

So far, we’ve learned that typing our data frame variable into the shell prints the structure of the data frame, not the data, unless you’re using eagerly evaluated Spark (referenced in chapter 2). We can also use the show() command to display a few records for exploration. I won’t show the results, but if you try it, you’ll see that the table-esque output is garbled, because we are showing too many columns at once. Time to select() our way to sanity.

At its simplest, select() can take one or more column objects — or strings representing column names — and will return a data frame containing only the listed columns. This way, we can keep our exploration tidy and check a few columns at the time. An example is displayed in listing 4.5.

Listing 4.5. Selecting 5 rows of the first 3 columns of our data frame

logs.select(*logs.columns[:3]).show(5, False)

# +--------------+------------+-------------------+

# |BroadcastLogID|LogServiceID|LogDate            |

# +--------------+------------+-------------------+

# |1196192316    |3157        |2018-08-01 00:00:00|

# |1196192317    |3157        |2018-08-01 00:00:00|

# |1196192318    |3157        |2018-08-01 00:00:00|

# |1196192319    |3157        |2018-08-01 00:00:00|

# |1196192320    |3157        |2018-08-01 00:00:00|

# +--------------+------------+-------------------+

# only showing top 5 rows

In chapter 2, you learn that .show(5, False) shows 5 rows without truncating their representation so we can show the whole content. The .select() statement is where the magic happens.

Each data frame created keeps a list of column names in its columns attribute, and we can slice it to pick a subset of columns to see. We’re using the implicit string to column conversion PySpark provides to avoid some boilerplate.

select(), takes a single parameter, named *cols. This star is used in Python for unpacking collections, or in our case to illustrate that the function takes a variable number of parameters that will be collected under the cols variable. From a PySpark perspective, the four statements in listing 4.6 are interpreted the same. Note how prefixing the list with a star removed the container so each element becomes a parameter of the function. If this looks a little confusing to you, fear not! Appendix D will provide you with a good overview of its usage.

Listing 4.6. Four ways to select colums in PySpark, all equivalent in term of results

# Using the string to column conversion

logs.select("BroadCastLogID", "LogServiceID", "LogDate")

logs.select(*["BroadCastLogID", "LogServiceID", "LogDate"])

# Passing the column object explicitly

logs.select(F.col("BroadCastLogID"), F.col("LogServiceID"), F.col("LogDate"))

logs.select(*[F.col("BroadCastLogID"), F.col("LogServiceID"), F.col("LogDate")])

When explicitly selecting a few columns, you don’t have to wrap them into a list. If you’re already working on a list of columns, you can unpack them with a star prefix. This argument unpacking pattern is worth remembering as many other data frame methods taking columns as input are using the same approach.

In the spirit of being clever (or lazy), let’s expand our selection code to see every column in groups of three. This will give us a sense of the content. Since logs.columns is a Python list, we can use a function on it without any problem. The code in listing 4.7 shows one of the ways we can do it.

Listing 4.7. Peeking at the data frame in chunks of 3 columns

column_split = np.array_split(np.array(logs.columns), len(logs.columns) // 3)  ❶

print(column_split)

# [array(['BroadcastLogID', 'LogServiceID', 'LogDate'], dtype='<U22'),

#  [...]

#  array(['Producer2', 'Language1', 'Language2'], dtype='<U22')]'

for x in column_split:

logs.select(*x).show(5, False)

# +--------------+------------+-------------------+

# |BroadcastLogID|LogServiceID|LogDate            |

# +--------------+------------+-------------------+

# |1196192316    |3157        |2018-08-01 00:00:00|

# |1196192317    |3157        |2018-08-01 00:00:00|

# |1196192318    |3157        |2018-08-01 00:00:00|

# |1196192319    |3157        |2018-08-01 00:00:00|

# |1196192320    |3157        |2018-08-01 00:00:00|

# +--------------+------------+-------------------+

# only showing top 5 rows

# ... and more tables of 3 columns

❶

The  split_array() function comes from the numpy package, imported as np at the beginning of this chapter.

Let’s take each line one at a time.

We start by splitting the logs.columns list into approximate groups of 3. To do so, we rely on a function from the numpy package called array_split(). The function takes an array and a number of desired sub-arrays N and returns a list of N sub-arrays. We wrap our list of columns logs.columns into an array via the np.array function and pass this as a first parameter. For the number of sub-arrays, we divide the number of columns by 3, using an integer division //.

Tip

To be perfectly honest, the call to  np.array can be eschewed since np.array_split() can work on lists. I am still using it since if you are using a static type checker, such as mypy, you’ll get a type error. Chapter 8 has a basic introduction to type checking in Python and Appendix D provides a little more guidance on how to type check your code.

The last part of listing 4.7 iterates over the list of sub-arrays, using select() so select the columns present inside each sub-array and show() to display them on the screen.

This example shows how easy it is to blend Python code with PySpark. On top of providing a trove of functions, the data frame API also exposes information, such as column names, into convenient Python structures. I won’t avoid using functionality from libraries when it makes sense, but like in listing 4.7, I’ll do my best to explain what it does and why we’re using it. Chapter 8 goes beyond on the subject of how you can further combine pure Python code in PySpark.

In this section, we used the select() method to rapidly peek at a sample of our data. Because of the width of our data frame, we split our columns into manageable sets of three to keep the output tidy on the screen. I use this pattern frequently to have a high-level view of what my data frame contains. Next, we do the opposite: specifying what we do not want to keep.

4.4.2  Keeping what we need: deleting columns

The other side of selecting columns is choosing what not to select. We could do the full trip with select(), carefully crafting our list of columns to keep only the one we want. Fortunately, PySpark also provides a shorter trip: just drop what you don’t want.

In our current data frame, let’s get rid of two columns in the spirit of tidying up. Hopefully, it will bring us joy.

- BroadCastLogID is the primary key of the table and will serve us no use in answering our questions.

- SequenceNo is a sequence number and won’t be useful either.

More will come off later when we start looking at the link tables. The code in listing 4.8 does the trick very simply.

Listing 4.8. Getting rid of columns using the drop() method

logs = logs.drop("BroadcastLogID", "SequenceNO")

# Testing if we effectively got rid of the columns

print("BroadcastLogID" in logs.columns)  # => False

print("SequenceNo" in logs.columns)  # => False

Just like select(), drop() takes a *cols and returns a data frame, this time excluding the columns passed as parameters. Just like every other method in PySpark, drop() returns a new data frame, so we overwrite our logs variable by assigning the result of our code.

Warning

Unlike  select(), where selecting a column that doesn’t exist will return a runtime error, dropping a non-existent column is a no-op. PySpark will just ignore the columns it doesn’t find. Careful with the spelling of your column names!

Depending on how many columns you want to preserve, select might be a neater way to keep just what you want. We can see drop() and select() as being two sides of the same coin: one drops what you specify, the other one keeps what you specify. We could reproduce listing 4.8 with a select() method, and listing 4.9 does just that.

Listing 4.9. Getting rid of columns, select-style

logs = logs.select(

*[x for x in logs.columns if x not in ["BroadcastLogID", "SequenceNO"]]

)

Advanced topic: An unfortunate inconsistency

In theory, you can also select() columns with a list without unpacking it. This code will work as expected.

logs = logs.select(

[x for x in logs.columns if x not in ["BroadcastLogID", "SequenceNO"]]

)

This is not the case for drop(), where you need to explicitly unpack.

logs.drop(logs.columns[:])

# TypeError: col should be a string or a Column

logs.drop(*logs.columns[:])

# DataFrame[]

I’d rather unpack explicitly and avoid the cognitive load of remembering when it’s mandatory and when it’s optional.

You now know the most fundamental operations to perform on a data frame. You can select and drop columns, and with the flexibility of select() presented in chapters 2 and 3, you can apply functions on existing columns to transform them. The next section will cover how you can create new columns without having to rely on select(), simplifying your code, and improving its resiliency.

Exercise 4.3

Create a new data frame logs_clean that contains only the colunms that do not end with ID.

Exercise 4.4

What is the printed result of this code?

sample_frame.columns

# ['item', 'price', 'quantity', 'UPC']

print(sample_frame.drop('item', 'UPC', 'prices').columns)

1. ['item' 'UPC']

2. ['item', 'upc']

3. ['price', 'quantity']

4. ['price', 'quantity', 'UPC']

5. Raises an error

4.4.3  Creating what’s not there: new columns with withColumn()

Creating new columns is such a basic operation that it seems a little far-fetched to rely on select(). It also puts a lot of pressure on code readability: for instance using drop() makes it obvious we’re removing some columns. It would be nice to have something that signals we’re creating a new column. PySpark named this function withColumn().

Before going crazy with column creation, let’s take a simple example, build what we need iteratively and then move them to withColumn(). Let’s take the Duration column, containing the length of each program shown.

logs.select(F.col("Duration")).show(5)

# +----------------+

# |        Duration|

# +----------------+

# |02:00:00.0000000|

# |00:00:30.0000000|

# |00:00:15.0000000|

# |00:00:15.0000000|

# |00:00:15.0000000|

# +----------------+

# only showing top 5 rows

print(logs.select(F.col("Duration")).dtypes)  ❶

# [('Duration', 'string')]

❶

The  dtypes attribute of a data frame contains the name of the column and its type, wrapped in a tuple.

PySpark doesn’t have a default type for time without dates or duration, so it kept the column as a string. We verified the exact type via the dtypes attribute, which returns both the name and type of a data frame’s columns. A string is a safe and reasonable option, but this isn’t remarkably useful for our purpose. Thanks to our peeking, we can see that the string is formatted like HH:MM:SS.mmmmmm, where

- HH is the duration in hours

- MM is the duration in minutes

- SS is the duration in seconds

- mmmmmmm is the duration in milliseconds

I ignore the duration in milliseconds since I don’t think it’ll make a huge difference. In listing 4.10, we are extracting the three other sub-fields.

Listing 4.10. Extracting the hours, minutes and seconds from the Duration column

logs.select(

F.col("Duration"),  ❶

F.col("Duration").substr(1, 2).cast("int").alias("dur_hours"),  ❷

F.col("Duration").substr(4, 2).cast("int").alias("dur_minutes"),  ❸

F.col("Duration").substr(7, 2).cast("int").alias("dur_seconds"),  ❹

).distinct().show(  ❺

5

)

# +----------------+---------+-----------+-----------+

# |        Duration|dur_hours|dur_minutes|dur_seconds|

# +----------------+---------+-----------+-----------+

# |00:10:06.0000000|        0|         10|          6|

# |00:10:37.0000000|        0|         10|         37|

# |00:04:52.0000000|        0|          4|         52|

# |00:26:41.0000000|        0|         26|         41|

# |00:08:18.0000000|        0|          8|         18|

# +----------------+---------+-----------+-----------+

# only showing top 5 rows

❶

The original column, for sanity

❷

The first two characters are the hours

❸

The fourth and fifth characters are the minutes

❹

The seventh and eighth characters are the seconds

❺

To avoid seeing identical rows, I’ve added a  distinct() to the results

The substr() method takes two parameters. The first gives the position of where the sub-string starts, the first character being 1, not 0 like in Python. The second gives the length of the sub-string we want to extract in number of characters. Following the application of substr(), we then cast the result as int (integer) using the cast method so we can treat them as integers. Casting is a very common and important operation and is covered in more detail in chapter 6. We finally provided an alias for each column so we know easily which one is which.

I used before show() the distinct() method, which de-dupes the data frame. This is explained further in chapter 5. I added distinct() to avoid seeing identical occurrences that would provide no additional information when displayed.

I think that we’re in good shape! Let’s merge all those values into a single field: the duration of the program in seconds. PySpark can perform arithmetic with column objects using the same operators as Python, so this will be a breeze! The code in listing 4.11 takes the code forming the additional columns in listing 4.10 and use it in the definition of a single column.

Listing 4.11. Creating a duration in second field from the Duration column

logs.select(

F.col("Duration"),

(

F.col("Duration").substr(1, 2).cast("int") * 60 * 60

+ F.col("Duration").substr(4, 2).cast("int") * 60

+ F.col("Duration").substr(7, 2).cast("int")

).alias("Duration_seconds"),

).distinct().show(5)

# +----------------+----------------+

# |        Duration|Duration_seconds|

# +----------------+----------------+

# |00:10:30.0000000|             630|

# |00:25:52.0000000|            1552|

# |00:28:08.0000000|            1688|

# |06:00:00.0000000|           21600|

# |00:32:08.0000000|            1928|

# +----------------+----------------+

# only showing top 5 rows

We kept the same definitions, removed the alias, and performed arithmetic directly on the columns. There are 60 seconds in a minute, and 60 * 60 seconds in an hour. PySpark respects operator precedence, so we don’t have to clobber our equation with parentheses. Overall, our code is quite easy to follow and we are ready to add our column to our data frame.

Instead of using select() on all the columns plus our new one, let’s use withColumn(). Applied to a data frame, it’ll return a data frame with the new column appended. Listing 4.12 takes our field and add it to our logs data frame. I also include a sample of the printSchema() so you can see the column added at the end.

Listing 4.12. Creating a new column with withColumn()

logs = logs.withColumn(

"Duration_seconds",

(

F.col("Duration").substr(1, 2).cast("int") * 60 * 60

+ F.col("Duration").substr(4, 2).cast("int") * 60

+ F.col("Duration").substr(7, 2).cast("int")

),

)

logs.printSchema()

# root

#  |-- LogServiceID: integer (nullable = true)

#  |-- LogDate: timestamp (nullable = true)

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  [... more columns]

#  |-- Language2: integer (nullable = true)

#  |-- Duration_seconds: integer (nullable = true)  ❶

❶

Our  Duration_seconds colums has been added at the end of our data frame.

Warning

If you’re creating a column  withColumn() and give it a name that already exists in your data frame, PySpark will happily overwrite the column. This is often very useful to keep the number of columns manageable, but make sure you are seeking this effect!

We can create columns using the same expression with select() and with withColumn(). Both approaches have their use. select() will be useful when you’re explicitly working with a few columns. When you need to create new ones without changing the rest of the data frame, prefer withColumn(). You’ll quickly get the intuition about which one is easiest to use when faced with the choice.

Figure 4.5. select() vs. withColumn(), visually. withColumn() keeps all the pre-existing columns without the need the specify them explicitly.

4.4.4  Tidying our data frame: renaming and re-ordering columns

This section covers how to make the order and name of the columns friendlier for you. It might seem a little vapid, but after a few hours of hammering code on a particularly tough piece of data, you’ll be happy to have this in your toolbox.

Renaming columns can be done with select() and alias, of course. We saw briefly in chapter 2 that PySpark provides you an easier way to do so. Enter withColumnRenamed()! In Listing 4.13, I use withColumnRenamed() to remove the capital letters of my newly created duration_seconds column.

Listing 4.13. Renaming one column at a type, the withColumnRenamed() way

logs = logs.withColumnRenamed("Duration_seconds", "duration_seconds")

logs.printSchema()

# root

#  |-- LogServiceID: integer (nullable = true)

#  |-- LogDate: timestamp (nullable = true)

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  [...]

#  |-- Language2: integer (nullable = true)

#  |-- Duration_seconds: integer (nullable = true)

I’m a huge fan of having column names without capital letters. I’m a lazy typist, and pressing Shift all the time really adds up folks! I could potentially use withColumnRenamed() with a for loop over all the columns to rename all my columns in my data frame. The PySpark developers thought about this and offered a better way to rename all the columns of your data frame in one fell swoop. This relies on a method, toDF(), that returns a new data frame with the new columns. Just like drop(), toDF() takes a *cols so we’ll need to unpack our column names if they’re in a list. The code in listing 4.14 shows how you can rename all your columns to lower case in a single line using that method.

Listing 4.14. Batch lower-casing your data frame using the toDF() method

logs.toDF(*[x.lower() for x in logs.columns]).printSchema()

# root

#  |-- logserviceid: integer (nullable = true)

#  |-- logdate: timestamp (nullable = true)

#  |-- audiencetargetageid: integer (nullable = true)

#  |-- audiencetargetethnicid: integer (nullable = true)

#  |-- categoryid: integer (nullable = true)

#  [...]

#  |-- language2: integer (nullable = true)

#  |-- duration_seconds: integer (nullable = true)

If you look at the line of code, you can see that I’m not assigning the resulting data frame. I wanted to showcase the functionality, but since we have ancillary tables with column names that match, I wanted to avoid the trouble of lower-casing every column in every table.

Our final step is re-ordering columns. Since re-ordering columns is equivalent to selecting columns in a different order, select() is the perfect method for the job. For instance, if we wanted to sort the columns alphabetically, we can use the sorted function on the list of our data frame columns, just like in listing 4.15.

Listing 4.15. Selecting our columns in alphabetical order using select()

logs.select(sorted(logs.columns)).printSchema()

# root

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  |-- BroadcastOriginPointID: integer (nullable = true)

#  |-- CategoryID: integer (nullable = true)

#  |-- ClosedCaptionID: integer (nullable = true)

#  |-- CompositionID: integer (nullable = true)

#  [...]

#  |-- Subtitle: string (nullable = true)

#  |-- duration_seconds: integer (nullable = true) ❶

❶

Remember that, in most programming languages, capital letters comes before lower-case ones.

4.4.5  Summarizing your data frame: describe() and summary()

When working with numerical data, looking at a long column of values isn’t very useful. We’re often more concerned about some key information, which may include count, mean, standard deviation, minimum, and maximum. The describe() method does exactly that. By default, it will show those statistics on all numerical and string columns, which will overflow our screen and be unreadable since we have many columns. I display the columns description one-by-one by iterating over the list of columns and showing the output of describe() in listing 4.16 . Note that describe() will (lazily) compute the data frame but won’t display it, so we have to show() the result.

Listing 4.16. Describing everything in one fell swoop

for i in logs.columns:

logs.describe(i).show()

# +-------+------------------+ ❶

# |summary|      LogServiceID|

# +-------+------------------+

# |  count|           7169318|

# |   mean|3453.8804215407936|

# | stddev|200.44137201584468|

# |    min|              3157|

# |    max|              3925|

# +-------+------------------+

#

# [...]

#

# +-------+ ❷

# |summary|

# +-------+

# |  count|

# |   mean|

# | stddev|

# |    min|

# |    max|

# +-------+

# [... many more little tables]

❶

Numerical columns will display the information in a description table like so.

❷

If the type of the column isn’t compatible, PySpark displays only the title column.

It will take more time than doing everything in one fell swoop, but the output will be a lot friendlier. Now, because the mean or standard deviation of a string is not a thing, you’ll see null values here. Furthermore, some columns won’t be displayed (you’ll see time tables with only the title column), as describe() will only work for numerical and string columns. For a short line to type, you still get a lot!

describe() is a fantastic method, but what if you want more? summary() at the rescue!

Where describe() will take *cols as a parameter (one or more columns, the same way as select() or drop()), summary() will take *statistics as a parameter. This means that you’ll need to select the columns you want to see before passing the summary() method. On the other hand, we can customize the statistics we want to see. By default, summary() shows everything describe() shows, adding the approximate 25%-50% and 75% percentiles. Listing 4.17 shows how you can replace describe() for summary() and the result of doing so.

Listing 4.17. Summarizing everything in one fell swoop: default or custom options.

for i in logs.columns:

logs.select(i).summary().show()  ❶

# +-------+------------------+

# |summary|      LogServiceID|

# +-------+------------------+

# |  count|           7169318|

# |   mean|3453.8804215407936|

# | stddev|200.44137201584468|

# |    min|              3157|

# |    25%|              3291|

# |    50%|              3384|

# |    75%|              3628|

# |    max|              3925|

# +-------+------------------+

#

# [... many more slightly larger tables]

for i in logs.columns:

logs.select(i).summary("min", "10%", "90%", "max").show()  ❷

# +-------+------------+

# |summary|LogServiceID|

# +-------+------------+

# |    min|        3157|

# |    10%|        3237|

# |    90%|        3710|

# |    max|        3925|

# +-------+------------+

#

# [...]

❶

By default, we have  count, mean, stddev, min, 25%, 50%, 75%, max as statistics.

❷

We can also pass our own following the same nomenclature convention.

If you want to limit yourself to a subset of those metrics, summary() will accept a number of string parameters representing the statistic. You can input count, mean, stddev, min or max directly. For approximate percentiles, you need to provide them in XX% format, such as 25%.

Both methods will work only on non-null values. For the summary statistics, it’s the expected behaviour, but the "count" entry will also count only the non-null values for each column. A good way to see which columns are mostly empty!

Warning

describe() and summary() are two very useful methods, but they are not meant to be used anywhere else than during development, to quickly peek at data. The PySpark developers don’t guarantee the backward compatibility of the output, so if you need one of the outputs for your program, use the corresponding function in pyspark.sql.functions. They’re all there.

This chapter covered the ingestion and discovery of a tabular data set, one of the most popular data representation formats. We built on the basics of PySpark data manipulation, covered in chapters 2 and 3, and added a new layer by working with columns. The next chapter will be the direct continuation of this one, where we will explore more advanced aspects of the data frame structure.

[6] In Quebecois, we say "s’enfarger dans les fleurs du tapis" to talk of someone who’s too bogged down on the details. Transliterated, it would be "to trip over the rug’s flowers".

4.5  SUMMARY

- PySpark uses the SparkReader object to read any kind of data directly in a data frame. The specialized `CSV SparkReader is used to ingest comma-separated value (CSV) files. Just like when reading text, the only mandatory parameter is the source location.

- The CSV format is very versatile, so PySpark provides many optional parameters to account for this flexibility. The most important ones are the field delimiter, the record delimiter, and the quotation character. All of those parameters have sensible defaults.

- PySpark can infer the Schema of a CSV file by setting the inferSchema optional parameter to True. PySpark accomplishes this by reading the data twice: once for setting the appropriate types for each columns, and another time to ingest the data in the inferred format.

- Tabular data is represented into a data frame in a series of Columns, each having a name and a type. Since the data frame is a column-major data structure, the concept of row is less relevant.

- You can use Python code to explore the data efficiently, using the column list as any Python list to expose the elements of the data frame of interest.

- The most common operations on a data frame are the selection, deletion, and creation or columns. In PySpark, the methods used are select(), delete() and withColumn(), respectively.

- select can be used for column re-ordering by passing a re-ordered list of columns.

- You can rename columns one by one with the withColumnRenamed() method, or all at once by using the toDF() method.

- You can display a summary of the columns with the describe() or summary() method. describe() has a fixed set of metrics, while summary() will take functions as parameters and apply them to all columns.

5

THE DATA FRAME THROUGH A NEW LENS: JOINING AND GROUPING

This chapter covers:

- Joining two data frames together.

- How to select the right type of join for your use-case.

- Grouping data and understanding the GroupedData transitional object.

- Breaking the GroupedData with an aggregation method and getting a summarized data frame.

- Filling null values in your data frame

In chapter 4, we looked at how we can transform a data frame using selection, dropping, creation, renaming, re-ordering, and summary of columns. Those operations constitute the foundation working with a data frame in PySpark. In this chapter, I will complete the review of the most common operations you will perform on a data frame: linking or joining data frames together, as well as grouping data (and performing operations on the GroupedData object). We conclude this chapter by wrapping our exploratory program into a single script we can submit, just like we performed in chapter 3.

We use the same logs table that we left in chapter 4. In practical steps, this chapter’s code enriches our table with the relevant information contained in the link tables and then get summarized into relevant groups, using what can be considered a graduate version of the describe() method I show in chapter 4. If you want to catch up with a minimal amount of fuzz, I provide an end_of_chapter.py script in the src/Ch04 directory.

5.1  FROM MANY TO ONE: JOINING DATA

When working with data, we’re most often working on one structure at a time. Up until now, we’ve been exploring the many ways we can slice, dice, and modify a data frame to our wildest desires. What happens when we need to link two sources together? This section will introduce joins and how we can apply them when using a star schema setup or another set of tables where values match exactly. This is the easiest use case: joins can get dicey, and chapter 9 drills deeper into the subject of efficient joins.

Joining data frames is a common operation when working with related tables together. If you’ve used other data processing libraries, you might have seen the same operation being called a merge or a link. Joins are very common operations, but they are very flexible: in the next section, we set a common vocabulary to avoid confusion and build our understanding on a solid foundation.

5.1.1  What’s what in the world of joins

At its most basic, a join operation is a way to take the data from a data frame and add it to another one according to a set of rules. To introduce the moving parts of a join in a practical fashion, I ingest in listing 5.1 a second table to be joined to our logs data frame. I use the same parametrization of the SparkReader.csv as used for the logs table to read our new log_identifier table. Once ingested, I filter the data frame so keep only the primary channels, as per the data documentation. With this, we should be good to go.

Listing 5.1. Exploring our first link table: log_identifier

DIRECTORY = "../../data/Ch04"

log_identifier = spark.read.csv(

os.path.join(DIRECTORY, "ReferenceTables/LogIdentifier.csv"),

sep="|",

header=True,

inferSchema=True,

)

log_identifier.printSchema()

# root

#  |-- LogIdentifierID: string (nullable = true) ❶

#  |-- LogServiceID: integer (nullable = true) ❷

#  |-- PrimaryFG: integer (nullable = true) ❸

log_identifier.show(5)

# +---------------+------------+---------+

# |LogIdentifierID|LogServiceID|PrimaryFG|

# +---------------+------------+---------+

# |           13ST|        3157|        1|

# |         2000SM|        3466|        1|

# |           70SM|        3883|        1|

# |           80SM|        3590|        1|

# |           90SM|        3470|        1|

# +---------------+------------+---------+

# only showing top 5 rows

log_identifier = log_identifier.where(F.col("PrimaryFG") == 1)

log_identifier.count()

# 920

❶

Channel identifier

❷

Channel key (maps to our centre table!)

❸

Boolean flag: is the channel primary (1) or not (0)? We want only the 1’s

We have two data frames, containing each a set of columns. The join operation has three major ingredients:

1. two tables, called a left and a right table respectively (more on this in a moment);

2. one or more predicates which are the series of conditions that determine if records between the two tables are joined;

3. finally, a method to indicate how we perform the join when the predicate succeeds and when it fails.

With those three ingredients, you can construct a join between two data frames in PySpark by filling the [KEYWORDS] with the desired behaviour. In listing 5.2, I show what a barebone join recipe looks like before we start digging on how to customize those ingredients. The next three sections are dedicated to each one of those parameters.

Listing 5.2. A barebone recipe for a join in PySpark: we need to replace the [KEYWORDS] block.

[LEFT].join(

[RIGHT],

on=[PREDICATES]

how=[METHOD]

)

5.1.2  Knowing our left from our right

A join is performed on two tables at a time. Because of the SQL heritage in the vocabulary of data manipulation, the two tables are being named left and right tables. Chapter 7 contains a little more explanation about why they are called this way. In PySpark, a neat way to remember which one is which is to say that the left table is to the left of the join() method where the right one is to the right (inside the parentheses). Knowing which one is which is very useful when choosing the join method: unsurprisingly, there is a left and right join type…

Our tables are now identified, so we can update our join blueprint as I do in listing 5.3. We now need to steer our attention to the next parameter, the predicates.

Listing 5.3. A barebone recipe for a join in PySpark, with the left and right tables filled in.

logs.join(

log_identifier,

on=[PREDICATES]

how=[METHOD]

)

5.1.3  The rules to a successful join: the predicates

The predicates of a PySpark join are rules between columns of the left and right data frames. A join is performed record-wise, where each record on the left data frame is compared (via the predicates) to each record on the right data frame. If the predicates return True, the join is a match and is a failure if False.

The best way to illustrate a predicate is to create a simple example and explore the results. For our two data frames, we will build the predicate logs["LogServiceID"] == log_identifier["LogServiceID"]. In plain English, we can translate this by the following.

Match the records from the logs data frame to the records from the log_identifier data frame when the value of their LogServiceID column is equal.

I’ve taken a small sample of the data in both data frames and illustrated the result of applying the predicate in figure 5.1. There are two important points to highlight:

1. If one record in the left table resolves the predicate with more than one record in the right table (or vice versa), this record will be duplicated in the joined table.

2. If one record in the left or in the right table does not resolve the predicate with any record in the other table, it will not be present in the resulting table, unless the join method specifies a protocol for failed predicates.

Figure 5.1. Our predicate is applied to a sample of our two tables. 3590 on the left table resolves the predicate twice while 3417 on the left/3883 on the right do not solve it.

In our example, the 3590 record on the left is equal to the two corresponding records on the right, and we see two solved predicates with this number in our result set. On the other hand, the 3417 record does not match anything on the right and therefore is not present in the result set. The same thing happens with the 3883 record on the right table.

You are not limited to a single test in your predicate. You can use multiple conditions by separating them with boolean operators such as | (or) or & (and). You can also use a different test than equality. Here are two examples and their plain English translation.

1. (logs["LogServiceID"] == log_identifier["LogServiceID"]) & (logs["left_col"] < log_identifier["right_col"]) : Will only match the records that have the same LogServiceID on both side and where the value of the left_col in the logs table is smaller than the value of the right_col in the log_identifier table.

2. (logs["LogServiceID"] == log_identifier["LogServiceID"]) | (logs["left_col"] > log_identifier["right_col"]) : Will only match the records that have the same LogServiceID on both side or where the value of the left_col in the logs table is greater than the value of the right_col in the log_identifier table.

You can build the operations as complicated as you want. I really recommend wrapping each condition in parentheses to avoid worrying about the operator precedence and facilitate the reading.

Before adding our predicate to our join in progress, I want to note that PySpark provides a few predicate shortcuts to reduce the complexity of the code.

If you have multiple and predicates (such as (left["col1"] == right["colA"]) & (left["col2"] > right["colB"]) & (left["col3"] != right["colC"])), you can put them into a list such as [left["col1"] == right["colA"], left["col2"] > right["colB"], left["col3"] != right["colC"]]. This makes you intent more explicit and avoids counting parentheses for long chains of conditions.

Finally, if you are performing an "equi-join", where you are testing for equality between identically named columns, you can just specify the name of the columns as a string or a list of strings as a predicate. In our case, it means that our predicate can only be "LogServiceID". This is what I put in listing 5.4.

Listing 5.4. A barebone recipe for a join in PySpark, with the left and right tables filled in, as well as the predicate.

logs.join(

log_identifier,

on="LogServiceID"

how=[METHOD]

)

The join method influences how you structure predicates, so Listing 5.9 revisits the whole join operation after we’re done with the ingredient-by-ingredient approach. The last parameter is the how, which completes our join operation.

5.1.4  How do you do it: the join method

The last ingredient of a successful join is the how parameter, which will indicate the join method. Most books explaining joins shows Venn diagrams indicating how each joins colors the different areas, but I find that it is only useful as a reminder, not a teaching tool. I’ll review each type of join with the same tables we’ve used in figure 5.1, giving the result of the operation.

A join method basically boils down to those two questions:

1. What happens when the return value of the predicates is True?

2. What happens when the return value of the predicates is False?

Classifying the join methods based on the answer to those two questions is an easy way to remember them.

Tip

PySpark’s joins are basically the same as SQL joins. If you are already comfortable with them, feel free to skip this section.

CROSS JOIN: THE NUCLEAR OPTION

A cross join (how="cross") is the nuclear option. It returns a record for every record pair, regardless of the value the predicates return. In our data frame example, our logs table contains 4 records and our logs_identifier contains 5 records, so the cross join will contain 4 x 5 = 20 records. The result is illustrated in figure 5.2.

Figure 5.2. A visual example of a cross join. Each record on the left is matched to every record on the right.

Cross joins are seldom the operation that you want, but they are useful when you want a table that contains every possible combination.

Tip

PySpark also provides an explicit  crossJoin() method that takes the right data frame as a parameter.

INNER JOIN

An inner join (how="inner") is the most common join by a landslide. PySpark will default to an inner join if you don’t pass a join method explicitly. It returns a record if the predicate is true and drops it if false. I consider inner join to be the natural way to think of joins because they are very simple to reason about.

If we look at our tables, we would have a table very similar to figure 5.1. The record with the LogServiceID == 3590 on the left will be duplicated because it matches two records on the right table. The result is illustrated in figure 5.3.

Figure 5.3. An inner join. Each successful predicate creates a joined record.

LEFT AND RIGHT OUTER JOIN

Left (how="left" or how="left_outer") and right (how="right" or how="right_outer") are like inner join, in which they generate a record for a successful predicate. The difference is what happens when the predicate is false:

- A left (also called a left outer) join will add the unmatched records from the left table in the joined table, filling the columns coming from the right table with None/null

- A right (also called a right outer) join will add the unmatched records from the right in the joined table, filling the columns coming from the left table with None/null

In practice, it means that your joined table is guaranteed to contain all the records of the table which feeds the join (left or right). Visually, figure 5.4 shows that, although 3417 doesn’t satisfy the predicate, it is still present in the left joined table. The same happens with 3883 and the right table. Just like an inner join, if the predicate is successful more than once, the record will be duplicated.

Figure 5.4. A left and right joined table. All the records of the direction table are present in the resulting table.

Left and right joins are very useful when you are not certain if the link table contains every key. You can then fill the null values — see listing 5.17 — or process them knowing you didn’t drop any records.

FULL OUTER JOIN

A full outer (how="outer", how="full" or how="full_outer") join is simply the fusion of a left and right join. It will add the unmatched records from the left and the right table, padding with None/null. It serves a similar purpose to the left and right join but is not as popular since you’ll generally have one (and just one) anchor table that you want to preserve all records.

Figure 5.5. A left and right joined table. We can see all the records from both table.

LEFT SEMI-JOIN AND LEFT ANTI-JOIN

The left semi-join and left anti-join are a little more esoteric, but are really easy to understand.

A left semi-join (how="left_semi") is the same as an inner join, but only keeps the columns in the left table. It also won’t duplicate the records in the left table if they fulfill the predicate with more than one record in the right table. Its main purpose is to filter out records from a table based on a predicate depending of another table.

A left anti-join is the opposite of an inner join. It will keep only the records that do not match the predicate with any record in the right table, dropping any successful match.

Our blueprint join is now finalized: we are going with an inner join since we want to keep only the records where the LogServiceID has additional information in our log_identifier table. Since our join is complete, I assign the result to a new variable logs_and_channels.

Listing 5.5. A barebone recipe for a join in PySpark, with the left and right tables filled in, as well as the predicate and the method.

logs_and_channels = logs.join(

log_identifier,

on="LogServiceID"

how="inner"  ❶

)

❶

I could have omitted the  how parameter outright, since inner join is the default.

With the first join done, we will link two additional tables to continue our data discovery and processing. The CategoryID table contains information about the type of programs and the ProgramClassID table contains the data that allows us to pinpoint the commercials.

This time, we are performing left joins since we are not entirely certain about the existence of the keys in the link table. In listing 5.6, we follow the same process as we did for the log_identifier table, in one fell swoop.

- We read the table using the SparkReader.csv and the same configuration as our other tables.

- We keep the relevant columns.

- We join the data to our logs_and_channels table, using PySpark’s method chaining.

Listing 5.6. Linking the category and program class tables using a left join

DIRECTORY = "../../data/Ch04"

cd_category = spark.read.csv(

os.path.join(DIRECTORY, "ReferenceTables/CD_Category.csv"),

sep="|",

header=True,

inferSchema=True,

).select(

"CategoryID",

"CategoryCD",

F.col("EnglishDescription").alias("Category_Description"),  ❶

)

cd_program_class = spark.read.csv(

os.path.join(DIRECTORY, "ReferenceTables/CD_ProgramClass.csv"),

sep="|",

header=True,

inferSchema=True,

).select(

"ProgramClassID",

"ProgramClassCD",

F.col("EnglishDescription").alias("ProgramClass_Description"),  ❷

)

full_log = logs_and_channels.join(cd_category, "CategoryID", how="left").join(

cd_program_class, "ProgramClassID", how="left"

)

❶

We’re aliasing the  EnglishDescription column to remember what it maps to

❷

Same as #1 here, but for the program class.

In listing 5.6, in the select statement, I aliased some columns that were both named EnglishDescription. How will PySpark know which one to keep? The next section takes a brief detour from our exploration to address one of the most frequent sources of frustration when dealing with columns: staying on top of you column names when joining data frames.

The science of joining in a distributed environment

When joining data in a distributed environment, the "we don’t care about where data is" doesn’t work anymore. To be able to process a comparison between records, the data needs to be on the same machine. If not, PySpark will move the data in an operation called a shuffle. As you can imagine, moving large amounts of data over the network is very slow, and we should aim to avoid this when possible.

This is one of the instances where PySpark’s abstraction model shows some weakness. Since joins are such an important part of working with multiple data sources, I’ve decided to introduce the syntax here so we can get things rolling. We revisit performance considerations and join strategies in chapter 9. For the moment, trust the optimizer!

5.1.5  Naming conventions in the joining world

By default, PySpark will not allow two columns to be named the same. If you create a column with withColumn() using an existing column name, PySpark will overwrite (or shadow) the column. When joining data frames, the situation is a little more complicated, as displayed in listing 5.7

Listing 5.7. A join that generates two seemingly identically named columns

logs_and_channels_verbose = logs.join(

log_identifier, logs["LogServiceID"] == log_identifier["LogServiceID"]

)

logs_and_channels_verbose.printSchema()

# root

#  |-- LogServiceID: integer (nullable = true) ❶

#  |-- LogDate: timestamp (nullable = true)

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  [...]

#  |-- duration_seconds: integer (nullable = true)

#  |-- LogIdentifierID: string (nullable = true)

#  |-- LogServiceID: integer (nullable = true) ❷

#  |-- PrimaryFG: integer (nullable = true)

try:

logs_and_channels_verbose.select("LogServiceID")

except AnalysisException as err:

print(err)

# "Reference 'LogServiceID' is ambiguous, could be: LogServiceID, LogServiceID.;" ❸

❶

One  LogServiceID column…

❷

… and another!

❸

PySpark doesn’t know which column we mean.

PySpark happily joins the two data frames together but fails when we try to work with the ambiguous column. This is a frequent situation when working with data that follows the same convention for column naming. Fortunately, solving this problem is easy. I show in this section three methods, from the easiest to the most general.

First, when performing an equi-join, prefer using the simplified syntax, since it takes care of removing the second instance of the predicate column. This only works when using equality comparison, since the data is identical in both columns from the predicate, preventing information loss. I show the code and schema of the resulting data frame when using a simplified equi-join in listing 5.8.

Listing 5.8. Using the simplified syntax for equi-joins results in no duplicate columns.

logs_and_channels = logs.join(log_identifier, "LogServiceID")

logs_and_channels.printSchema()

# root

#  |-- LogServiceID: integer (nullable = true)

#  |-- LogDate: timestamp (nullable = true)

#  |-- AudienceTargetAgeID: integer (nullable = true)

#  |-- AudienceTargetEthnicID: integer (nullable = true)

#  |-- CategoryID: integer (nullable = true)

#  [...]

#  |-- Language2: integer (nullable = true)

#  |-- duration_seconds: integer (nullable = true)

#  |-- LogIdentifierID: string (nullable = true)

#  |-- PrimaryFG: integer (nullable = true)

The second approach relies on the fact that PySpark joined data frames remembers the provenance of the columns. Because of this, we can refer to the LogServiceID columns using the same nomenclature as before, e.g. log_identifier["LogServiceID"]. We can then rename this column or delete is, solving our issue. I use this approach in listing 5.9

Listing 5.9. Using the origin name of the column allows for unambiguous selection of the columns after a join

logs_and_channels_verbose = logs.join(

log_identifier, logs["LogServiceID"] == log_identifier["LogServiceID"]

)

logs_and_channels.drop(log_identifier["LogServiceID"]).select("LogServiceID")  ❶

# DataFrame[LogServiceID: int]

❶

By dropping one of the two duplicated columns, we can then use the name for the other without any problem.

The last approach is convenient if you use the Column object directly. PySpark will not resolve the origin name when you rely on F.col() to work with columns. To solve this the most general way, we need to alias() our tables when performing the join, as shown in listing 5.10.

Listing 5.10. Aliasing our tables makes the origin resolved when using F.col() to refer to columns.

logs_and_channels_verbose = logs.alias("left").join(  ❶

log_identifier.alias("right"),  ❷

logs["LogServiceID"] == log_identifier["LogServiceID"],

)

logs_and_channels_verbose.drop(F.col("right.LogServiceID")).select(

"LogServiceID"

)  ❸

# DataFrame[LogServiceID: int]

❶

Our  logs table gets aliased as left.

❷

Our  log_identifier gets aliased as right.

❸

F.col() will resolve left and right as prefix for the column names.

All three approaches are valid. The first one works only in the case of equijoins, but the two others are mostly interchangeable. PySpark gives you a lot of control over the structure and naming of your data frame but requires you to be explicit.

This section packed a lot of information about joins, a very important tool when working with interrelated data frames. Although the possibilities are endless, the syntax is simple and easy to understand.

left.join(right, who’s the first parameter. on decides if it’s a match. how indicates how to operate on match success and failures.

Maybe we could turn this into a rap…

With our table nicely augmented, let’s carry on to our last step: summarizing the table using groupings.

Exercise 5.1

What is the result of this code?

one = left.join(right, how="left_semi", on="my_column"

two = left.join(right, how="left_anti", on="my_column")

one.union(two)

Exercise 5.2

Write a PySpark code that will return the result of the following code block without using a left anti join.

left.join(right, how="left_anti", on="my_column").select("my_column").distinct()

Exercise 5.3 (hard)

Write a PySpark code that will return the result of the following code block without using a left semi-join.

left.join(right, how="left_semi", on="my_column").select("my_column").distinct()

5.2  SUMMARIZING THE DATA VIA: GROUPBY AND GROUPEDDATA

When displaying data, especially large amounts of data, you’ll often summarize data using statistics as a first step. As a matter of fact, chapter 4 showed how you can use summary() and display() to compute mean, min, max, etc. over the whole data frame. What if we need to look using a different lens?

This section covers the groupby() method in greater detail than seen in chapter 3. I introduce here the GroupedData object and its usage. In practical terms, we’ll use groupby() to answer our original question: what are the channels with the most and least proportion of commercials? In order to answer this, we have to take each channel and sum the duration_seconds in two ways:

1. One to get the number of seconds when the program is a commercial.

2. One to get the number of seconds of total programming.

Our plan, before we start summing, is to identify what is considered a commercial and what is not. The documentation doesn’t provide formal guidance on how to do so, so we’ll explore the data and draw our conclusion. Let’s group!

5.2.1  A simple groupby blueprint

In chapter 3, we performed a very simple groupby() to count the occurrences of each word. It was a very simple example of grouping and counting records based on the words inside the (only) column. In this section, we expand on that simple example by grouping over many columns. I also introduce a more general notation than the count() we’ve used previously, so we compute more that one summary function.

Since you are already acquainted with the basic syntax of groupby(), this section starts by presenting a full code block that computes the total duration (in seconds) of the program class. In listing 5.11 we perform the grouping, compute the aggregate function, and present the results in decreasing order.

Listing 5.11. Displaying the most popular types of programs

full_log.groupby("ProgramClassCD", "ProgramClass_Description").agg(

F.sum("duration_seconds").alias("duration_total")

).orderBy("duration_total", ascending=False).show(100, False)

# +--------------+--------------------------------------+--------------+

# |ProgramClassCD|ProgramClass_Description              |duration_total|

# +--------------+--------------------------------------+--------------+

# |PGR           |PROGRAM                               |652802250     |

# |COM           |COMMERCIAL MESSAGE                    |106810189     |

# |PFS           |PROGRAM FIRST SEGMENT                 |38817891      |

# |SEG           |SEGMENT OF A PROGRAM                  |34891264      |

# |PRC           |PROMOTION OF UPCOMING CANADIAN PROGRAM|27017583      |

# |PGI           |PROGRAM INFOMERCIAL                   |23196392      |

# |PRO           |PROMOTION OF NON-CANADIAN PROGRAM     |10213461      |

# |OFF           |SCHEDULED OFF AIR TIME PERIOD         |4537071       |

# [... more rows]

# |COR           |CORNERSTONE                           |null          |

# +--------------+--------------------------------------+--------------+

This small program has a few new parts, so let’s review them one by one.

Our grouping routing starts with the groupby() method. A "grouped by" data frame is not a data frame anymore, instead, it becomes a GroupedData object, displayed in all its glory in listing 5.13. This object a transitional object: you can’t really inspect it (there is no .show() method) and it’s waiting for further instructions to become show-able again. Illustrated, it would look like the right-hand side of figure 5.7. You have the key (or keys, if you groupby() multiple columns), and the rest of the columns are grouped inside some "cell", awaiting a summary function so they can be promoted to a bona fide column again.

agg() for the lazy

agg() also accepts a dictionary, in the form {column_name: aggregation_function} where both are string. Because of this, we can rewrite listing 5.11 like so.

Listing 5.12. Displaying the most popular types of programs, using a dictionary expression inside agg()

full_log.groupby("ProgramClassCD", "ProgramClass_Description").agg(

{"duration_seconds": "sum"}

).withColumnRenamed("sum(duration_seconds)", "duration_total").orderBy(

"duration_total", ascending=False

).show(

100, False

)

It makes rapid prototyping very easy (you can, just like with column objects, use the "*" to refer to all columns). I personally don’t like this approach for most cases since you don’t get to alias your columns when creating them. I am including it since you will see it when reading other people’s code.

Listing 5.13. A GroupedData object representation. Unlike the data frame, we have no information about the columns or the structure.

full_log.groupby()

# <pyspark.sql.group.GroupedData at 0x119baa4e0>

Figure 5.6. The original data frame, with the focus on the columns we are grouping by.

Figure 5.7. The GroupedData object resulting from grouping by (ProgramClassID, ProgramClass_Description). The non-key columns are all in stand-by in the group cell.

In chapter 3, we brought back the GroupedData into a data frame by using the count() method, which returns the count of each group. There are a few other, such as min(), max(), mean() or sum(). We could have used the sum() method directly, but we wouldn’t have had the option of aliasing the resulting column, getting stuck with sum(duration_seconds) for a name. Instead, we use the oddly named agg().

The agg() method, for aggregate (or aggregation?), will take one or more aggregate functions from the pyspark.sql.functions module we all know and love and apply them on each group of the GroupedData object. In figure 5.8, I start on the left with our GroupedData oblect. Calling agg() with an appropriate aggregate function pulls the column from the group cell, extracts the values, and performs the function, yielding the answer. Compared to using the sum() function on the group by object, agg() trades a few keystrokes for 2 main advantages:

1. agg() takes an arbitrary number of aggregate functions, unlike using a summary method directly. You can’t chain multiple functions on GroupedData objects: the first one will transform it into a data frame, and the second one will fail.

2. You can alias resulting columns, so you control their name and improve the robustness of your code.

Figure 5.8. A data frame arising from the application of the agg() method (aggregate function: F.sum() on Duration_seconds)

After the application of the aggregate function on our GroupedData object, we’re back with a data frame. We can then use the orderBy method to order the data by decreasing order of duration_total, our newly created column. We finish by showing 100 rows, which is more than what the data frame contains, so it shows everything.

Let’s select our commercials. Table 5.1  shows my picks.

Table 5.1. The types of programs we’ll be considering as commercials

ProgramClassCD

ProgramClass_Description

duration_total

COM

COMMERCIAL MESSAGE

106810189

PRC

PROMOTION OF UPCOMING CANADIAN PROGRAM

27017583

PGI

PROGRAM INFOMERCIAL

23196392

PRO

PROMOTION OF NON-CANADIAN PROGRAM

10213461

LOC

LOCAL ADVERTISING

483042

SPO

SPONSORSHIP MESSAGE

45257

MER

MERCHANDISING

40695

SOL

SOLICITATION MESSAGE

7808

Now that we’ve done the hard job of identifying our commercial codes, we can start counting!

agg() is not the only player in town

Since PySpark 2.3, you can also use groupby() with the apply() method, in the creatively named "split-apply-combine" pattern. I cover this pattern in chapter 8.

5.2.2  A column is a column: using agg with custom column definitions

When grouping and aggregating columns in PySpark, we have the whole power of the Column object at our fingertips. This means that we can group by and aggregate on custom columns! For this section, we will start by building a definition of duration_commercial, which takes the duration of a program only if it is a commercial, and use this in our agg() statement to seamlessly compute both the total duration and the commercial duration.

If we encode the content of Table 5.1  into a PySpark definition, this gives us listing 5.14

Listing 5.14. Computing only the commercial time for each program in our table

F.when(

F.trim(F.col("ProgramClassCD")).isin(

["COM", "PRC", "PGI", "PRO", "PSA", "MAG", "LOC", "SPO", "MER", "SOL"]

),

F.col("duration_seconds"),

).otherwise(0)

I think that the best way to describe the code this time is to literally translate it into plain English.

When the field of the col(umn) "ProgramClass", trim(med) of spaces at the beginning and end of the field is in our list of commercial codes, then take the value of the field in the column "duration_seconds". Otherwise, use 0 as a value.

The blueprint of the F.when() function is as follows. It is possible to chain multiple when() if we have more than one condition, and to omit the otherwise() if we’re okay with having null values when none of the tests are positive.

(

F.when([BOOLEAN TEST], [RESULT IF TRUE])

.when([ANOTHER BOOLEAN TEST], [RESULT IF TRUE])

.otherwise([DEFAULT RESULT, WILL DEFAULT TO null IF OMITTED])

)

We now have a column ready to use. While, we could create the column before grouping by, using withColumn(), let’s take it up a notch and use our definition directly in the agg() clause. Listing 5.15 does just that, and at the same time, gives us our answer!

Listing 5.15. Using our new column into agg() to compute our final answer!

answer = (

full_log.groupby("LogIdentifierID")

.agg(

F.sum(

F.when(

F.trim(F.col("ProgramClassCD")).isin(

["COM", "PRC", "PGI", "PRO", "LOC", "SPO", "MER", "SOL"]

),

F.col("duration_seconds"),

).otherwise(0)

).alias("duration_commercial"),

F.sum("duration_seconds").alias("duration_total"),

)

.withColumn(

"commercial_ratio", F.col("duration_commercial") / F.col("duration_total")

)

)

answer.orderBy("commercial_ratio", ascending=False).show(1000, False)

# +---------------+-------------------+--------------+---------------------+

# |LogIdentifierID|duration_commercial|duration_total|commercial_ratio     |

# +---------------+-------------------+--------------+---------------------+

# |HPITV          |403                |403           |1.0                  |

# |TLNSP          |234455             |234455        |1.0                  |

# |MSET           |101670             |101670        |1.0                  |

# |TELENO         |545255             |545255        |1.0                  |

# |CIMT           |19935              |19935         |1.0                  |

# |TANG           |271468             |271468        |1.0                  |

# |INVST          |623057             |633659        |0.9832686034602207   |

# [...]

# |OTN3           |0                  |2678400       |0.0                  |

# |PENT           |0                  |2678400       |0.0                  |

# |ATN14          |0                  |2678400       |0.0                  |

# |ATN11          |0                  |2678400       |0.0                  |

# |ZOOM           |0                  |2678400       |0.0                  |

# |EURO           |0                  |null          |null                 |

# |NINOS          |0                  |null          |null                 |

# +---------------+-------------------+--------------+---------------------+

Wait a moment? Are some channels only commercials? That can’t be it. If we look at the total duration, we can see that some channels don’t broadcast a lot (1 day = 86,400 seconds). Still, we accomplished our goal: we identified the channels with the most commercials. We finish this chapter with one last task: processing those null values.

5.3  TAKING CARE OF NULL VALUES: DROP AND FILL

Null values represent the absence of value. I find this to be a great oxymoron: a value for no value? Philosophy aside, we have some nulls in our result set and I would like them gone.

PySpark provides two main functionalities to deal with null values: you can either dropna() the record containing them of fillna() the null with a value. We explore in this section both options to see which one is best for our analysis.

5.3.1  Dropping it like it’s hot

Our first option would be to plainly ignore the records that have null values. For this, we use the data frame dropna() method. It takes three parameters:

1. how, which can take the value any or all. If any is selected, PySpark will drop records where at least one of the fields are null. In the case of all, only the records where all fields are null will be removed. By default, PySpark will take the any mode.

2. thresh takes an integer value. If set (its default is None), PySpark will ignore the how parameter and only drop the records with less than thresh non-null values.

3. Finally, subset will take an optional list of columns that drop will use to make its decision.

In our case, we want to keep only the records that have a commercial_ratio that is non-null. We just have to pass our column to the subset parameter, like in listing 5.16.

Listing 5.16. Dropping only the records that have a commercial_ratio value of null.

answer_no_null = answer.dropna(subset=["commercial_ratio"])

answer_no_null.orderBy("commercial_ratio", ascending=False).show(1000, False)

# +---------------+-------------------+--------------+---------------------+

# |LogIdentifierID|duration_commercial|duration_total|commercial_ratio     |

# +---------------+-------------------+--------------+---------------------+

# |HPITV          |403                |403           |1.0                  |

# |TLNSP          |234455             |234455        |1.0                  |

# |MSET           |101670             |101670        |1.0                  |

# |TELENO         |545255             |545255        |1.0                  |

# |CIMT           |19935              |19935         |1.0                  |

# |TANG           |271468             |271468        |1.0                  |

# |INVST          |623057             |633659        |0.9832686034602207   |

# [...]

# |OTN3           |0                  |2678400       |0.0                  |

# |PENT           |0                  |2678400       |0.0                  |

# |ATN14          |0                  |2678400       |0.0                  |

# |ATN11          |0                  |2678400       |0.0                  |

# |ZOOM           |0                  |2678400       |0.0                  |

# +---------------+-------------------+--------------+---------------------+

print(answer_no_null.count())  # 322

This option is completely legitimate, but it removes some records from our data frame. What if we want to keep everything?

5.3.2  Filling values to our heart’s content

The yin to dropna() yang is to provide a default value to the null values. For this, PySpark provided the fillna() method. This method takes two parameters.

1. The value, which is either a Python int, float, string or bool. PySpark will only fill the compatible columns: for instance, if we were to fillna("zero"), our commercial_ratio, being a double, would not be filled.

2. The same subset parameter we encountered in dropna(). We can limit the scope of our filling to only the columns we want.

Concretely, a null value in any of our numerical columns means that the value should be zero, so listing 5.17 fills the null values with 0.

Listing 5.17. Filling our numerical records with 0 using the fillna() method.

answer_no_null = answer.fillna(0)

answer_no_null.orderBy("commercial_ratio", ascending=False).show(1000, False)

# +---------------+-------------------+--------------+---------------------+

# |LogIdentifierID|duration_commercial|duration_total|commercial_ratio     |

# +---------------+-------------------+--------------+---------------------+

# |HPITV          |403                |403           |1.0                  |

# |TLNSP          |234455             |234455        |1.0                  |

# |MSET           |101670             |101670        |1.0                  |

# |TELENO         |545255             |545255        |1.0                  |

# |CIMT           |19935              |19935         |1.0                  |

# |TANG           |271468             |271468        |1.0                  |

# |INVST          |623057             |633659        |0.9832686034602207   |

# [...]

# |OTN3           |0                  |2678400       |0.0                  |

# |PENT           |0                  |2678400       |0.0                  |

# |ATN14          |0                  |2678400       |0.0                  |

# |ATN11          |0                  |2678400       |0.0                  |

# |ZOOM           |0                  |2678400       |0.0                  |

# +---------------+-------------------+--------------+---------------------+

print(answer_no_null.count())  # 324 ❶

❶

We have the two additional records that  listing 5.16 dropped.

The return of the dict

You can also pass a dict to the fillna method, with the column names as key and the values as dict values. If we were to use this method for our filling, the code would be like listing 5.18.

Listing 5.18. Filling our numerical records with 0 using the fillna() method and a dict.

answer_no_null = answer.fillna(

{"duration_commercial": 0, "duration_total": 0, "commercial_ratio": 0}

)

Just like with agg(), I prefer avoiding the dict approach because I find it less readable. In this case, you can chain multiple fillna() to achieve the same result, with better readability.

Our program now is devoid of null values, and we have a full list of channels and their associated ratio of commercial programming.

5.4  WHAT WAS OUR QUESTION AGAIN: OUR END-TO-END PROGRAM

At the beginning of the chapter, we gave ourselves an anchor question to start exploring the data and uncover some insights. Through the chapter, we’ve assembled a cohesive dataset containing the relevant information to identify commercial programs and have ranked the channels based on how much of their programming is commercial. In Listing 5.19, I’ve assembled all the relevant code blocks introduced in the chapter into a single program you can spark-submit. The code is also available in the book’s repository, under src/Ch05/commercials.py. The end-of-chapter exercises also use this code.

Not counting data ingestion, comments or docstring, our code is a rather small 35 lines of code. We could play code golf (trying to shrink the number of characters as much as we can), but I think we’ve struck a good balance between terseness and ease of reading. Once again, we haven’t paid much attention to the distributed nature of PySpark. Once again, we took a very descriptive view of our problem and translated it into code via PySpark’s powerful data frame abstraction and rich function ecosystems.

This chapter is the last chapter of the first part of the book. You are now familiar with the PySpark ecosystem and how you can use its main data structure, the data frame, to ingest and manipulate two very common sources of data, textual and tabular. You know a variety and method and functions that can be applied to data frames and columns and can apply those to your own data problem. You can also leverage the documentation provided through the PySpark docstrings, straight from the PySpark shell.

There is a lot more you can get from the plain data manipulation portion of the book. Because of this, I created appendix F — how to use the online PySpark API documentation — as a guide to becoming self-reliant using the API documentation. Now that you have a solid understanding of the data model and how to structure simple data manipulation programs, adding new functions to your PySpark quiver will be easy.

The second part of the book builds heavily on what you’ve learned so far.

- We dig deeper into PySpark’s data model and find opportunities to refine our code. We will also look at PySpark’s column types, how they bridge to Python’s types, and how to use them to improve the reliability of our code.

- We also look at how PySpark modernizes SQL, an influential language for tabular data manipulation, and how you can blend SQL and Python in a single program.

- We look at promoting pure Python code to run in the Spark distributed environment. We formally introduce a lower-level structure, the Resilient Distributed Dataset (RDD) and its row-major model. We also look at User Defined Functions (UDF) as a way to augment the functionality of the data frame.

- Finally, we go beyond rows and columns by using the document-data capacities of PySpark’s data frame.

Listing 5.19. Our full program, ordering channels by decreasing proportion of commercials in their airings.

"""commercials.py

This program computes the commercial ratio for each channel present in the

dataset.

"""

import os

import pyspark.sql.functions as F

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName(

"Getting the Canadian TV channels with the highest/lowest proportion of commercials."

).getOrCreate()

spark.sparkContext.setLogLevel("WARN")

###############################################################################

# Reading all the relevant data sources

###############################################################################

DIRECTORY = "./data/Ch04"

logs = spark.read.csv(

os.path.join(DIRECTORY, "BroadcastLogs_2018_Q3_M8.CSV"),

sep="|",

header=True,

inferSchema=True,

)

log_identifier = spark.read.csv(

os.path.join(DIRECTORY, "ReferenceTables/LogIdentifier.csv"),

sep="|",

header=True,

inferSchema=True,

)

cd_category = spark.read.csv(

os.path.join(DIRECTORY, "ReferenceTables/CD_Category.csv"),

sep="|",

header=True,

inferSchema=True,

).select(

"CategoryID",

"CategoryCD",

F.col("EnglishDescription").alias("Category_Description"),

)

cd_program_class = spark.read.csv(

"./data/Ch03/ReferenceTables/CD_ProgramClass.csv",

sep="|",

header=True,

inferSchema=True,

).select(

"ProgramClassID",

"ProgramClassCD",

F.col("EnglishDescription").alias("ProgramClass_Description"),

)

###############################################################################

# Data processing

###############################################################################

logs = logs.drop("BroadcastLogID", "SequenceNO")

logs = logs.withColumn(

"duration_seconds",

(

F.col("Duration").substr(1, 2).cast("int") * 60 * 60

+ F.col("Duration").substr(4, 2).cast("int") * 60

+ F.col("Duration").substr(7, 2).cast("int")

),

)

log_identifier = log_identifier.where(F.col("PrimaryFG") == 1)

logs_and_channels = logs.join(log_identifier, "LogServiceID")

full_log = logs_and_channels.join(cd_category, "CategoryID", how="left").join(

cd_program_class, "ProgramClassID", how="left"

)

answer = (

full_log.groupby("LogIdentifierID")

.agg(

F.sum(

F.when(

F.trim(F.col("ProgramClassCD")).isin(

["COM", "PRC", "PGI", "PRO", "LOC", "SPO", "MER", "SOL"]

),

F.col("duration_seconds"),

).otherwise(0)

).alias("duration_commercial"),

F.sum("duration_seconds").alias("duration_total"),

)

.withColumn(

"commercial_ratio", F.col("duration_commercial") / F.col("duration_total")

)

.fillna(0)

)

answer.orderBy("commercial_ratio", ascending=False).show(1000, False)

5.5  SUMMARY

- PySpark implements seven join functionalities, using the common "what, on what, how?" questions: cross, inner, left, right, full, left semi and left anti. Choosing the right join method depends on how to process the records that resolve the predicates and the ones that do not.

- PySpark keeps lineage information when joining data frames. Using this information, we can avoid column naming clashes.

- You can group similar values together using the groupby() method on a data frame. The method takes a number of column objects or strings representing columns and returns a GroupedData object.

- GroupedData objects are transitional structures. They contain two types of columns: the key columns, which are the one you grouped by with, and the group cell, which is a container for all the other columns. In order to return to a data frame, the most common way is to summarize the values in the column via the agg() function, or via one of the direct aggregation methods, such as count() or min().

5.6  EXERCISES

5.6.1  Exercise 5.1

Using the data from the data/Ch04/Call_Signs.csv (not a typo) — careful: the delimiter here is the comma, not the pipe! — add the Undertaking_Name to our final table to display a human-readable description of the channel.

5.6.2  Exercise 5.2

The government of Canada is asking for your analysis, but they’d like the PRC to be weighted differently. They’d like each PRC second to be considered 0.75 commercial second. Modify the program to account for this change.

5.6.3  Exercise 5.3

On the data frame returned from commercials.py, return the percentage of channels in each bucket based on their commercial_ratio. (Hint: look at the documentation for round in how to truncate a value.)

commercial_ratio

proportion_of_channels

1.0

0.9

0.8

…

0.1

0.0

6

MULTI-DIMENTIONAL DATA FRAMES: USING PYSPARK WITH JSON DATA

So far, we have used PySpark’s data frame to work with textual (chapter 2 and 3) and tabular (chapter 4 and 5). Both formats are for the most part bi-dimenstional, meaning that we have rows and columns filled with data. PySpark represents data in many types — strings, numbers, even array/lists — within its cells. What if a column could contain a column?

Columns within columns. The ultimate flexibility.

This chapter is about ingesting and working with JSON data, using the PySpark data frame. I introduce the JSON format and how we can draw parallels to Python data structures. I then quickly review the scalar data types we use in PySpark and how they are used for encoding data within a column. I go over the three container structures available for the data frame: the array, the map, and the struct. I cover how we can use them to represent multidimensional data, and how the struct can represent hierarchical information. Finally, I wrap that information into a schema, a very useful construct for documenting what’s in your data frame.

6.1  READING JSON DATA: GETTING READY FOR THE SCHEMAPOCALYPSE

Every data processing job in PySpark starts with data ingestion and JSON documents are no exception. This section explains what is JSON, how to use the specialized JSON reader with PySpark and how a JSON file is represented within a data frame.

For this chapter, we use a JSON dump of the information about the TV Show Silicon Valley, from TV Maze. I have uploaded the data in the book’s repository (under ./data/Ch06), but you can download it directly from the TV Maze API (available online: api.tvmaze.com/singlesearch/shows?q=silicon-valley&embed=episodes). A simplified version of the JSON document is illustrated in listing 6.1: the main parts are numerated, and I go over each one of them.

Listing 6.1. A simplified sample of the JSON object we ingested, as represented in the shows-silicon-valley.json file.

{  ❶

"id": 143,  ❷

"name": "Silicon Valley",

"type": "Scripted",

"language": "English",

"genres": [ ❸

"Comedy"

],

"network": {  ❹

"id": 8,

"name": "HBO",

"country": {

"name": "United States",

"code": "US",

"timezone": "America/New_York"

}

},

"_embedded": {

"episodes": [  ❺

{

"id": 10897,

"name": "Minimum Viable Product",

"season": 1,

"number": 1,

},

{

"id": 10898,

"name": "The Cap Table",

"season": 1,

"number": 2,

}

]

}

}

❶

At the top level, a JSON object looks like a Python dictionary. Both use the brackets to delimit object boundaries.

❷

JSON data is encoded into key-value pairs, just like in a dictionary. JSON keys must be strings.

❸

JSON arrays can contain multiple values (here, we have only a single string).

❹

Objects can be values too: you can nest objects within one another this way.

❺

Our episodes are each an object, contained within an array.

6.1.1  Starting small: JSON data as Python dictionary

In this section, we cover a brief introduction to the JSON format and how we can build a mental model of the data with Python data structures. Following this, we validate our intuition by parsing a small JSON message in Python.

JSON data is a relatively new data interchange format that became massively popular for its readability and it’s relatively small size. JSON stands for JavaScript Object Notation, a fitting name considering that each JSON file can be thought of a JavaScript object. The official JSON website (json.org) contains a more formal introduction to the JSON data format. Since we focus on the Python programming language, I will try to frame my exploration of the JSON spec through the lenses of the Python family of data structures.

Looking at listing 6.1 and in figure 6.1, we notice that our document starts with an opening bracket, {. Every valid JSON document starts with an object, which is JavaScript uses the bracket as a delimiter. In Python, the direct correspondance for an object, as far as JSON goes, it the dictionary. We can therefore think of objects in a JSON document being akin to a Python dictionary. We call the top-level object the root object or root element.

A JSON object (or a Python dictionary) both have keys and values. In our small example, we notice that the keys in our JSON object are all strings. According to the JSON specification, the keys of a JSON object must be a string. Python dictionaries don’t have that limitation, but we can adapt without any problem.

Figure 6.1. A simple JSON object, illustrating its main components: the root object, the keys and the values.

Finally, the values of a JSON object have a little more flexibility about the kind of data they can represent. JSON allows for the following value types.

1. strings (which use the double quote character " as a quoting character);

2. numbers (JavaScript does not make a difference between integers and floating point numbers);

3. booleans (true or false, which are not capitalized like in Python)

4. null, which is akin to the Python None;

5. arrays, which are delimited by the square bracket [ . They are akin to the Python list, and;

6. objects.

If you make the switch between the JSON terms and the Python terms (arrays to list and objects to dictionaries), working with JSON will be a breeze in Python. To finish our analogy, I read in listing 6.2 my simple JSON object using the json module, available in the Python standard library.

Listing 6.2. Reading a simple JSON document as a Python dictionary

import json  ❶

sample_json = """{

"id": 143,

"name": "Silicon Valley",

"type": "Scripted",

"language": "English",

"genres": [

"Comedy"

],

"network": {

"id": 8,

"name": "HBO",

"country": {

"name": "United States",

"code": "US",

"timezone": "America/New_York"

}

}

}"""

document = json.loads(sample_json)

print(document)  ❷

# {'id': 143,

#  'name': 'Silicon Valley',

#  'type': 'Scripted',

#  'language': 'English',

#  'genres': ['Comedy'],

#  'network': {'id': 8,

#   'name': 'HBO',

#   'country': {'name': 'United States',

#    'code': 'US',

#    'timezone': 'America/New_York'}}}

type(document)

# dict  ❸

❶

I import the  json module, available within the Python standard library.

❷

Our loaded document looks like a Python dictionary with string keys. Python recognized that  8 was an integer and parsed the number as a integer.

❸

Our loaded document is of type  dict.

In this section, I introduced how the JSON object can be thought of a limited Python dictionary. Keys are always strings, and values can take string, numerical, boolean or null values. You can also have arrays of values or objects as value, which enables nesting and hierarchical organization of the data. Now that we understand how it works in Python, the next sections will show how to read JSON data using PySpark and will introduce the most complex data frame schema we’ve encountered so far. Before you know, you’ll conquer the schemapocalypse!

6.1.2  Going bigger: reading JSON data in PySpark

This section introduces reading JSON data using the specialized JSON SparkReader object. We discuss the most common and useful parameters of the reader.

For this section, we will take the data introduced at the beginning of the chapter. We read the JSON document in one fell swoop, using the specialized SparkReader object. The result is available in Listing 6.3.

Listing 6.3. Ingesting a JSON document using the JSON specialized SparkReader and printing some metadata about the resulting data frame.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

shows = spark.read.json("../../data/Ch06/shows-silicon-valley.json")

shows.count()

# 1  ❶

❶

The document I ingested contains only a single record.

Two elements pop to mind when reviewing the code.

First, we do not use any optional parameter. JSON data doesn’t have the delimiter or data casting (is the value 03843 a number of a string?) which reduces the need to doctor the reading process by a fair bit. Still, a couple options are noteworthy. For the complete list, I recommend reading the docstring of the spark.read.json method or reviewing the API doc.

- dateFormat and timestampFormat will help when reading dates and timestamps. JSON is of no help with representing those two data types, so PySpark will allow to provide a format to parse date and time information. The default value is the very reasonable ISO8601 date and time standard: yyyy-MM-dd for the date and yyyy-MM-dd’T’HH:mm:ss.SSSXXX for the timestamp (the 'T is a literal T character). We will use this option in listing 6.19.

- prefersDecimal is a boolean flag that, when set to True, will force every numerical value to be interpreted as a decimal (if possible). Decimal data types have a predefined number of significant digits and are very useful when dealing with money where precision counts.

Many other options are available for relaxing the JSON specification (allowing single quotes for strings, comments, or unquoted keys, for instance). If your JSON document is "up-to-spec" and you have no special need for some values not covered within the data types that JSON provided, the stock reader will work fine. For when the data is less than pristing, the options to bend the reader to your will are there, ready to assist.

The second odd thing about our data ingestion is that we have only a single record. In our case, it makes some sense, as we have a single document. In the PySpark world, reading JSON follows this rule: one JSON document, one line, one record. This means that if you want to have multiple JSON records in the same document, you need to have one document per line, and no newline within your document. This is also calles the JSON Lines document format (jsonlines.org/).

If you want to ingest multiple documents across multiple files, you need to set the multiLine (careful about the capital L!) parameter to True. This will change the JSON reading rule to the following: one JSON document, one FILE, one record. With this , you can use the glob pattern (or pass a directory containing only JSON files with the same schema as argument to the reader) and avoid the newline rule. I have made available two more shows in the data/Ch06 directory (Breaking Bad and The Golden Girls, to cover a wide gamut). In listing 6.4, I read the three JSON documents in one fell swoop and show that I indeed have 3 records.

Listing 6.4. Reading multiple JSON documents (one per file) using the multiLine option.

three_shows = spark.read.json("../../data/Ch06/shows-*.json", multiLine=True)

three_shows.count()

# 3

This section covered how to import a simple JSON document in PySpark and how we can tweak the specialized JSON reader to accomodate common use cases. We will focus in the next section on the complex fields, such as _embedded, _links, externals, genres, and others.

6.2  BREAKING THE SECOND DIMENSION WITH COMPLEX DATA TYPES

This section takes the JSON data model and applies it in the context of the PySpark data frame. I go a little deeper in PySpark’s complex data types, the array and the map. I take PySpark’s columnar model and translate it to hierarchical data models. At the end of this section, the PySpark schema will have no secrets for you!

PySpark’s ability to use complex types inside the data frame is what allows its remarkable flexibility. While you still have the tabular abstraction to work with, your cells are supercharged since they can contain more than just a value. It’s just like going from 2D to 3D, and even beyond!

A complex type isn’t really complex: I often use the term container or compound type and will interchangeably during the book. In a nutshell, the difference between them and simple/scalar types is their ability to contain more than a single value. In Python, the main complex types are the list, the tuple, and the dictionary. In PySpark, we have the array, the map, and the struct. With those 3, you will be able to express an infinite amount of data layout.

No type left behind: if you want to dig deeper into scalar data types

Scalar types are still the bread and butter of data manipulation in PySpark, but since they map so seamlessly with Python types, I didn’t want to describe them for the sake of describing them. Instead, this book will introduce peculiarities of Spark’s data types as needed, when it makes the most sense. In appendix E, there is a more thorough and coherent — if a little dry — introduction to each scalar data type.

I think I’ve held the punch for long enough: behold, listing 6.5 reveals our data frame’s schema!

Listing 6.5. Our shows data frame’s schema (subset) is all it’s complex splendor. Not that PySpark displays the hierarchy in a clear fashion.

shows.printSchema()

# root  ❶

#  |-- _embedded: struct (nullable = true)

#  |    |-- episodes: array (nullable = true)

#  |    |    |-- element: struct (containsNull = true)

#  |    |    |    |-- _links: struct (nullable = true)

#  |    |    |    |    |-- self: struct (nullable = true)

#  |    |    |    |    |    |-- href: string (nullable = true)

#  |    |    |    |-- airdate: string (nullable = true)

#  |    |    |    |-- airstamp: string (nullable = true)

#  |    |    |    |-- airtime: string (nullable = true)

#  |    |    |    |-- id: long (nullable = true)

#  |    |    |    |-- image: struct (nullable = true)

#  |    |    |    |    |-- medium: string (nullable = true)

#  |    |    |    |    |-- original: string (nullable = true)

#  |    |    |    |-- name: string (nullable = true)

#  |    |    |    |-- number: long (nullable = true)

#  |    |    |    |-- runtime: long (nullable = true)

#  |    |    |    |-- season: long (nullable = true)

#  |    |    |    |-- summary: string (nullable = true)

#  |    |    |    |-- url: string (nullable = true)

#  |-- _links: struct (nullable = true)

#  |    |-- previousepisode: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |    |-- self: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |-- externals: struct (nullable = true)

#  |    |-- imdb: string (nullable = true)

#  |    |-- thetvdb: long (nullable = true)

#  |    |-- tvrage: long (nullable = true)

#  |-- genres: array (nullable = true)

#  |    |-- element: string (containsNull = true)

#  |-- id: long (nullable = true)

# [and more columns...]

I had to truncate the schema so we can focus on the important point here: the hierarchy within the schema. PySpark every top-level keys — the keys from the root object — and parsed them as column (see listing 6.6 for the top level columns). When a column had a scalar value, the type was inferred according to the JSON specification we saw in 6.1.1.

Listing 6.6. Printing the columns of the shows data frame. Only the top-level columns are returned.

print(shows.columns)

# ['_embedded', '_links', 'externals', 'genres', 'id', 'image',

#  'language', 'name', 'network', 'officialSite', 'premiered',

#  'rating', 'runtime', 'schedule', 'status', 'summary', 'type',

#  'updated', 'url', 'webChannel', 'weight']

6.2.1  When you have more than one value: the array

In this section, I introduce the simplest container type in PySpark: the array. I explain where the array is most commonly used as well as the main methods to create it, operate it and break it down.

In 6.1.1, I loosely equated a JSON array to a Python list. In the PySpark world, the same follows, with an important distinction: PySpark arrays are container for values of the same type. This precision has an important impact on how PySpark ingests both JSON documents and, more generally, nested structures, so I’ll explain in more details.

In listing 6.5, the genres array points to an element item, which is of type string (I have reproduced the relevant section below). Like any other types within the data frame, we need to provide a complete type story for any complex type, including the array. With this loss of flexibility in what an array can contain, we gain a better grasp of the data contained within the column and can avoid hard to track bugs.

|-- genres: array (nullable = true)

|    |-- element: string (containsNull = true)

Warning

PySpark will not raise an error if you try to read an array-type column with multiple types instead. Instead, it will simply default to the lowest common denominator, usually the string. This way, your don’t lose any data, but you need to take this into account when developping.

To work a little with the array, I select a subset of the shows data frame as to not lose focus in this huge schema. In listing 6.7, I select the name and genres columns and show the record. Unfortunately, Silicon Valley is a single genre show, so our array is a little too basic for my taste. Lets make it a little more interesting.

Listing 6.7. Selecting the name and genres columns of the shows dataframe, to bring focus to the array column.

array_subset = shows.select("name", "genres")

array_subset.show(1, False)

# +--------------+--------+

# |name          |genres  |

# +--------------+--------+

# |Silicon Valley|[Comedy]|

# +--------------+--------+

Conceptually, our genres column can be thought of containing lists of elements within each record. In Chapter 2, we had a similar situation with breaking our lines into words. Visually, it looks like figure 6.2.

Figure 6.2. Figure caption

We start by extracting the value from the array. Just like in Python lists, PySpark array elemest are accessible by index. In listing 6.8, I show the main ways to access the (only) element in my array. Arrays are zero-indexed when retrieving elements inside, just like Python lists. Unlike Python lists, passing an index that would go beyond the content of the list returns null.

Listing 6.8. Listing title

import pyspark.sql.functions as F

array_subset = array_subset.select(

"name",

array_subset.genres[0].alias("dot_and_index"),  ❶

F.col("genres")[0].alias("col_and_index"),

array_subset.genres.getItem(0).alias("dot_and_method"),  ❷

F.col("genres").getItem(0).alias("col_and_method"),

)

array_subset.show()

# +--------------+-------------+-------------+--------------+--------------+

# |          name|dot_and_index|col_and_index|dot_and_method|col_and_method|

# +--------------+-------------+-------------+--------------+--------------+

# |Silicon Valley|       Comedy|       Comedy|        Comedy|        Comedy|

# +--------------+-------------+-------------+--------------+--------------+

❶

Using the dot notation, and the usual square bracket with the index inside.

❷

Instead of the index in square bracket syntax, we can use the  getItem() method on the Column object.

Warning

Although the square bracket approach looks very Pythonic, you can’t use it as a slicing tool. PySpark will accept only one integer as an index, so  array_subset.genres[0:10] will return an AnalysisException with a cryptic error message.

PySpark’s array functions — available in the pyspark.sql.functions module — are almost all prefixed with the array_ keyword (some, like size() in listing 6.9, can be applied to more than one complex type and therefore are not prefixed). It is therefore pretty easy to review them in one fell swoop in the API documentation. Next, we use functions to create a beefier array and do a little exploration with it. In listing 6.9 I perform the following tasks.

1. I create three litteral columns (using lit() to create scalar columns, then make_array() to ) to create an array of possible genres. PySpark won’t accept Python lists as an argument to lit(), so we have to do the long trip. See Chapter 8 for how to create UDF that can return array columns.

2. I then use the function array_repeat() to create a column repeating the "Comedy" string we’ve extracted in listing 6.8 five times. I finally compute the size of both columns, dedupe both array, and intersect them, yielding our original [Comedy] array from listing 6.7.

Listing 6.9. Listing title

array_subset_repeated = array_subset.select(

"name",

F.lit("Comedy").alias("one"),

F.lit("Horror").alias("two"),

F.lit("Drama").alias("three"),

F.col("dot_and_index"),

).select(

"name",

F.array("one", "two", "three").alias("Some_Genres"),

F.array_repeat("dot_and_index", 5).alias("Repeated_Genres"),

)

array_subset_repeated.show(1, False)

# +--------------+-----------------------+----------------------------------------+

# |name          |Some_Genres            |Repeated_Genres                         |

# +--------------+-----------------------+----------------------------------------+

# |Silicon Valley|[Comedy, Horror, Drama]|[Comedy, Comedy, Comedy, Comedy, Comedy]|

# +--------------+-----------------------+----------------------------------------+

array_subset_repeated.select(

"name", F.size("Some_Genres"), F.size("Repeated_Genres")

).show()

# +--------------+-----------------+---------------------+

# |          name|size(Some_Genres)|size(Repeated_Genres)|

# +--------------+-----------------+---------------------+

# |Silicon Valley|                3|                    5|

# +--------------+-----------------+---------------------+

array_subset_repeated.select(

"name", F.array_distinct("Some_Genres"), F.array_distinct("Repeated_Genres")

).show(1, False)

# +--------------+---------------------------+-------------------------------+

# |name          |array_distinct(Some_Genres)|array_distinct(Repeated_Genres)|

# +--------------+---------------------------+-------------------------------+

# |Silicon Valley|[Comedy, Horror, Drama]    |[Comedy]                       |

# +--------------+---------------------------+-------------------------------+

array_subset_repeated = array_subset_repeated.select(

"name", F.array_intersect("Some_Genres", "Repeated_Genres").alias("Genres")

)

array_subset_repeated.show()

# +--------------+--------+

# |          name|  Genres|

# +--------------+--------+

# |Silicon Valley|[Comedy]|

# +--------------+--------+

One potentially confusing behavior of PySpark working with arrays happens when using the array_position(). The function will return the position of the item in an array if it exists, starting with 1 (not 0, like in Python). This is illustrated in listing 6.10. I remember this difference by calling the position via getItem() or the square brackets index, versus position for the return value of the array_position() function, just like in the PySpark API.

Listing 6.10. Listing title

array_subset_repeated.select("Genres", F.array_position("Genres", "Comedy")).show()

# +--------+------------------------------+

# |  Genres|array_position(Genres, Comedy)|

# +--------+------------------------------+

# |[Comedy]|                             1|

# +--------+------------------------------+

In this section, we looked at the array using our shows data frame. We saw that a PySpark array contains elements of the same time, and an array column has access to some container functions (such as size()) as well as a handful of array specific functions, prefixed by array_. The next section will introduce an equally useful but more less frequently used complex type, the map.

6.2.2  The map type: keys and values within a column

This section covers the map column type and where it can be used succesfully. Maps are less common as a column type — reading a JSON document won’t yield columns of type map but they are very useful nonetheless.

A map is conceptually very close to a Python typed dictionary: you have keys and values just like in a dictionary, but just like with the array, your keys need to be of the same type and you values need to be of the same type (the type for the keys can be different than the type for the values).

One of the easiest way to create a map is from two columns of type array. We will do so by collecting some information about the name, language, type and url columns into an array and using the map_from_arrays() function, like in listing 6.11

Listing 6.11. Creating a map from two arrays: one for the keys, one for the values. This create a hash-map within the column record.

columns = ["name", "language", "type"]

shows_map = shows.select(

*[F.lit(column) for column in columns], F.array(*columns).alias("values")

)

shows_map = shows_map.select(F.array(*columns).alias("keys"), "values")

shows_map.show(1)

# +--------------------+--------------------+

# |                keys|              values|

# +--------------------+--------------------+

# |[name, language, ...|[Silicon Valley, ...|

# +--------------------+--------------------+

shows_map = shows_map.select(F.map_from_arrays("keys", "values").alias("mapped"))

shows_map.printSchema()

# root

#  |-- mapped: map (nullable = false)

#  |    |-- key: string

#  |    |-- value: string (valueContainsNull = true)

shows_map.show(1, False)

# +---------------------------------------------------------------+

# |mapped                                                         |

# +---------------------------------------------------------------+

# |[name -> Silicon Valley, language -> English, type -> Scripted]|

# +---------------------------------------------------------------+

shows_map.select(

F.col("mapped.name"),  ❶

F.col("mapped")["name"],  ❷

shows_map.mapped["name"],  ❸

).show()

# +--------------+--------------+--------------+

# |          name|  mapped[name]|  mapped[name]|

# +--------------+--------------+--------------+

# |Silicon Valley|Silicon Valley|Silicon Valley|

# +--------------+--------------+--------------+

❶

Annotation

❷

Annotation

❸

Annotation

Just like with the array, PySpark provides a few functions to work with maps under the pyspark.sql.functions module. Most of them are prefixed or suffixed with map, such as map_values() (creates an array column out of the map values) or create_map() (create a map from the columns passed as a parameter, alternating between keys and values). The exercises at the end of this section and at the end of the chapter provide more practice with the map column type.

If the map maps (pun intended) to a Python dictionary, why did our JSON document didn’t have any maps? Because maps keys and values need to be the same type respectively — something JSON objects are not forced to — we need a more flexible container to accomodate objects. It’s also much more useful to have the top level name/value pairs as colums, like PySpark did with our shows data frame in listing 6.3. The next section will introduce the struct, which is the backbone of the data frame as we know it.

6.3  THE STRUCT: NESTING COLUMNS WITHIN COLUMS

This section covers the struct as a column type, but also as the foundation of the dataframe. We look at how we can reason about our data frame in terms of structs and how to navigate a data frame with nested structs.

The struct is akin to a JSON object, in the sense that the key or name of each pair is a string and that each record can be of a different type. If we take a small subset of the colums of our data frame, like in listing 6.12, we can see that the schedule column contains two fields.

1. days, which is an array of strings.

2. time, which is a string.

Listing 6.12. The schedule column or the shows data frame contains two fields, an array of strings and a string.

shows.select("schedule").printSchema()

# root

#  |-- schedule: struct (nullable = true)

#  |    |-- days: array (nullable = true)

#  |    |    |-- element: string (containsNull = true)

#  |    |-- time: string (nullable = true)

The struct is very different from the array and the map in that the number of fields and their names are known ahead of time. In our case, the schedule struct column is fixed: we know that each record of our data frame will contain that `schedule struct (or a null value, if we want to be pedantic), and within that struct, there will be an array of strings days and a string time. The array and the map enforce the type of the values, but not their number or names. The struct allows for more versatility of types, as long as you name each field and provide the type ahead of time.

Conceptually, I find that the easiest way to think about the struct column type is to imagine a small data frame within your column records. Taking our example in listing 6.12, we can visualize that schedule is actually a data frame of two columns (days and time) trapped within the column. I illustrated the nested column analogy in figure 6.3.

Figure 6.3. Figure caption

Structs are nestable within one another. The first field of our data frame, _embedded, is a struct that contains an array field, episodes. That array contains structs _links that contains a struct self, which contains a string field, href. That is why I call a pretty confusing nesting of struct. Don’t be worried if this is still a little confusing: the next section will decypher the nesting dolls arrangement of structs by navigating our data frame.

6.3.1  Navigating structs as if they were nested columns

This section covers how to extract values from nested structs inside a data frame. I will cover the dot notation, the bracket notation, and explain how PySpark treats nesting when using other complex structures. We will work with the _embedded column by cleaning the useless nesting.

Before going all hands on keyboard, we’ll draft the structure of the _embedded column as a tree to get a sense of what we’re working with. I provide in listing 6.13 the output of the printSchema() command which I drew in figure 6.4.

Listing 6.13. Listing title

shows.select(F.col("_embedded")).printSchema()

# root

#  |-- _embedded: struct (nullable = true)

#  |    |-- episodes: array (nullable = true)

#  |    |    |-- element: struct (containsNull = true)

#  |    |    |    |-- _links: struct (nullable = true)

#  |    |    |    |    |-- self: struct (nullable = true)

#  |    |    |    |    |    |-- href: string (nullable = true)

#  |    |    |    |-- airdate: string (nullable = true)

#  |    |    |    |-- id: long (nullable = true)

#  |    |    |    |-- image: struct (nullable = true)

#  |    |    |    |    |-- medium: string (nullable = true)

#  |    |    |    |    |-- original: string (nullable = true)

#  |    |    |    |-- name: string (nullable = true)

#  |    |    |    |-- number: long (nullable = true)

#  |    |    |    |-- runtime: long (nullable = true)

#  |    |    |    |-- season: long (nullable = true)

#  |    |    |    |-- summary: string (nullable = true)

#  |    |    |    |-- url: string (nullable = true)

Figure 6.4. Figure caption

For starters, we see in figure 6.4 that _embedded is a useless struct, as it contains only one field. Ie listing 6.14, I create a new top-level column called episodes which refers directly to the episodes field in the _embedded struct. For this, I use the dot notation _embedded.episodes. This is consistent with the "struct as a mini data frame" mental model: you can refer to struct fields using the same notation as you would for a data frame. You can also use bracket notation like shows["_embedded"]["episodes"] or even the col function col("_embedded.episodes"). As I mentioned in chapter 2, I prefer the col function because it keeps the column names independent from the data frame name, which is convenient when you build your code from the bottom up. You can create multiple intermediate data frames and then merge your data manipulation pipeline into a single chain of transformations without having to rename every intermediate reference. This is a question of personal preference and I sometimes use the dot notation since it works with autocomplete.

Listing 6.14. Listing title

shows_clean = shows.withColumn("episodes", shows._embedded.episodes).drop("_embedded")

shows_clean.printSchema()

# root

#  |-- _links: struct (nullable = true)

#  |    |-- previousepisode: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |    |-- self: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |-- externals: struct (nullable = true)

#  |    |-- imdb: string (nullable = true)

#  [...]

#  |    |-- element: struct (containsNull = true)

#  |    |    |-- _links: struct (nullable = true)

#  |    |    |    |-- self: struct (nullable = true)

#  |    |    |    |    |-- href: string (nullable = true)

#  |    |    |-- airdate: string (nullable = true)

#  |    |    |-- airstamp: string (nullable = true)

#  |    |    |-- airtime: string (nullable = true)

#  |    |    |-- id: long (nullable = true)

#  |    |    |-- image: struct (nullable = true)

#  |    |    |    |-- medium: string (nullable = true)

#  |    |    |    |-- original: string (nullable = true)

# [... rest of schema]

Tip

If you want to unpack all the fields in a struct, use the  * field, just like when you select all columns in a data frame. For instance, if you have a struct image with multiple fields you want to unpack, you can use image.*.

Now, we look at drilling through structs nested in arrays. In 6.2.1, I explained that we can refer to individual elements in the array using the index in brackets after the column reference. What about extracting the names of all the episodes, which are within the episodes array of structs?

Turns out PySpark will allow you to drill within an array and will return the subset of the struct still in an array form. This is best explained by an example: in listing 6.15, I extract the episodes.name field from the shows_clean data frame. Since episodes is an array of struct and name one of the string fields, then episodes.name is an array of strings.

Listing 6.15. Listing title

episodes_name = shows_clean.select(F.col("episodes.name"))

episodes_name.printSchema()

# root

#  |-- name: array (nullable = true)

#  |    |-- element: string (containsNull = true)

episodes_name.select(F.explode("name").alias("name")).show(3, False)

# +-------------------------+

# |name                     |

# +-------------------------+

# |Minimum Viable Product   |

# |The Cap Table            |

# |Articles of Incorporation|

# +-------------------------+

This section walked through the struct hierarchy using the same notation as one would use for extracting columns from a data frame. We now can extract any field from our JSON document and know exactly what to expect. The next section will leverage our knowledge of complex data types and use it in crafting a schema. I also touch on the advantages and trade-offs or using hierarchical schemas and complex data types.

6.4  BUILDING AND USING THE DATA FRAME SCHEMA

In this section, I cover how to define and use a schema with a PySpark data frame. We build the schema for our JSON object programatically and review the types PySpark offers out of the box.

In 6.3, I explained that we can think of a struct column as a mini data frame nested in said column. The opposite also works: you can think of a data frame as having a single struct entity, with the columns being the top-level fields of the "root" struct. In any output of printSchema() (I reproduced the relevant part of printSchema in listing 6.16 for convenience), all the top level fields are connected to the root.

Listing 6.16. Listing title

shows.printSchema()

# root  ❶

#  |-- _links: struct (nullable = true)

#  |    |-- previousepisode: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |    |-- self: struct (nullable = true)

#  |    |    |-- href: string (nullable = true)

#  |-- externals: struct (nullable = true)

#  |    |-- imdb: string (nullable = true)

#  [... rest of schema]

❶

All the top-level fields (or columns) are children of a  root implicit struct.

There are two syntaxes you can use to create a schema. In the next section, we review the explicit, programmatic one. PySpark also accepts a DDL-style (Data Definition Language) schema which is covered in Chapter 7 when we discuss PySpark and SQL.

6.4.1  Using Spark types as the base blocks of a schema

In this section, I covers the column types in the context of schema definition. I build the schema for our shows data frame from scratch and brush on some programatic niceties of the PySpark schema-building capabilities.

The types objects we use to build a schema are located in the pyspark.sql.types module. They are such a frequent import when working with data frames that, just like function, as usually imported with the qualified prefix T.

Tip

Just like with functions using a capital  F, the common convention is to use a capital T when importing the types module. I strongly recommend following the same.

Within the pyspark.sql.types, there are two main kind of objects.

First, you have the types object, representing a column of a certain type. All of those objects follow the ValueType(): for instance, a long column would be represented by a LongType() object. Most scalar types do not take any parameters (with the exception of the DecimalType(precision, scale), which is covered in Appendix E). Complex types, such as the array and the map, take the types of their values directly in the constructor. For example, an array of strings would be ArrayType(StringType()) and a map of strings mapping to longs would be MapType(StringType(), LongType()).

Second, you have the field object, a.k.a. the StructField(). PySpark provides a StructType that can contain an arbitrary number of named fields; programatically, this translates to a StructType() taking a list of StructField(). Easy as that!

A StructField() contains two mandatory parameters as well as two optional.

1. The name of the field, passed as a string.

2. The dataType of the field, passed as a type object.

3. (Optional) a nullable flag, which determines if the field can be null or not (by default True)

4. (Optional) a metadata dictionary which contains arbitrary information. We will use the metadata when working with ML pipelines (Chapter 14).

Putting all this together, the summary string field of the shows data frame would be encoded in a StructField like so.

T.StructField("summary", T.StringType())

In listing 6.17, I’ve done the _embedded schema of the shows data frame. While very verbose, we gain an intimate knowledge of the data frame structure. Since the data frame schemas are regular Python classes, we can assign them to variables and build our schema from the bottom up. I usually split the structs containing more than three or so fields in their own variables, so my code doesn’t read like a whole block of structs intersperced with brackets. The full schema ingestion code is available at the end of the section if you are interested. I’ve assigned it to shows_schema since we will need it for reading our data frame once more.

Listing 6.17. Listing title

import pyspark.sql.types as T

episode_links_schema = T.StructType(

[T.StructField("self", T.StructType([T.StructField("href", T.StringType())]))]

)

episode_image_schema = T.StructType(

[

T.StructField("medium", T.StringType()),

T.StructField("original", T.StringType()),

]

)

episode_schema = T.StructType(

[

T.StructField("_links", episode_links_schema),

T.StructField("airdate", T.DateType()),

T.StructField("airstamp", T.TimestampType()),

T.StructField("airtime", T.StringType()),

T.StructField("id", T.StringType()),

T.StructField("image", episode_image_schema),

T.StructField("name", T.StringType()),

T.StructField("number", T.LongType()),

T.StructField("runtime", T.LongType()),

T.StructField("season", T.LongType()),

T.StructField("summary", T.StringType()),

T.StructField("url", T.StringType()),

]

)

embedded_schema = T.StructType(

[T.StructField("episodes", T.ArrayType(episode_schema))]  ❶

)

❶

Annotation

This section covered how to build a schema from the bottom up: you can use the types and field from the pyspark.sql.types module and create one field for each column. When you have a struct column, you treat it the same way: create a StructType() and assign struct fields. With those simple rules, you should be able to construct any schema you need. The next section will leverage our schema to read the JSON is a strict fashion.

Listing 6.18. Listing title

episode_links_schema = T.StructType(

[T.StructField("self", T.StructType([T.StructField("href", T.StringType())]))]

)

episode_image_schema = T.StructType(

[

T.StructField("medium", T.StringType()),

T.StructField("original", T.StringType()),

]

)

episode_schema = T.StructType(

[

T.StructField("_links", episode_links_schema),

T.StructField("airdate", T.DateType()),

T.StructField("airstamp", T.TimestampType()),

T.StructField("airtime", T.StringType()),

T.StructField("id", T.StringType()),

T.StructField("image", episode_image_schema),

T.StructField("name", T.StringType()),

T.StructField("number", T.LongType()),

T.StructField("runtime", T.LongType()),

T.StructField("season", T.LongType()),

T.StructField("summary", T.StringType()),

T.StructField("url", T.StringType()),

]

)

embedded_schema = T.StructType([T.StructField("episodes", T.ArrayType(episode_schema))])

network_schema = T.StructType(

[

T.StructField(

"country",

T.StructType(

[

T.StructField("code", T.StringType()),

T.StructField("name", T.StringType()),

T.StructField("timezone", T.StringType()),

]

),

),

T.StructField("id", T.LongType()),

T.StructField("name", T.StringType()),

]

)

shows_schema = T.StructType(

[

T.StructField("_embedded", embedded_schema),

T.StructField("language", T.StringType()),

T.StructField("name", T.StringType()),

T.StructField("network", network_schema),

T.StructField("officialSite", T.StringType()),

T.StructField("premiered", T.StringType()),

T.StructField(

"rating", T.StructType([T.StructField("average", T.DoubleType())])

),

T.StructField("runtime", T.LongType()),

T.StructField(

"schedule",

T.StructType(

[

T.StructField("days", T.ArrayType(T.StringType())),

T.StructField("time", T.StringType()),

]

),

),

T.StructField("status", T.StringType()),

T.StructField("summary", T.StringType()),

T.StructField("type", T.StringType()),

T.StructField("updated", T.LongType()),

T.StructField("url", T.StringType()),

T.StructField("webChannel", T.StringType()),

T.StructField("weight", T.LongType()),

]

)

6.4.2  Reading a JSON document with a strict schema in place

Besides codifying the schema of my JSON document in terms of the data frame representation, I also casted airdate as a DateType() and airstamp as a TimestampType(). In 6.1.2, I discussed that the specialized JSON reader has options to read certain strings as date and timestamp. In order to do so, you need to provide a schema: good thing we have one ready. In listing 6.19>, I read my JSON document once more, but this time I provide an explicit schema. Note the change in type for airdate and airstamp. I also provide a new parameter mode, which when set to FAILFAST will error if it encounters a malformed record versus the schema provided.

Listing 6.19. Listing title

shows_with_schema = spark.read.json(

"../../data/Ch06/shows-silicon-valley.json",

schema=shows_schema,  ❶

mode="FAILFAST",  #  ❷

)

❶

Annotation

❷

Annotation

A succesful read is promising, but since I want to verify my new date and timestamp field, I drill, explode and show the fields in listing 6.20.

Listing 6.20. Listing title

for column in ["airdate", "airstamp"]:

shows.select(f"_embedded.episodes.{column}").select(F.explode(column)).show(5)

# +----------+

# |       col|

# +----------+

# |2014-04-06|

# |2014-04-13|

# |2014-04-20|

# |2014-04-27|

# |2014-05-04|

# +----------+

# only showing top 5 rows

# +-------------------+

# |                col|

# +-------------------+

# |2014-04-06 22:00:00|

# |2014-04-13 22:00:00|

# |2014-04-20 22:00:00|

# |2014-04-27 22:00:00|

# |2014-05-04 22:00:00|

# +-------------------+

# only showing top 5 rows

Everything looks fine and dandy. We were lucky since our JSON document followed the same convention as Spark for date and timestamp string representation. If you are not that lucky, you can easily set the dateFormat and timestampFormat using the patterns information available at the Spark official documentation (spark.apache.org/docs/latest/sql-ref-datetime-pattern.html).

What happens if the schema does not match? PySpark, even in FAILFAST, will allow absent fields in the document if the schema allows for null values. In listing 6.21, I pollute my schema, changing two StringType() to LongType(). I did not include the whole stacktrace, but the resulting error is a Py4JJavaError (refer to Chapter 1 for a quick intro to Py4J) that hits it right on the head: our string value is not a bigint (or long). You won’t know which one, though: the stack trace just gives what it tried to parse and what it expected.

Listing 6.21. Listing title

from py4j.protocol import Py4JJavaError

shows_schema2 = T.StructType(

[

T.StructField("_embedded", embedded_schema),

T.StructField("language", T.StringType()),

T.StructField("name", T.StringType()),

T.StructField("network", network_schema),

T.StructField("officialSite", T.StringType()),

T.StructField("premiered", T.StringType()),

T.StructField(

"rating", T.StructType([T.StructField("average", T.DoubleType())])

),

T.StructField("runtime", T.LongType()),

T.StructField(

"schedule",

T.StructType(

[

T.StructField("days", T.ArrayType(T.StringType())),

T.StructField("time", T.StringType()),

]

),

),

T.StructField("status", T.StringType()),

T.StructField("summary", T.StringType()),

T.StructField("type", T.LongType()),  ❶

T.StructField("updated", T.LongType()),

T.StructField("url", T.LongType()),  ❷

T.StructField("webChannel", T.StringType()),

T.StructField("weight", T.LongType()),

]

)

shows_with_schema_wrong = spark.read.json(

"../../data/Ch06/shows-silicon-valley.json", schema=shows_schema2, mode="FAILFAST",

)

try:

shows_with_schema_wrong.show()

except Py4JJavaError:

pass

# Huge Spark ERROR stacktrace, relevant bit:

#

# Caused by: java.lang.RuntimeException: Failed to parse a value for data type

#   bigint (current token: VALUE_STRING).

❶

Annotation

This section was a very short one but is incredibly useful nonetheless. We saw how to use the schema information to create a strict

FAILFAST: When do you want to get in trouble?

It seems a little paranoid to FAILFAST while setting a verbose schema all by hand. Unfortunately, data is messy, people can be sloppy, and when you rely on data to make decisions, garbage in, garbage out.

In my professional career, I’ve encountered data integrity problems so often when reading data that I am now a firm believer that you need to diagnose them as early as possible. FAILFAST mode is one example: by default, PySpark will set malformed records to null (the PERMISSIVE approach). When exploring, I consider this to be perfectly legitimate. But I’ve had enough sleepless night having a business stakeholder calling me at the last minute because "results are weird" to try to minimize data drama at every opportunity.

Chapter 13, which talks about testing PySpark code, also has a section about defensive data coding. When bad data comes in, ask yourself "when do I want to deal with it?". From where I stand, the earlier, the better.

6.4.3  Going full circle: specifying your schemas in JSON

6.5  PUTTING IT ALL TOGETHER: REDUCING DUPLICATE DATA WITH COMPLEX DATA TYPES

6.6  SUMMARY

6.7  EXERCISES

7

BILINGUAL PYSPARK: BLENDING PYTHON AND SQL CODE

This chapter covers:

- How PySpark’s own data manipulation module takes inspiration from SQL’s vocabulary and way of doing things.

- How to register data frames as temporary views or tables to query them using Spark SQL.

- How the catalog stores metadata about registered tables and views, how to list the existing references and delete them.

- How common data manipulations are expressed in PySpark and Spark SQL and how you can move from one to the other.

- How to use SQL-style clauses inside certain PySpark methods.

My answer to "Python versus SQL, which one should I learn?" is "yes".

When it comes to manipulating tabular data, SQL is the reigning king. For multiple decades now, it has been the workhorse language for relational databases, and even today, learning how to tame it is a worthwhile exercise. Spark acknowledge the power of SQL heads on: you can use a mature SQL API to transform data frames. On top of that, you can also seamlessly blend SQL code withing your Spark or PySpark program, making it easier than ever to migrate those old SQL ETL jobs without reinventing the wheel.

This chapter is dedicated on SQL interop with PySpark. I will cover how we can move from one language to the other. I will also cover how we can use a SQL-like syntax within data frame methods to speed up your code and some of trade-offs you can face. Finally, we’ll blend Python and SQL code together to get the best of both worlds.

If you already have notable exposure to SQL, this chapter will be a breeze for you. Feel free to skim over the SQL-specific sections, but don’t skip the sections on Python and SQL interop, as there is some PySpark idiosyncrasies I cover there. For those brand new to SQL, this will be — I hope — an eye opener moment and you’ll add another tool under your belt. If you feel that you’d like a deeper dive into SQL, SQL in motion by Ben Brumm (Manning, 2017) is a good video source. If you prefer a book, a very exhaustive reference is Joe Celko’s SQL for Smarties (Morgan Kauffman, 2014).

In order to follow the examples in this chapter, here are the imports I am using.

from pyspark.sql import SparkSession

from pyspark.sql.utils import AnalysisException  ❶

import pyspark.sql.functions as F

import pyspark.sql.types as T

spark = SparkSession.builder.getOrCreate()

❶

This is a PySpark specific exception I am using when I am writing code that fails on purpose.

Tip

This chapter will not be a deep dive in SQL rather than a comparison with PySpark and a way to combine both. I am leaving certain concepts out on purpose, but it doesn’t mean you can’t use them if you’ve comfortable with SQL.

ANSI SQL vs. HiveQL

Spark supports both ANSI SQL and the vast majority of HiveQLfootnote::[You can see the functionality supported (and unsupported on the Spark website: docs.databricks.com/spark/latest/spark-sql/compatibility/hive.html] as a SQL dialect. Spark SQL also has some Spark specific functions baked in to ensure a common set of functionality across languages.

In a nutshell, Hive is a SQL-like interface that can be used over a variety of data storage. It became very popular because it allowed to query files in HDFS (Hadoop Distributed File System) like if they were a table. Spark can integrate with Hive when your environment has it installed. Spark SQL also provides additional syntax to work with larger datasets, which I will cover as we need it.

Because of the amount of material and its longevity, and also because they are very similar in syntax for basic and intermediate queries, I usually recommend learning ANSI SQL at first and then complete with HiveQL as you go along. This way, your knowledge will transfer to other SQL-based products.

Understanding Hive in depth is something I won’t cover in this book as it is not a component of PySpark. For those who are curious, Practical Hive (Apress, 2016) seems like a good reference.

7.1  BANKING ON WHAT WE KNOW: PYSPARK.SQL VS PLAIN SQL

PySpark’s data frame manipulation hints at its SQL heritage: the name of the module — pyspark.sql — is a dead giveaway. PySpark developers recognized the heritage of the SQL programming language for data manipulation and used the same vocabulary to name their method. Let’s look at a quick example in both SQL and plain PySpark and look at similarities between the keywords used. In listing 7.1, I load a CSV containing information about the periodic table of elements and I query the data set to find the number of entries with a liquid state per period. The code is presented both in PySpark and SQL form, and without much context, we can see similarities between the two.

In figure 7.1, I put the code side by side and illustrate where the similarities start and end. Even though it’s a small example, it shows the main differences between PySpark and SQL.

1. PySpark will always start with the name of the data frame you are working with. SQL refers to the table (or target) using a from keyword;

2. PySpark chains the transformations and actions as methods on the data frame, where SQL splits them into two groups: the operation group and the condition group. The first one is before the from clause and operates on columns. The second is after the from clause and group, filters, and orders the structure of the result table.

Tip

SQL is not case sensitive, so you can either use lower-case or upper-case. I usually prototype in lower-case then convert in upper-case when I am confident about my query, to differentiate the two visually.

elements = spark.read.csv(

"../../data/Ch07/Periodic_Table_Of_Elements.csv",

header=True,

inferSchema=True,

)

elements.where(F.col("phase") == "liq").groupby("period").count().show()

// In SQL: We assume that the data is in a table called `elements`

SELECT

period,

count(*)

FROM elements

WHERE phase = "liq"

GROUP BY period;

Figure 7.1. The same simple statement, represented in PySpark and SQL

Both would return the same results: one element in period four (Bromine) and one in period six (Mercury).

Whether you prefer the order of operations from PySpark and SQL will depend on how you build query mentally and how familiar you are with the respective language. Fortunately, PySpark makes it easy to move from on to the other, and even work with both all at once.

7.2  USING SQL QUERIES ON A DATA FRAME

Since we can think of PySpark data frames like tables on steroids, it’s not farfetched to think about querying them using a language designed to query tables. Spark provides a full SQL API which is documented in the same fashion as the PySpark one (spark.apache.org/docs/latest/api/sql/index.html). We also see that the functions defined in pyspark.sql.functions are defined.

Note

Spark’s SQL API only covers the data manipulation subset of Spark. You won’t be able to do machine learning using SQL, for instance.

7.2.1  Promoting a data frame to a Spark table

PySpark maintains boundaries between its own namespacing and Spark SQL’s namespacing. This means that a data set loaded in PySpark won’t be automatically available to querying using SQL. The code in listing 7.2 shows an example of this.

Listing 7.1. Trying (and failing) at querying a data frame SQL-style

try:

spark.sql(

"select period, count(*) from elements where phase='liq' group by period"

).show(5)

except AnalysisException as e:

print(e)

# 'Table or view not found: elements; line 1 pos 29'

Here, PySpark doesn’t make the link between the python variable elements, which points to the data frame, and a potential table elements that can be queried by Spark SQL. In order to allow a data frame to be queried via SQL, we need to register them as tables. I illustrated the process in figure 7.1 . When we assign a data frame to a variable, Python points to the data frame. Spark SQL does not have visibility over the variables Python assigns.

When you want to create a table to query with Spark SQL, you can use the createOrReplaceTempView() method. This method takes a single string parameter which is the name of the table you want to use. This transformation will look at the data frame referenced by the Python variable on which the method was applied and will create a Spark SQL reference to the same data frame. We see an example of this in the bottom half.

Note

Although you can name the table the same name as the variable you are using, you are not forced to do so. I could have named my table  starneifwpy if I wanted to.

Once we have registered our elements table that point to the same data frame that our Python variable of the same name, we can query our table without any problem. Let’s re-run the same code block as listing 7.2 and see that it succeds.

Note

In this chapter, I use the term "table" and "view" pretty loosely. In SQL, they are distinct concepts: the table is materialized in memory and the view is computed on the fly. Spark’s temp views are conceptually closer to a view than a table. Spark SQL also has tables but we will not be using them, preferring reading and materializing our data into a data frame.

Listing 7.2. Trying (and succeding) at querying a data frame SQL-style

elements.createOrReplaceTempView("elements")  ❶

spark.sql(

"select period, count(*) from elements where phase='liq' group by period"

).show(5)

# +------+--------+

# |period|count(1)|

# +------+--------+

# |     6|       1|

# |     4|       1|

# +------+--------+ ❷

❶

We register our table using the  createOrReplaceTempView() method on the element data frame.

❷

The same query works once Spark is able to dereference the SQL view name.

So now we have a view registered. In the case of a low number of views to manage, it’s pretty easy to keep their name in memory. What about if you have dozens of them or if you need to delete some? Enter the catalog, Spark’s way of managing its SQL namespace.

Advanced-ish topic: Spark SQL views and persistence

TL;DR: Use createOrRepleaceTempView() as it does what you expect.

PySpark has four methods to create temporary views and they look quite similar at first glance.

- createGlobalTempView

- createOrReplaceGlobalTempView

- createOrReplaceTempView

- createTempView

We can see that there is two by two matrix of possibilities:

1. Do I want to replace an existing view (OrReplace)?

2. Do I want to create a Global view (Global)?

The first one is relatively easy to answer: if you use createTempView with a name already being used for another table, the method will fail, where it’ll replace the reference if you use createOrReplaceTempView(). In SQL, it is equivalent to use CREATE VIEW vs. CREATE OR REPLACE VIEW. I personally always use the latter as it mimics Python’s way of doing things: when re-assigning a variable, you comply.

What about Global? The difference between a local view and a global view has to do with how long it will last in memory. A local table is tied to your SparkSession where a global table is tied to the Spark application. The differences at this time are not significant at all as we are not using multiple SparkSession that need to share data. Because of this, I usually don’t use the Global methods.

7.2.2  Using the Spark catalog

The Spark catalog is an object that allows working with Spark SQL tables and views. A lot of its methods has to do with managing the metadata of those tables, such as their name and the level of caching (which I’ll cover in details in Chapter 9). We will look at the most basic set of functionality here, leaving the more advanced parts, such as caching and user-defined functions, to their respective chapters.

We can use the catalog to list the tables/views we have registered and drop them if we are done. The code in listing 7.4 provides the simple methods to do those tasks. Since they are mostly mimicing PySpark’s data frame functionality, I think that an example shows it best.

Listing 7.3. Using the catalog to display our registered view and then drop it

spark.catalog  ❶

#  <pyspark.sql.catalog.Catalog at 0x117ef0c18>

spark.catalog.listTables()  ❷

#  [Table(name='elements', database=None, description=None,

#         tableType='TEMPORARY', isTemporary=True)]

spark.catalog.dropTempView("elements")  ❸

spark.catalog.listTables()  ❹

# []

❶

The catalog is reached through the  catalog property of our SparkSession

❷

The  listTables method gives us a list of Table objects containing the information we want

❸

To delete a view, we use the method  dropTempView(), passing the name of the view as a parameter

❹

Our catalog now has no table for us to query

How that we understand how we can manage a Spark SQL view within PySpark, we can start looking at manipulating data using both languages.

7.3  SQL AND PYSPARK

The integration between Python (through PySpark) and SQL is very well thought out and can improve the speed at which we can write code. This section will cover the fundamental case, which is using a single language. I will review the most common operations from a pure-SQL and a purist PySpark way, to illustrate how basic manipulations are written.

For the remainder of the chapter, we will be using a public data set provided by BackBlaze which provided hard drive data and statistics. Their data is in the gigabytes range, which although not "big" yet is certainly Spark worthy as it’ll be more than the memory available on your home computer. For those willing to scale their program beyond a single computer, I recommend peeking at Appendix B to provision a cloud cluster and use the whole data provided on the website. A convenience shell script is also provided for downloading everything in one fell swoop. For those working locally and afraid of blowing your memory, you can use only Q3 2019. The syntax will differ marginally between both workflows. A 16GB laptop should be able to use all files.

Backblaze provides documentation mostly in the form of SQL statements, which is perfect for what we’re learning.

To get the files, you can either download them from the website (www.backblaze.com/b2/hard-drive-test-data.html) or use the backblaze_download_data.py available in the code repository, which need the wget package to be installed. The data needs to be in the ./data/Ch07 directory.

Listing 7.4. Downloading the data from backblaze

$ pip install wget

$ python backblaze_download_data.py full

# [some data download progress bars]

$ ls  ../../data/Ch07   ❶

# Periodic_Table_Of_Elements.csv data_Q2_2019.zip               data_Q4_2019.zip

# data_Q1_2019.zip               data_Q3_2019.zip

$ unzip '../../data/Ch07/*.zip'

❶

Windows users, use  dir ..\..\data\Ch07

Make sure you unzip the files into the directory before trying to read them. Unlike many other codecs (Gzip, Bzip2, Snappy and LZO, for instance), PySpark will not decompress zip files automatically when reading them, so we need to do it ahead of time. The unzip command can be used if you are using the command line (you might need to install the tool on Linux). On Windows, I usually use the Windows Explorer and unzip by hand.

The code to ingest and prep the data is pretty straightforward. We read each data source separately, and then we make sure that each data frame has the same columns as its peers. In our case, the data for the fourth quarter has two more columns than the others, so we add the missing columns. When unioning the four data frames together, we use a select method on the data frames so their column order is all the same. We continue by casting all the columns containing a SMART measurement as a Long, since they are documented as integral values. Finally, we register our data frame as a view so we can use SQL statements on it.

Listing 7.5. Reading the backblaze data into a data frame and registering a view

DATA_DIRECTORY = "../../data/Ch07/"

q1 = spark.read.csv(

DATA_DIRECTORY + "drive_stats_2019_Q1", header=True, inferSchema=True

)

q2 = spark.read.csv(

DATA_DIRECTORY + "data_Q2_2019", header=True, inferSchema=True

)

q3 = spark.read.csv(

DATA_DIRECTORY + "data_Q3_2019", header=True, inferSchema=True

)

q4 = spark.read.csv(

DATA_DIRECTORY + "data_Q4_2019", header=True, inferSchema=True

)

# Q4 has two more fields than the rest

q4_fields_extra = set(q4.columns) - set(q1.columns)

for i in q4_fields_extra:

q1 = q1.withColumn(i, F.lit(None).cast(T.StringType()))

q2 = q2.withColumn(i, F.lit(None).cast(T.StringType()))

q3 = q3.withColumn(i, F.lit(None).cast(T.StringType()))

# if you are only using the minimal set of data, use this version

backblaze_2019 = q3

# if you are using the full set of data, use this version

backblaze_2019 = (

q1.select(q4.columns)

.union(q2.select(q4.columns))

.union(q3.select(q4.columns))

.union(q4)

)

# Setting the layout for each column according to the schema

q = backblaze_2019.select(

[

F.col(x).cast(T.LongType()) if x.startswith("smart") else F.col(x)

for x in backblaze_2019.columns

]

)

backblaze_2019.createOrReplaceTempView("backblaze_stats_2019")

7.4  USING SQL-LIKE SYNTAX WITHIN DATA FRAME METHODS

Our goal in this section is to perform a quick exploratory data analysis on a subset of the columns presented. We will reproduce the failure rates that Backblaze computes themselves and identify the models with the most and least amount of failures in 2019.

7.4.1  Select and where

Those operations are like the bread and butter of data manipulation, yet we already see a difference in how SQL orders the operations differently than PySpark. The code in listing 7.7 explores a few models that have failed. This is useful as a quick-and-dirty way to see if the data looks consistent and see the serial numbers nomenclature.

Listing 7.6. Comparing select and where in PySpark and SQL

spark.sql(

"select serial_number from backblaze_stats_2019 where failure = 1"

).show(

5

)  ❶

backblaze_2019.where("failure = 1").select(F.col("serial_number")).show(5)

# +-------------+

# |serial_number|

# +-------------+

# |    57GGPD9NT|

# |     ZJV02GJM|

# |     ZJV03Y00|

# |     ZDEB33GK|

# |     Z302T6CW|

# +-------------+

# only showing top 5 rows

❶

Since a SQL statement returns a data frame, we still have to  show() it to see the results.

PySpark makes you think about how you want to chain the operations. In our case, we start by filtering the data frame and then we select the column of interest. SQL presents an alternative construction:

1. You put the columns you want to select at the beginning of your statement. This is called the SQL operation

2. You them add one or more tables to query, called the target

3. After, you add the conditions, such as filtering.

Every operation we will look in this chapter will be classified as an operation, a target, or a condition, so you can know where it fits in the statement.

Tip

If you have a table you want to extract as a data frame, you can just assign the result of a  SELECT statement to variable.

7.4.2  Group and order by

Here, we are looking at the capacity in gigabytes of the hard drives included in the data, by model. For this, we use a little bit of arithmetic and the pow() function, that elevates its first argument to the power of the second. We can see similarities about the SQL and PySpark vocabulary, but once again the order of the transformations is different.

Listing 7.7. Grouping and ordering in PySpark and SQL

spark.sql(

"""SELECT

model,

min(capacity_bytes / pow(1024, 3)) min_GB,

max(capacity_bytes/ pow(1024, 3)) max_GB

FROM backblaze_stats_2019

GROUP BY 1

ORDER BY 3 DESC"""

).show(5)

backblaze_2019.groupby(F.col("model")).agg(

F.min(F.col("capacity_bytes") / F.pow(F.lit(1024), 3)).alias("min_GB"),

F.max(F.col("capacity_bytes") / F.pow(F.lit(1024), 3)).alias("max_GB"),

).orderBy(F.col("max_GB"), ascending=False).show(5)

# +--------------------+--------------------+-------+

# |               model|              min_GB| max_GB|

# +--------------------+--------------------+-------+

# |       ST16000NM001G|             14902.0|14902.0|

# | TOSHIBA MG07ACA14TA|-9.31322574615478...|13039.0|

# |HGST HUH721212ALE600|             11176.0|11176.0|

# |       ST12000NM0007|-9.31322574615478...|11176.0|

# |       ST12000NM0008|             11176.0|11176.0|

# +--------------------+--------------------+-------+

# only showing top 5 rows

In PySpark, once again, we look at the logical order of operations. We groupby the capacity_GB columns. which is a computed column. Just like in PySpark, arithmetic operations can be performed using the usual syntax in SQL. Furthermore, the pow() function — available in pyspark.sql.functions — is also implemented in Spark SQL. If you need to see which functions can be used out of the box, the Spark SQL API doc contains the necessary information (spark.apache.org/docs/latest/api/sql/index.html).

Grouping and ordering are conditions in SQL, so they are at the end of the statement. One thing worth noting is that we group by 1 and order by 3 DESC. This is a short-hand way of referring to the columns in the SQL operation by position rather than name. In this case, it saves us from writing group by capacity_bytes / pow(1024, 3) or order by max(capacity_bytes / pow(1024,3)) DESC in the conditions block.

Looking at the results from our query, there are some drives that report more than one capacity. Furthermore, we have some drives reporting negative capacity, which is really odd. Let’s focus on seeing how prevalent this is.

7.4.3  Having

Let’s assume that, for each model, the maximum reported capacity is the correct one. Because of how SQL is evaluated, we can’t refer to an aliased field in our WHERE clause. Because of this, we have to rely to another keyword in order to compare our min and max reported capacity. The code in listing 7.9 shows how we can accomplish this in both languages.

Listing 7.8. Using having in SQL, and relying on where in PySpark

spark.sql(

"""SELECT

model,

min(capacity_bytes / pow(1024, 3)) min_GB,

max(capacity_bytes/ pow(1024, 3)) max_GB

FROM backblaze_stats_2019

GROUP BY 1

HAVING min_GB != max_GB

ORDER BY 3 DESC"""

).show(5)

backblaze_2019.groupby(F.col("model")).agg(

F.min(F.col("capacity_bytes") / F.pow(F.lit(1024), 3)).alias("min_GB"),

F.max(F.col("capacity_bytes") / F.pow(F.lit(1024), 3)).alias("max_GB"),

).where(F.col("min_GB") != F.col("max_GB")).orderBy(

F.col("max_GB"), ascending=False

).show(

5

)

# +--------------------+--------------------+-------+

# |               model|              min_GB| max_GB|

# +--------------------+--------------------+-------+

# | TOSHIBA MG07ACA14TA|-9.31322574615478...|13039.0|

# |       ST12000NM0007|-9.31322574615478...|11176.0|

# |HGST HUH721212ALN604|-9.31322574615478...|11176.0|

# |       ST10000NM0086|-9.31322574615478...| 9314.0|

# |HGST HUH721010ALE600|-9.31322574615478...| 9314.0|

# +--------------------+--------------------+-------+

# only showing top 5 rows

Having is a syntax unique to SQL: it can be thought of a where clause that can only be applied to aggregate fields, such as count(*) or min(date). Since it’s equivalent in functionality to where, having is in the condition block after the group by clause. In PySpark, we do not have having as a method. Since each method returns a new data frame, we do not have to have a different keyword, and can just use where with the column we created instead.

We will ignore (for now) those capacity reporting inconsistencies. They’ll come back as exercises.

7.4.4  Create tables/views

Now that we have queried the data and are getting the hang of it in SQL, we might want to checkpoint our work and save some data so we do not have to process everything from scratch the next time. For this, we can either create a table of a view, which we’ll then be able to query directly.

Creating a table or a view is very easy in SQL: we just have to prefix our query by CREATE TABLE/VIEW. Here, creating a table or a view will have a different impact. If you have a Hive metastore connected, creating a table will materialize the data (for more information, see appendix B) where a view will only keep the query. To take a baking analogy, CREATE TABLE will store a cake, where CREATE VIEW will refer to the ingredients (the original data) and the recipe (the query).

To demonstrate this, I will reproduce the drive_days and failures that compute the number of days of operation a model has and the number of drive failures it has had, respectively. The code in listing 7.10 shows how it is done: you prefix your select query with a CREATE [TABLE/VIEW].

In PySpark, we do not have to rely on extra syntax. A newly created data frame just has to be assigned to a variable and we are good to go.

Listing 7.9. Creating a view in Spark SQL and in PySpark

spark.sql(

"""

CREATE VIEW drive_days AS

SELECT model, count(*) AS drive_days

FROM drive_stats

GROUP BY model"""

)

spark.sql(

"""CREATE VIEW failures AS

SELECT model, count(*) AS failures

FROM drive_stats

WHERE failure = 1

GROUP BY model"""

)

drive_days = backblaze_2019.groupby(F.col("model")).agg(

F.count(F.col("*")).alias("drive_days")

)

failures = (

backblaze_2019.where(F.col("failure") == 1)

.groupby(F.col("model"))

.agg(F.count(F.col("*")).alias("failures"))

)

Creating tables from data in SQL

You can also create a table from data on a hard drive or HDFS. For this, you can use a modified SQL query like so. Since we are reading a CSV file, we prefix our path by csv..

spark.sql("create table q1 as select * from csv.`../../Data/Ch07/drive_stats_2019_Q1`")

I much prefer relying on PySpark syntax for reading and setting the schema from my data source and then using SQL, but the option is there for the taking.

7.4.5  Union and join

So far, we’ve seen how to query a single table at a time. In practice, you’ll often get multiple tables related to one another. We already witnessed this problem by having one historical table per quarter, which needed to be stacked (or unioned) together, and with our drive_days and failures tables, which each paint a single dimension of the story until they are merged (or joined) together.

Joins and unions are the only clauses we’ll see that are modifying the target piece in our SQL statement. In SQL, a query is operating on a single target at a time. We already saw at the beginning of the chapter how to use PySpark to union tables together. In SQL, we follow the same blueprint: SELECT columns FROM table1 UNION ALL SELECT columns FROM table2

PySpark’s union() ≠ SQL UNION

In SQL,  UNION removes the duplicate records. PySpark’s union() doesn’t, which is why it’s equivalent to a SQL UNION ALL. If you want to drop the duplicates, which is an expensive operation when working in a distributed context, use the distinct() function after your union(). This is one of the rare case where PySpark’s vocabulary doesn’t follow SQL’s, but it’s for a good reason. Most of the time, you’ll want the UNION ALL behaviour.

It is always a good idea to make sure that your data frames have the same columns, with the same types, in the same order, before attempting a union. In the PySpark solution, we used the fact that we can extract the columns in a list to select the data frames in the same fashion. Spark SQL does not have a simple way to do the same, so one would have to type all the columns. It’s alright when you just have a few, but we’re talking hundred here.

One easy way to circumvent this is to use the fact that a Spark SQL statement is a string. We can take our list of columns, transform it into a SQL-esque string and be done with it. This is exactly what I did in listing 7.11. It’s not a pure Spark SQL solution, but it’s much friendlier than making you type all the columns one by one.

Warning

Do not allow for plain string insertion if you are processing user input! This is the best way to have a SQL injection, where a user can craft a string that will wreck havok on your data. For more information about SQL injections and why , have a look at  owasp.org/www-community/attacks/SQL_Injection.

Listing 7.10. Unioning tables together in Spark SQL and in PySpark

columns_backblaze = ", ".join(q4.columns)  ❶

q1.createOrReplaceTempView("Q1")  ❷

q1.createOrReplaceTempView("Q2")

q1.createOrReplaceTempView("Q3")

q1.createOrReplaceTempView("Q4")

spark.sql(

"""

CREATE VIEW backblaze_2019 AS

SELECT {col} FROM Q1 UNION ALL

SELECT {col} FROM Q2 UNION ALL

SELECT {col} FROM Q3 UNION ALL

SELECT {col} FROM Q4

""".format(

col=columns_backblaze

)

)

backblaze_2019 = (  ❸

q1.select(q4.columns)

.union(q2.select(q4.columns))

.union(q3.select(q4.columns))

.union(q4)

)

❶

We use the  join() method on a separator string to create a string containing all the elements in the list, separated by `, `

❷

We promote our quarterly data frames to Spark SQL views so we can use them in our query

❸

This is taken from  listing 7.6

Join are equally as simple in SQL. We add a [DIRECTION] JOIN table [ON] [LEFT COLUMN] [OP] [RIGHT COLUMN] in the target portion of our statement. The direction is the same parameter of our how in PySpark. The on clause is a series of comparison between columns. In the listing 7.12 example, we are joining the records where the value in the model column is equal (=) on both drive_days and failures tables.

Listing 7.11. Joining the drive_days and failures tables together, in Spark SQL and in PySpark

spark.sql(

"""select

drive_days.model,

drive_days,

failures

from drive_days

left join failures

on

drive_days.model = failures.model"""

).show(5)

drive_days.join(failures, on="model", how="left").show(5)

7.4.6  Subqueries and common table expressions

The last piece of SQL syntax we will look on its own is the subquery and the common table expression. A lot of SQL references do not talk about them until very late which is a shame because they are a) easy to understand and b) are very helpful in keeping your code clean. In a nutshell, they allow to create tables that are local to your query. In Python, this would be similar to using the with statement or using a function block to limit the scope of a query. I will show the function approach as it is much more commonfootnote::[The with statement is usually used with resources that need to be cleaned up at the end. It doesn’t really apply here, but I felt like the comparison was worth mentionning.].

For our example, we will take our drive_days and failures table definitions and bundle them into a single query that will measure the models with the highest rate of failure in 2019. The code in listing 7.13 shows how we can do this using a subquery. A subquery simply replace a table name with a stand-alone SQL query. In the example, we can see that the name of the table has been replaced by the SELECT query that formed the table. We can alias the "table" referred in the subquery by adding the name at the end of the statement, after the closing parenthesis.

Listing 7.12. Finding the drive models with the highest rate of failure using subqueries

spark.sql(

"""

SELECT

model,

failures / drive_days failure_rate

FROM (

SELECT

model,

count(*) AS drive_days

FROM drive_stats

GROUP BY model) drive_days

INNER JOIN (

SELECT

model,

count(*) AS failures

FROM drive_stats

WHERE failure = 1

GROUP BY model) failures

ON

drive_days.model = failures.model

ORDER BY 2 desc

"""

).show(5)

Subqueries are cool but can be hard to read and debug, since you are adding some complexity into the main query. This is where common table expressions, or CTE, are especially useful. A CTE is a table definition, just like in the subquery case. The difference here is that you put them at the top of your main statement (before your main SELECT) and prefix with the word WITH. In listing 7.14, I am taking the same statement as with the subquery case but using two CTE instead. They can also be considered makeshift CREATE statements that get dropped at the end of the query, just like the with keyword in Python.

Listing 7.13. Finding the drive models with the highest rate of failure using common table expressions

spark.sql(

"""

WITH drive_days as (

SELECT

model,

count(*) AS drive_days

FROM drive_stats

GROUP BY model),

failures as (

SELECT

model,

count(*) AS failures

FROM drive_stats

WHERE failure = 1

GROUP BY model)

SELECT

model,

failures / drive_days failure_rate

FROM drive_days

INNER JOIN failures

ON

drive_days.model = failures.model

ORDER BY 2 desc

"""

).show(5)

In Python, the best comparison I’ve found was to wrap our statements in a function. Any intermediate variable created in the scope of the function would not be kept once the function returns. My version of the query using PySpark is in listing 7.15.

Listing 7.14. Finding the drive models with the highest rate of failure using Python scope rules to approximate CTE.

def failure_rate(drive_stats):

drive_days = drive_stats.groupby(F.col("model")).agg(  ❶

F.count(F.col("*")).alias("drive_days")

)

failures = (

drive_stats.where(F.col("failure") == 1)

.groupby(F.col("model"))

.agg(F.count(F.col("*")).alias("failures"))

)

answer = (  ❷

drive_days.join(failures, on="model", how="inner")

.withColumn("failure_rate", F.col("failures") / F.col("drive_days"))

.orderBy(F.col("failure_rate").desc())

)

return answer

failure_rate(drive_stats).show(5)

print("drive_days" in dir())  ❸

❶

We are creating intermediate data frames within the body of the function to avoid having a monster query.

❷

Our  answer data frame uses both intermediate data frames

❸

Testing if we have a variable  drive_days in scope once the function returned confirms that our intermediates frames are neatly confined inside the function scope.

This section took a (very) simple and high level EDA and demonstrated how we can do it using PySpark or Spark SQL. PySpark gives the floor to SQL without too much ceremony. This can be very convenient if you happen to hang out with DBAs and SQL developpers, as you can collaborate using their preferred language, while knowing that Python is right around the corner. Everybody wins!

7.4.7  A quick summary of PySpark vs. SQL syntax

PySpark borrowed a lot of vocabulary from the SQL world. I think this was a very smart idea: there are generations of programmers that know SQL, and adopting the same keywords Python-style makes it easy to communicate. Where we see a lot of difference is in the order of the operations: PySpark will naturally encourage you to think about the order of which the operations should be performed. SQL follows a more rigit framework where you need to remember if your operation belong in the operation, the target or the condition clause.

I find PySpark’s way of treating data manipulation more intuitive, but will rely on my years of SQL experience as a data analyst when convenient. When writing SQL, I usually write my query out order, starting with the target and building as I go. Not everything needs to be top to bottom!

So far, I’ve made a lot of effort to keep both languages in a vacuum. We’ll break the barrier now and unleash the power of Python+SQL. This will simplify how we write certain transformations, make our code easier to write and a less busy.

7.5  SIMPLIFYING OUR CODE: BLENDING SQL AND PYTHON TOGETHER

PySpark is rather accommodating when taking method and function parameters: you can pass a column name instead of a column object (F.col()) when using groupby(), for instance. On top of this, there are a few methods we can use to cram a little SQL syntax into our PySpark code. You’ll see that there isn’t many methods where you can use this, but it’s so useful and well done that you’ll end up using it all the time.

This section will build on the code we’ve written so far. We’re going to write a function that, for a given capacity, will return the top 3 most reliable drives according to our failure rate. We’ll leverage the code written so far and simplify it.

7.5.1  Reading our data

We start by simplifying the code to read the data. The data ingestion part of the program is displayed in listing 7.16. Compared to our original data ingestion, I’ve made a few changes.

First, I’ve put all the directories in a list so I could read them using a list comprehension. This removes some repetitive code and will also work easily if I remove or add some files (if you were only using Q3 2019, you can remove the other entries in the list).

Second, since we do not need the SMART measurements, I’ve taken the intersection of the columns instead of trying to fill the missing columns with null values. In order to create a common intersection that would apply to any number of data sources, I’ve used reduce which applies the anonymous function on all the column sets, resulting in the common set of columns between all the data frames. For those unfamiliar with reduce and other higher-order functions, Appendix D contains a deeper explanation of how they works. I have also added an assertion on the common set of columns, as I want to make sure it contains the columns I need for the analysis. Assertions are a good way to short circuit an analysis if certains conditions are not met. In this case, if I am missing one of the colums, I’d rather have my program fail early with an AssertionError rather than have a huge stack trace later. Assertions are covered in detail in Chapter 13.

Finally, I have used a second reduce for unioning all the distinct data frames in a cohesive one. The same principle is used as when I was creating the common set of variables. This makes the code a lot cleaner and it will work without any modifications should I want to add more sources or remove some.

Listing 7.15. The data ingestion part of our program

from functools import reduce

import pyspark.sql.functions as F

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

DATA_DIRECTORY = "../../data/Ch07/"

DATA_FILES = [

"drive_stats_2019_Q1",

"data_Q2_2019",

"data_Q3_2019",

"data_Q4_2019",

]

data = [

spark.read.csv(DATA_DIRECTORY + file, header=True, inferSchema=True)

for file in DATA_FILES

]

common_columns = list(

reduce(lambda x, y: x.intersection(y), [set(df.columns) for df in data])

)

assert set(["model", "capacity_bytes", "date", "failure"]).issubset(

set(common_columns)

)

full_data = reduce(

lambda x, y: x.select(common_columns).union(y.select(common_columns)), data

)

7.5.2  Using SQL-style expressions in PySpark

Now that our data has been read and is in a steady state, we can process it so we have an easy time answering our question. I will use SQL-style expressions when appropriate to showcase when it makes sense to fuse both languages. At the end of this section, we’ll have code that will:

1. Select only the useful columns for our query

2. Get our drive capacity in gigabytes

3. Compute the drive_days and failures data frames

4. Join the two data frames into a summarized one and compute the failure rate

The code is available in listing 7.17.

Listing 7.16. Processing our data so it’s ready for the query function

full_data = full_data.selectExpr(

"model", "capacity_bytes / pow(1024, 3) capacity_GB", "date", "failure"

)

drive_days = full_data.groupby("model", "capacity_GB").agg(

F.count("*").alias("drive_days")

)

failures = (

full_data.where("failure = 1")

.groupby("model", "capacity_GB")

.agg(F.count("*").alias("failures"))

)

summarized_data = (

drive_days.join(failures, on=["model", "capacity_GB"], how="left")

.fillna(0.0, ["failures"])

.selectExpr("model", "capacity_GB", "failures / drive_days failure_rate")

.cache()

)

SQL-style expression, one that we already have been using. One of the new ones is selectExpr()` which is just like the select() method with the exception that it will process SQL-style operations. I am quite a fan of this since it removes a bit of syntax when manipulating columns with functions and arithmetic. In our case, the PySpark alternative (displayed in listing 7.18) is a little more verbose and cumbersome to write and read, especially since we have to create a literal 1024 column to apply the pow() function.

Listing 7.17. Replacing selectExpr() by a regular select() in our final program

full_data = full_data.selectExpr(

F.col("model"),

(F.col("capacity_bytes") / F.pow(F.lit(1024), 3)).alias("capacity_GB"),

F.col("date"),

F.col("failure")

)

The second one is simply called expr(). It wraps a SQL-style expression into a column. This is kind of a generalized selectExpr(), where you can use it in lieu of of F.col() (or the column name) when you want to modify a column. If we take our failures table from listing 7.17, we can use an expr (or expression) as the agg() argument. This alternative syntax is shown in listing 7.19. I like doing it in agg() parameters, because it saves a lot of alias().

Listing 7.18. Using a SQL expr`ession in our `failures data frame code.

failures = (

full_data.where("failure = 1")

.groupby("model", "capacity_GB")

.agg(F.expr("count(*) failures"))

)

The third one, and my favourite, is the where()/filter() method. Filtering conditions in SQL is something I am very familiar with and being able to use them directly in PySpark with no ceremony is a godsend. In our final program, I was able to use full_data.where("failure = 1") instead of having to wrap the column name in F.col() like we’ve been doing since the beginning of this book.

I re-use this convenience in the query function, which is displayed in listing 7.20. This time, I’ve used string interpolation in conjunction with between. This doesn’t save many keystokes, but it’s very easy to understand and you don’t get as much line noise as when using the data.capacity_GB.between(capacity_min, capacity_max) or (if you prefer using the column function) F.col("capacity_GB").between(capacity_min, capacity_max). At this point, it’s very much a question of personal style and how familiar you are with each approach. I recommend you to try the SQL one if you don’t have a favourite yet.

Listing 7.19. The most_reliable_drive_for_capacity() function, that computes the the top N drives for a given capacity

def most_reliable_drive_for_capacity(data, capacity_GB=2048, precision=0.25, top_n=3):

"""Returns the top 3 drives for a given approximate capacity.

Given a capacity in GB and a precision as a decimal number, we keep the N

drives where:

- the capacity is between (capacity * 1/(1+precision)), capacity * (1+precision)

- the failure rate is the lowest

"""

capacity_min = capacity_GB / (1 + precision)

capacity_max = capacity_GB * (1 + precision)

answer = (

data.where(f"capacity_GB between {capacity_min} and {capacity_max}")  ❶

.orderBy("failure_rate", "capacity_GB", ascending=[True, False])

.limit(top_n)  ❷

)

return answer

❶

I used an SQL-style expression in my  where() method, without having to use any other special syntax or method

❷

Since we want to return the top N results, not just show them, I use  limit() instead of show().

7.6  CONCLUSION

You do not need to learn or use SQL to effectively work with PySpark. That being said, since the data manipulation API shares so much vocabulary and functionality with SQL makes it a plus to at least have a basic understanding of the syntax and query structure.

My family speaks both English and French, and sometimes you don’t always know where one language starts and one ends. I tend to think in both languages, and sometimes blends them in a single sentence. Likewise, I find that some problems are easier to solve with Python and some are more in SQL’s territory. You will find your own balance as well, which is why it’s nice to have the option. Just like spoken languages, the goal is to express your thoughts and intentions as clearly as possible, keeping your audience in mind.

7.7  SUMMARY

- Spark provides an SQL API for data manipulation. This API supports ANSI SQL.

- Spark (and PySpark’s by extension) borrows a lot of vocabulary and expected functionality from the way SQL manipulates tables. This is especially evident since the data manipulation module is called pyspark.sql.

- PySpark’s data frames need to be registered as views or tables before they can be queried with Spark SQL. You can give them a different name than the data frame you’re registering.

- PySpark’s own data frame manipulation methods and functions borrow SQL functionality for the most part. Some exceptions, such as union(), are present and documented in the API.

- Spark SQL queries can be inserted in a PySpark program through the spark.sql function, where spark is the running SparkSession.

- Spark SQL tables references are kept in a Catalog which contains the metadata for all tables accessible to Spark SQL.

- PySpark will accept SQL-style clauses in where() , expr() and selectExpr(), which can simplify the syntax for complex filtering and selection.

- When using Spark SQL queries with user-provided input, be careful about sanitizing the inputs to avoid potential SQL injection attacks.

7.8  EXERCISES

7.8.1  Exercise 7.1

If we look at the code in listing 7.17, we can simplify it even further, avoiding the creation of the two tables outright. Can you write a summarized_data without having to use another table than full_data and no join? (Bonus: try using pure PySpark, then pure Spark SQL, then a combo of both.)

7.8.2  Exercise 7.2

Our analysis is a flawed in that the age of a drive is not taken into consideration. Instead of ordering the drives by failure rate, order them by average age at failure (assume that every drive fails on 2020-01-01 if they are still alive at the end of the year).

7.8.3  Exercise 7.3

What is the total capacity (in TB) that BackBlaze records at the end of each month?

7.8.4  Exercise 7.4

Note: We will revisit this exercise in Chapter 10 when we look at window functions. In the meantime, this can be solved with a judicious usage of group bys and joins.

If you look at the data, you’ll see that some drive models can report an erroneous capacity. In the data preparation stage, use the most common capacity for each drive so we do not have more than one entry in our function.

8

EXTENDING PYSPARK WITH PYTHON: RDD AND USER-DEFINED-FUNCTIONS

This chapter covers:

- How to use the RDD as a low level, flexible data container.

- How to promote regular Python functions to UDF to run in a distributed fashion.

- How to use scalar UDF as an alternative to Python UDF, using pandas' API.

- How to use grouped map and grouped aggregate UDF on GroupedData object to split data frame computation on manageable chunks.

- How to apply UDF on local data to ease debugging.

Our journey with PySpark so far has proven that it is a powerful and versatile data processing tool. So far, we’ve explored many out-of-the-box functions and methods to manipulate data in a data frame. PySpark’s data frame manipulation functionality takes our Python code and applies an optimized query plan, introduced in Chapter 1. This makes our data jobs efficient, consistent, and predictable, just like coloring within the lines. What if we need to go off-script and manipulate our data according to our own rules?

In this chapter, I cover how we can build Python functions and scale them in PySpark. I start by introducing the resilient distributed dataset (or RDD), a more primitive and lower-level structure compared to the data frame. I explain how you manipulate data in an RDD and how its element (or row) major nature complements the data frame column-major approach.

I build on the knowledge of the RDD and explain how we can transfer this to the data frame through user-defined functions (or UDF). Following this, I move to pandas UDF, a speedy and powerful way to distribute Python functions on data frames. Finally, I close the loop by discussing how to use Scala modules in PySpark, so your programs can build on the shoulders of others. With those additional tools at your disposal, no data manipulation task will leave you stumped!

This chapter uses pandas from 8.3 onwards, for pandas UDF. Extensive pandas knowledge is a nice-to-have but is in no way expected. This chapter will cover the necessary pandas skills to use in within a basic pandas UDF. If your wish to level up you pandas skills to become a pandas UDF ninja, I warmly recommend the Pandas in Action book, by Boris Parkhaver (Manning, 2021).

For the examples in the chapter, we will need three previously unused libraries: pandas, scikit-learn, and PyArrow. If you have installed Anaconda (see appendix B), you can use conda to install the libraries; otherwise, you can use pip  [7].

# Conda installation

conda install pandas sklearn pyarrow

# Pip installation

pip install pandas sklearn pyarrow

8.1  PYSPARK, FREESTYLE: THE RESILIENT DISTRIBUTED DATASET

This section covers the resilient distributed dataset (RDD) and how you use it to manipulate data. My goal is to provide a good overview of what an RDD is and how you manipulate data using Python functions. Besides being useful in itself, this section’s material is a perfect introduction to user-defined functions, the data frame’s answer to the RDD operating model. I cover user-defined functions in the following sections.

The RDD can be thought of as the ultimate flexible data container structure. You can cram pretty much anything that can be pickled (python’s way of serializing objects) into it. Where the data frame has good structure documentation though Column objects, types, and schemas, the RDD does not force you into a specific layout. Each element is independent of the other. In listing 8.1, I create a list containing multiple objects of different types, then promote it to an RDD via the parallelize method. The resulting RDD is depicted in figure 8.1.

Listing 8.1. Promoting a Python list to a resilient distributed dataset

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

collection = [1, "two", 3.0, ("four", 4), {"five": 5}]  ❶

sc = spark.sparkContext  ❷

collection_rdd = sc.parallelize(collection)  ❸

print(collection_rdd)

# ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:195  ❹

❶

My  collection is a list of an integer, a string, a float, a tuple, and a dict.

❷

The RDD functions and methods are under the  SparkContext object, accessible as an attribute of our SparkSession. I alias it to sc for convenience.

❸

The list gets promoted to an RDD using the  parallelize method of the SparkContext

❹

Our  collection_rdd object is effectively an RDD. PySpark returns the type of the collection when we print the object.

Figure 8.1. The colletion_rdd RDD. Each object is independent from each other in the container. No column, no structure, no schema.

Compared to a data frame, the RDD is much more free style (pardon the 90’s reference) in terms of what it accepts. Manipulating data follows the same pattern: instead of using transformations on column objects, we go straight to the data, element by element. The next section introduces the RDD data manipulation API.

8.1.1  Manipulating data the RDD way: map, filter and reduce

This section explains the building blocks of data manipulation using an RDD. I discuss the concept of higher-order function and we use them to transform data. I finish with a quick overview of MapReduce, a fundamental concept in large-scale data processing, and place in the context of Spark and the RDD.

Manipulating data with an RDD feels to me like giving orders to an army as a general: you have full obedience from your privates/divisions into the field/RDD, but if you give an incomplete or wrong order, you’ll cause havoc within your troop. Furthermore, each division has its own, specific type of order they can perform, and you don’t have a reminder of what’s what (unlike with a data frame schema). Sounds like a fun job…

An RDD provides many methods (which you can find in the API documentation for the pyspark.RDD object), but we put our focus on three specific methods: map(), filter() and reduce(). Together, they capture the ethos of data manipulation with an RDD; knowing how those three work will give you the necessary foundation to understand the others.

map(), filter() and reduce() all take a function (that we will call f) as their only parameter. We call functions that take other functions as parameters higher-order functions. They can be a little difficult to understand if it’s the first time you are encountering them; fear not, after seeing them in action, you’ll be very comfortable using them in PySpark (and in Python, if you have a look at appendix D).

MAP

map() is the easiest one to figure out: it applies the function taken as a parameter to every element of the RDD. I try, and fail, to map a simple function that adds 1 to a value in listing 8.2. What’s happening?

Listing 8.2. Mapping a simple function add_one() to each element to an RDD.

def add_one(value):

return value + 1  ❶

collection_rdd = collection_rdd.map(add_one)  ❷

print(collection_rdd.collect())  ❸

# Stack trace galore! The important bit, you'll get one of the following:

# TypeError: can only concatenate str (not "int") to str

# TypeError: unsupported operand type(s) for +: 'dict' and 'int'

# TypeError: can only concatenate tuple (not "int") to tuple

❶

A seeminly inoffensive function;  add_one() adds one to the value passed as an argument.

❷

I apply my function to every element in the RDD, via the  map() method.

❸

collect() materializes an RDD into a Python list on the master node.

To understand why our code is failing, we’ll break down the mapping process, illustrated in figure 8.2. I apply the add_one() function to each element in the RDD by passing it as an argument to the map() method. add_one() is a regular Python function, applied to regular Python objects. Since we have incompatible types (taking an example, "two" + 1 is not a legal operation in Python), three of our elements are TypeError. When I collect() the RDD to peek at the values, it explodes into a stack trace right in my REPL.

Note

The RDD is a lazy collection. If you have an error in your function application, it will not be visible until you perform an action (such as  collect()), just like with the data frame.

Figure 8.2. Applying the add_one() function to each element of the RDD via map. If the function cannot be applied, an error will be raised during action time.

Fortunately, since we are working with Python, we can use a try/except block to prevent errors. I provide an improved safer_add_one() function in listing 8.3, which returns the original element if the function runs into a type error.

Listing 8.3. Mapping the function safer_add_one() to each element to an RDD. Each element that can’t be incremented will remain the same in the resulting RDD.

collection_rdd = sc.parallelize(collection)  ❶

def safer_add_one(value):

try:

return value + 1

except TypeError:

return value  ❷

collection_rdd = collection_rdd.map(safer_add_one)

print(collection_rdd.collect())

# [2, 'two', 4.0, ('four', 4), {'five': 5}] ❸

❶

I recreate my RDD from scratch to remove the erreneous operation in the thunk (see chapter 1 for a description of a computation thunk).

❷

Our function returns the original value untouched if it encounters a  TypeError

❸

The relevant elements of the RDD have been incremented by one.

In summary, you use map() to apply a function to every element of the RDD. Because of the flexibility of the RDD, PySpark does not give you any safeguards about the content of the RDD. You are responsible, as the developer, to make your function robust regardless of the input.

FILTER

filter() is used to keep only the element that satisfies a predicate. In the data frame world, we encountered the where()/filter() methods that do just that. In the RDD world, filter() is much more flexible: it takes a function f and keeps only the elements that return a truthful value. In listing 8.4, I filter my RDD to keep only the integer and float elements, using a lambda function. The isinstance() function returns True if the first argument’s type is present in the second argument; in our case, it’ll test if each element is either a float or an int.

Listing 8.4. Filtering our RDD with a lambda function. Our resulting RDD only has int and float values.

collection_rdd = collection_rdd.filter(lambda elem: isinstance(elem, (float, int)))

print(collection_rdd.collect())

# [2, 4.0]

Just like map(), the function passed as a parameter to filter() is applied to every element in the RDD. This time, though, instead of returning the result in a new data frame, we keep the original value if the result of the function is truthy. If the result is falsey, we drop the element. I show the breakdown of the filter() operation in figure 8.3.

Truthy/Falsey in Python

Python has its own rules for boolean testing: because of this, I avoid using absolute True/False when talking about filtering in Python and PySpark. As a rule of thumb, False, 0 (the number zero in any Python numerical type), and empty sequences and collections (list, tuple, dict, set, range) are falsey. For more precisions on how Python imputes boolean values on non-booleans types, refer straight to the Python documentation: docs.python.org/3/library/stdtypes.html#truth-value-testing.

Figure 8.3. Filtering our RDD to keep only int and float. Our predicate function is applied element-wise, and only the values leading to truthy predicates are kept.

REDUCE

The last operation is reduce() and as its name implies, it is being used to reduce elements in an RDD. By reducing, I mean taking 2 elements and applying a function that will return only one element. PySpark will apply the function to the first two elements, then apply it again to the result and the third element, and so on until there are no elements left. I find the concept easier when explained visually, so figure 8.4 shows the process of summing the value of the elements in the RDD using reduce. In code form, listing 8.5 presents how to use the reduce() method on a data frame.

Figure 8.4. Reducing our RDD by summing the values of the elements.

Listing 8.5. Applying the add() function via reduce() to get the sum of the values of the elements of our RDD.

from operator import add  ❶

collection_rdd = sc.parallelize([4, 7, 9, 1, 3])

print(collection_rdd.reduce(add))  # 24

❶

The  operator module contains the function version or common operators such as + (add()), so we do not have to pass a lambda a, b: a + b.

map(), filter(), and reduce() appear at a first glance like simple concept: they take a function and apply it to all the elements inside the collection. The result is treated differently depending on the method chosen, and reduce() requires a function of two arguments returning a single value. Well, in 2004, Google used this humble concept and caused a revolution in the large-scale data processing world by publishing its MapReduce framework (research.google/pubs/pub62/). You can’t argue on inspiration: the name is a combination of map() and reduce()! This framework was a direct inspiration to big data frameworks such as Hadoop and Spark. Although modern abstractions such as the data frame aren’t as close to the original MapReduce, the ideas remain, and by understanding at a high level the building blocks, it’ll be easier to understand some higher-level design choices.

reduce() in a distributed world

Because of PySpark’s distributed nature, the data of an RDD can be distributed across multiple partitions. The reduce() function will be applied independently on each partition, and then each intermediate value will be sent to the master node for the final reduction. Because of this, you need to provide a commutative and associative function to reduce().

A commutative function is a function in which you do not care about the order in which the arguments are applied. For example, add() is commutative, since a + b = b + a. Oh the flip side, subtract() is not: a - b != b - a.

An associative function is a function in which you do not care about how the values are grouped. add() is associative, since (a + b) + c = a + (b + c). subtract() is not: (a - b) - c != a - (b = c).

add(), multiply(), min() and max() are both associative and commutative.

This concludes our whirlwind tour of the PySpark RDD API. We covered how the RDD applies transformations to its elements through higher-order functions such as map(), filter(), and reduce(). Those higher-order functions apply the functions passed as parameters to each element, making the RDD "element-major" (or "row-major"). If you are curious about the other applications of the RDD, I recommend looking at Appendix F as a companion to the PySpark online API documentation. Most of the methods on the RDD have a direct equivalent to the data frame or grow directly from the usage of map(), filter(), and reduce(). We revisit the RDD briefly in chapter 9 when talking about performance. The next sections build on the concept of applying Python functions directly, but this time on a data frame. The fun only begins!

Exercise 8.1

The PySpark RDD API provides a count() method that returns the number of elements in the RDD as an integer. Reproduce the behavior of this method using map(), filter() and/or reduce().

Exercise 8.2

What is the return value of the following code block?

a_rdd = sc.parallelize([0, 1, None, [], 0.0])

a_rdd.filter(lambda x: x).collect()

1. [1]

2. [0, 1]

3. [0, 1, 0.0]

4. []

5. [1, []]

Optional topic: going full circle, a data frame is an RDD!

To show the ultimate flexibility of the RDD, have a look at this: you can access an implicit RDD within a data frame via the rdd attribute of a data frame.

Listing 8.6. Uncovering the RDD from within a data frame using the rdd attribute

df = spark.createDataFrame([[1], [2], [3]], schema=["column"])

print(df.rdd)

# MapPartitionsRDD[22] at javaToPython at NativeMethodAccessorImpl.java:0

print(df.rdd.collect())

# [Row(column=1), Row(column=2), Row(column=3)]

From a PySpark perspective, a data frame is also an RDD[Row] (from pyspark.sql.Row), where each row can be thought of a dictionary: the key is the column name and the value is the value contained in the record. To do the opposite trip, you can pass the RDD to spark.createDataFrame with an optional schema. Remember that, when moving from a data frame to an RDD, you give up the schema safety of the data frame!

It can be tempting to move back and forth between a data frame and an RDD depending on the operation you wish to perform. Bear in mind that this will come at a performance cost (which we’ll explore further in chapter 9), but also makes your code harder to follow. You will also have to make sure all your Row follow the same schema before promoting your RDD into a data frame. The next sections cover how you can harness most of the power of the RDD without leaving the comfort of the data frame.

[7] On windows, you might sometimes have issues with the pip wheels. If this is your case, refer to the PyArrow documentation page for installing: arrow.apache.org/docs/python/install.html

8.2  USING PYTHON TO EXTEND PYSPARK VIA USER-DEFINED FUNCTIONS

In the previous section, we got a taste of flexibility with the RDD approach to data manipulation. This section takes the same question — how can we run Python code on our data? — and applies it to the data frame. More specifically, we focus on the map() transformation: for each record that comes in, one record comes out. Map-type transformations are by far the most frequent and the easiest to implement.

Unlike the RDD, the data frame has structure enforced by columns. To address this constraint, PySpark provides the possibility to create user-defined functions via the pyspark.sql.functions.udf() function. What comes in is a regular Python function, and out is a function promoted to work on PySpark columns.

To illustrate this, we will mock up a data type not present in PySpark: the Fraction. Fractions are made of a numerator and a denominator. In PySpark, we’ll represent this as an array of two integers. In listing 8.7, I create a data frame containing two columns, standing for the numerator and the denominator. I fuse the two columns in an array column via the array() function.

Listing 8.7. Creating a data frame containing a single array column, where the first element is the numerator and the second the denominator.

import pyspark.sql.functions as F

import pyspark.sql.types as T

fractions = [[x, y] for x in range(100) for y in range(1, 100)]  ❶

frac_df = spark.createDataFrame(fractions, ["numerator", "denominator"])

frac_df = frac_df.select(

F.array(F.col("numerator"), F.col("denominator")).alias("fraction"),  ❷

)

frac_df.show(5, False)

# +--------+

# |fraction|

# +--------+

# |[0, 1]  |

# |[0, 2]  |

# |[0, 3]  |

# |[0, 4]  |

# |[0, 5]  |

# +--------+

# only showing top 5 rows

❶

I start the range for the denominator at 1, since a fraction with 0 for the denominator is undefined

❷

The  array() function takes two or more columns of the same type and creates a single column containing an array of the columns passed as parameter.

To support our new makeshift fraction type, we create a few functions that provide basic functionality. This is a perfect job for Python UDF, and I take the opportunity to introduce the two ways PySpark enables its creation.

8.2.1  It all starts with plain Python: using typed Python functions

This section covers creating a Python function that will work seamlessly with a PySpark data frame. While Python and Spark are like PB & J, creating and using UDF requires a few precautions. I introduce how you can use Python type hints to make sure your code will work seamlessly with PySpark types. At the end of this section, we will have a function to reduce a fraction and one to transform a fraction into a floating-point number.

My blueprint when creating a function destined to become a Python UDF is as follow:

1. Create and document the function

2. Make sure the input and output types are compatible

3. Test the function

Testing PySpark code (including UDF) is covered in chapter 14. For this section, I provide a couple of assertions to make sure the function is behaving like expected.

Behind every UDF is a Python function, so our two functions are in listing 8.8. I introduce Python type annotations in this code block: the rest of the section covers how they are used in this context and why they are a powerful tool when combined with Python UDF.

Listing 8.8. Creating our three python functions, complete with type annotation and assertions

from fractions import Fraction  ❶

from typing import Tuple, Optional  ❷

Frac = Tuple[int, int]  ❸

def py_reduce_fraction(frac: Frac) -> Optional[Frac]:  ❹

"""Reduce a fraction represented as a 2-tuple of integers."""

num, denom = frac

if denom:

answer = Fraction(num, denom)

return answer.numerator, answer.denominator

return None

assert py_reduce_fraction((3, 6)) == (1, 2)  ❺

assert py_reduce_fraction((1, 0)) is None

def py_fraction_to_float(frac: Frac) -> Optional[float]:

"""Transforms a fraction represented as a 2-tuple of integers into a float."""

num, denom = frac

if denom:

return num / denom

return None

assert py_fraction_to_float((2, 8)) == 0.25

assert py_fraction_to_float((10, 0)) is None

❶

We rely on the  Fraction data type from the fractions module to avoid reinventing the wheel

❷

Some specific types need to be imported to be used: the standard library contains the types for scalar values, but containers like  Option and Tuple need to be explicitly imported.

❸

We create a type synonym  Frac. This is equivalent to telling Python/mypy "When you see Frac , assume it’s a Tuple[int, int] " (a tuple containing two integers). This makes the type annotations easier to read.

❹

Our function takes a  Frac as argument and returns a Optional[Frac], which translates to "either a Frac or None ".

❺

I create a few assertions to sanity check my code and make sure I get the expected behavior.

Both functions are very similar, so I’ll take py_reduce_fraction and go through it line by line.

My function definition has a few new elements. The frac parameter has a : Frac and we have a → Optional[Frac] before the colon. Those additions are type annotations and are an amazing tool in making sure the function accepts and returns what we expect. Python is a dynamic language: this means that the type of an object is known at runtime. When working with PySpark’s data frame, where each column has one and only one type, we need to make sure that our UDF will return consistent types. We can use type hints to ensure this.

Python’s type checking is enabled by using a library called mypy. You install it via pip install mypy. Once installed, you can run mypy on your file with mypy MY_FILE.py. Appendix D contains a deeper introduction to the typing module and mypy and how it applies (and why it should apply) beyond UDF. I’ll add type annotation when relevant, as they can be useful documentation besides making our code more robust. (What does my function expect? What does it return?)

In my function definition, I announce that the frac function parameter is of type Frac, which is equivalent to a Tuple[int, int], or a 2-element tuple containing two integers. If I get to share my code with others, this type annotation sends a signal about the input type of my function. Furthermore, mypy will complain if I try to pass an incompatible argument to my function: if I try to do py_reduce_fraction("one half"), mypy will tell me the following.

error: Argument 1 to "py_reduce_fraction" has incompatible type "str"; expected "Tuple[int, int]"

I can already see the type errors vanishing…

The second type annotation, located after the function arguments and prefixed with an arrow, is the type annotation for the return type of the function. We recognize the Frac, but this time, I wrapped it into an Optional type.

In 8.1, when creating functions to be distributed over the RDD, I needed to make sure that they would not trigger an error, returning None instead. I apply the same concept here. I test for denom being a truthy value: if it is equal to 0, I return None. This is such a frequent use-case that Python provides the Optional[…] type, which means "either the type between the brackets or None ". PySpark will accept None values as null (see appendix D for the complete list of Python vs. Spark types).

Type annotations: stop cluttering my code!

Type annotations are incredibly useful out of the box, but they are especially nifty when used with Python UDF. Since PySpark’s execution model is lazy, you’ll often get your error stack trace at action time. UDF stack traces are not any harder to read than any other stack trace in PySpark — which is not saying much — but a vast majority of the bugs are because of a bad input or return value. Now, type annotations are not a silver bullet, but they are a great tool to avoid and diagnose type errors.

With all this said, Python’s type annotations are not available everywhere. When working with pandas, numpy and PySpark’s data structures, you might encounter "no stub file for [module]". Python’s typing story is still early, and you might run into some rough edges. Fortunately, mypy tries to minimize the annoyance by only checking the annotated functions, and you can search for "data science types" on PyPI for an interim solution.

The rest of the function is relatively straightforward: I ingest the numerator and denominator in a Fraction object, which reduces the fraction. I then extract the numerator and denominator from the Fraction and return them as a tuple of 2 integers, as I promised in my return type annotation.

We have our two functions, with well-defined input and output types. In the next section, I show how you promote regular Python functions to UDF and apply them to your data frame.

8.2.2  From Python function to UDF: two approaches

PySpark provides two equivalent ways to promote a function to a UDF. In this section, I explain how to use both approaches. I also discuss how to provide return Python type hints when creating a UDF.

PySpark provides a udf() function in the pyspark.sql.functions module to promote Python function to their UDF equivalent. The function takes two parameters.

1. The function you want to promote.

2. Optionally, the return type of the generated UDF. In table 8.1, I summarize the type equivalences between Python and PySpark. If you provide a return type, it must be compatible with the return value of your UDF.

Table 8.1. A summary of the types in PySpark. A star next to the "Python equivalent" column means the Python type is more precise or can contain larger values, so you need to be careful with the values you return.

Type Constructor

String representation

Python equivalent

NullType()

null

None

StringType()

string

Python’s regular strings

BinaryType()

N/A

bytearray

BooleanType()

boolean

bool

DateType()

date

datetime.date (from the datetime library)

TimestampType()

timestamp

datetime.datetime (from the datetime library)

DecimalType(p,s)

decimal

decimal.Decimal (from the decimal library)*

DoubleType()

double

float

FloatType()

float

float*

ByteType()

byte or tinyint

int*

IntegerType()

int

int*

LongType()

long or bigint

int*

ShortType()

short or smallint

int*

ArrayType(T)

N/A

list, tuple or Numpy array (from the  numpy library)

MapType(K, V)

N/A

dict

StructType([…])

N/A

list or tuple

In listing 8.9, I promote the py_reduce_fraction() function to a UDF via the udf() function. Just like I did with the Python equivalent, I provide a return type to the UDF (this time, an Array of Long, since Array is the companion type of the tuple and Long the one for Python integers). Once the UDF is created, we can apply it like any other PySpark function on columns. I chose to create a new column to showcase the before and after: in the sample shown, the fraction appears properly reduced.

Listing 8.9. Creating a UDF explicitly with the udf() function, and applying it to our data frame.

SparkFrac = T.ArrayType(T.LongType())  ❶

reduce_fraction = F.udf(py_reduce_fraction, SparkFrac)  ❷

frac_df = frac_df.withColumn(

"reduced_fraction", reduce_fraction(F.col("fraction"))  ❸

)

frac_df.show(5, False)

# +--------+----------------+

# |fraction|reduced_fraction|

# +--------+----------------+

# |[0, 1]  |[0, 1]          |

# |[0, 2]  |[0, 1]          |

# |[0, 3]  |[0, 1]          |

# |[0, 4]  |[0, 1]          |

# |[0, 5]  |[0, 1]          |

# +--------+----------------+

# only showing top 5 rows

❶

I alias the "array of long" PySpark type to the  SparkFrac variable.

❷

I promote my Python function using the  udf() function, passing my SparkFrac type alias as the return type.

❸

A UDF can be used like any other PySpark column function

You also have the option to create your Python function and promote it as a UDF using the udf function as a decorator. In listing 8.10, I define by py_fraction_to_float() (now called simply fraction_to_float()) directly as a UDF by preceding my function definition by @F.udf([return_type]). In both cases, you can access the underlying function from the UDF by calling the attribute frac.

Listing 8.10. Creating a UDF directly using the udf() decorator.

@F.udf(T.DoubleType())  ❶

def fraction_to_float(frac: Frac) -> Optional[float]:

"""Transforms a fraction represented as a 2-tuple of integers into a float."""

num, denom = frac

if denom:

return num / denom

return None

frac_df = frac_df.withColumn(

"fraction_float", fraction_to_float(F.col("reduced_fraction"))

)

frac_df.select("reduced_fraction", "fraction_float").distinct().show(5, False)

# +----------------+-------------------+

# |reduced_fraction|fraction_float     |

# +----------------+-------------------+

# |[3, 50]         |0.06               |

# |[3, 67]         |0.04477611940298507|

# |[7, 76]         |0.09210526315789473|

# |[9, 23]         |0.391304347826087  |

# |[9, 25]         |0.36               |

# +----------------+-------------------+

# only showing top 5 rows

assert fraction_to_float.func((1, 2)) == 0.5  ❷

❶

The decorator performs the same function as the  udf() function, but return a UDF bearing the name of the function defined under.

❷

In order to perform my assertion, I use the  func attribute of the UDF, which returns the function ready to be called.

Exercise 8.3

Create a UDF that adds two fractions together, and test it by adding the reduced_fraction to itself in the frac_df data frame.

Exercise 8.4

Our py_reduce_fraction will not work if the numerator or denominator exceeds pow(2, 63)-1 or is lower than -pow(2, 63). Modify the py_reduce_fraction to return None if this is the case.

Bonus: Does this change the type annotation provided? Why?

8.3  BIG DATA IS JUST A LOT OF SMALL DATA: USING PANDAS UDF

Note

Spark 3.0 brings a ton of improvements and new functionality to pandas UDF, including scalar iterator, map iterator, and cogrouped map pandas UDF. The chapter is written with Spark 2.4.5, the latest available version, in mind, but I plan on including a new section containing the new Spark 3.0 UDF (and refresh the current material if necessary) once I get them under the microscope.

Python UDF, while very flexible, only operates on a single record at a time, just like the map() method of the RDD. This section introduces a fresh way to approach UDF: pandas (or vectorized) UDF. Just like their name indicates, they rely on pandas, a very popular data manipulation library in Python.

Warning

Vectorized UDF were introduced in Spark 2.3 (scalar, grouped map) and improved upon in Spark 2.4 (grouped aggregate). I recommend using the most up-to-date stable version everywhere in the book, but this section requires it.

At the core, pandas UDF can be seen as nothing more than distributing pandas data manipulation code within a data frame. In chapter 1, I explained that Spark distributes a large amount of data in multiple partitions and orchestrates transformations and actions through a master-workers split. In our model, a pandas UDF is just like treating each chunk of data like an independent pandas data frame.

8.3.1  Setting our environment: connectors and libraries

For this section, I use the National Oceanic and Atmospheric Administration (NOAA) Global Surface Summary of the Day (GSOD) dataset. This data is available from multiple sources, but one of the easiest to access is Google pubic data set repository, made available through BigQuery. I use the BigQuery connector to Spark to ingest the data (github.com/GoogleCloudDataproc/spark-bigquery-connector). The instructions on their Github might change over time, so refer to their README as necessary. For Spark 2.4.5, you will need to download the spark_bigquery_latest.jar. If you are using Spark 3.0, you will need the jar for your Scala version (2.11 or 2.12, depending on your Spark installation).

To access the data, you also need a Google Cloud Platform (GCP) account. Once your account is created, you need to create a service account and a service account key to tell BigQuery to give you access to the public data programmatically. To do so, select "Service Account" (under "IAM & Admin") and click "+ Create Service Account". Give a funny name to your service account name. In the service account permissions menu, select "BigQuery → BigQuery admin" and click "continue". In the last step, click "+ CREATE KEY" and select JSON. Download the key and store it somewhere safe.

Warning

Treat this key like any other password. If a malicious person steals your key, go back to the "Service Accounts" menu and delete this key, recreating a new one.

The last step before analyzing our data using vectorized UDF is to install PyArrow. PyArrow is the python bindings to the Apache Arrow project (arrow.apache.org/), an in-memory data serialization library. It provides a bridge between the PySpark data frame and the pandas data frame. If you are using Spark 2.3 or 2.4, you also need to set a flag in the conf/spark-env.sh file of your Spark root directory. In the conf/ directory, you should find a spark-env.sh.template file. Make a copy, name it spark-env.sh and add this line in the file.

ARROW_PRE_0_15_IPC_FORMAT=1

This will tell PyArrow to use a serialization format compatible with Spark 2.X, instead of the newer one only compatible with Spark 3.0. The Spark JIRA ticket contains more information about this (issues.apache.org/jira/browse/SPARK-29367). You can also use PyArrow version 0.14 and avoid the problem altogether.

Tip

If you are using PySpark in the cloud, refer to your provider documentation. Each cloud provider has a different way of managing Spark dependencies and libraries. For a quick review of the most popular Spark cloud offerings, see Appendix C.

Finally, we can (re-)start our PySpark shell, with the new library installed. The pyspark and spark-submit commands take an optional --jars parameter that loads external dependencies on your Spark installation.

Listing 8.11. Starting a PySpark shell with the BigQuery connector installed

pyspark --jars spark-bigquery-latest.jar

Alternatively, if you use PySpark through your Python/IPython shell, you can load the library directly from Maven (Java/Scala’s equivalent of PyPI) when creating your SparkSession.

Listing 8.12. Initializing PySpark withing your python shell with the BigQuery connector enabled.

from pyspark.sql import SparkSession

spark = SparkSession.builder.config(

"spark.jars.packages",

"com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.15.1-beta",  ❶

).getOrCreate()

# Ivy Default Cache set to: /Users/jonathan_rioux/.ivy2/cache

# The jars for the packages stored in: /Users/jonathan_rioux/.ivy2/jars

# :: loading settings :: url = jar:file:/usr/local/Cellar/apache-spark/2.4.5/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

# com.google.cloud.spark#spark-bigquery-with-dependencies_2.11 added as a dependency

# :: resolving dependencies :: org.apache.spark#spark-submit-parent-035f1392-cda4-4935-a62b-969bda5449d5;1.0

#       confs: [default]

#       found com.google.cloud.spark#spark-bigquery-with-dependencies_2.11;0.15.1-beta in central

# :: resolution report :: resolve 134ms :: artifacts dl 2ms

#       :: modules in use:

#       com.google.cloud.spark#spark-bigquery-with-dependencies_2.11;0.15.1-beta from central in [default]

#       ---------------------------------------------------------------------

#       |                  |            modules            ||   artifacts   |

#       |       conf       | number| search|dwnlded|evicted|| number|dwnlded|

#       ---------------------------------------------------------------------

#       |      default     |   1   |   0   |   0   |   0   ||   1   |   0   |

#       ---------------------------------------------------------------------

# :: retrieving :: org.apache.spark#spark-submit-parent-035f1392-cda4-4935-a62b-969bda5449d5

#       confs: [default]

#       0 artifacts copied, 1 already retrieved (0kB/4ms)

# [...]

❶

I took the package version recommended for the most recent Spark/Scala version (2.4.5/2.11)

Warning

If you have a SparkSession already in progress, it is  not enough to just spark.stop() and try to restart. You need to stop the JVM process altogether. Trying to do this without restarting your PySpark/Python REPL is an exercise in frustration, so just kill and start fresh. If you use the method in listing 8.12 and you don’t see similar jar verbiage, it will not work.

8.3.2  Preparing our data

Before we can start working on our pandas UDF, we need to extract the data from BigQuery and assemble the multiple tables in a cohesive data frame. Reading data from BigQuery is straightforward. I use the bigquery specialized SparkReader — provided by the connector library we embedded to our PySpark shell — providing two options:

1. The table parameter, pointing to the table we want to ingest. The format is project.dataset.table: the bigquery-public-data is a project available to all.

2. The credentialsFile is the JSON key downloaded in 8.3.1. You need to adjust the path and file name accordingly to the location of the file.

Tip

If you are using Google DataProc, you do not have to provide a  credentialsFile since the permissions will be granted through your GCP account. The documentation for the BigQuery connector will provide the most up-to-date instructions. Appendix C covers Spark in the cloud, including Google DataProc.

The code is available in listing 8.13. For my gsod table, I need to union the tables together in a single cohesive data frame. While I can chain multiple union() like in chapter 7, I went a more elegant route using the reduce operator, this time by applying it to my list comprehension.

Listing 8.13. Reading the stations and gsod tables for 2010 to 2020.

from functools import reduce

from pyspark.sql import DataFrame

def read_df_from_bq(year):  ❶

return (

spark.read.format("bigquery")  ❷

.option("table", f"bigquery-public-data.noaa_gsod.gsod{year}")  ❸

.option("credentialsFile", "bq-key.json")  ❹

.load()

)

gsod = (

reduce(

DataFrame.union, [read_df_from_bq(year) for year in range(2010, 2020)]  ❺

)

.dropna(subset=["year", "mo", "da", "temp"])

.where(F.col("temp") != 9999.9)

)

❶

Since all the tables are read the same way, I abstract my reading routine in a re-usable function, returning the resulting data frame.

❷

I use the  bigquery specialized reader via the format() method.

❸

The stations table is available in BigQuery under  bigquery-public-data.noaa_gsod.gsodXXXX, where XXXX is the four-digit year.

❹

I pass my JSON service account key to the  credentialsFile option, to tell Google I am allowed to use the BigQuery service.

❺

DataFrame.union can be passed as a parameter to reduce, where it’ll union all the tables in my list pair-wise into a single table.

It’s easier to understand the reduce operation if we break it down into discrete steps.

I start with a range of years (in my example 2010 to 2020, including 2010 but excluding 2020). For this, I use the range() function.

I apply my helper function read_df_from_bq() to each year via a list comprehension, yielding a list of data frames. I don’t have to worry about memory consumption as the list contains only a reference to the data frame (DataFrame[…]).

As a reducing function, I use the DataFrame.union function. This method, when applied to a data frame (df.union()), takes a single parameter, since there is an implicit self that maps to the data frame calling the method. If we apply the function statically, using it from the generic DataFrame object, then it’ll take two data frames as parameter and union the data frames in a single one.

We could do this iteratively, using a for loop. In listing 8.14, I show how to accomplish the same goal without using reduce(). Since higher-order functions usually yield cleaner code, I prefer using them to looping constructs where it make sense.

Listing 8.14. Reading the gsod data from 2010 to 2020 using an iterative/looping approach. We need to load a first table to initialize the process.

gsod_alt = read_df_from_bq(2010)  ❶

for year in range(2011, 2020):

gsod_alt = gsod_alt.union(read_df_from_bq(year))

❶

When using a looping approach to union tables, you need an explicit starting seed. I use the table from 2010.

The reduce approach only works if all the tables union without any problem; same schema, from column name, order, and types. Google is doing us a solid here by having the data all pre-cleaned for us

Tip

If you are using a local Spark, loading 2010-2019 will make the examples in this chapter rather slow. I use 2018 only when working on my local instance so I don’t have to wait too long for code execution. Inversely, if you are working with a more powerful setup, you can add years to the range. The  gsod tables go back to 1929.

8.3.3  Scalar UDF

Scalar UDF are the most common type of pandas UDF. As their name indicates, they work on scalar values: for each record passed in, it needs to return one record. They behave just like regular Python UDF, with one key difference. Python UDF work on one record at a time and you express your logic through regular Python code. Scalar UDF work on one series at a time and you express your logic through pandas code. The difference is subtle and it’s easier to explain visually.

In a Python UDF, when you pass column objects to you UDF, PySpark will unpack each value, perform the computation, and then return the value for each record in a Column object. In a Scalar UDF, depicted in figure 8.5, PySpark will serialize (through PyArrow) each partitioned column into a pandas Series object (pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html). You then perform the operations on the Series object directly, returning a Series of the same dimension from your UDF. From an end-user perspective, they are the same functionally In Chapter 9, I discuss the performance implications of Python UDF vs. a scalar UDF.

Figure 8.5. Comparing a Python UDF to a pandas scalar UDF. The former splits a column in individual records, where the latter breaks them in Series.

Note

PySpark makes no guarantees about how the columns you pass to your scalar UDF will be split in Series, so you need to make sure your UDF doesn’t depend on a specific breakdown. For more control over the breaks, see  listing 8.17.

Now armed with the how it works? of scalar UDF, let’s create one ourselves. I chose to create a simple function that will transform Fahrenheit degrees to Celcius. In Canada, we use both scales depending on the usage: F for cooking, C for body or outside temperature. I have no idea of if 95 degrees F is hot or cold, but I know how to dress when it’s 10 degrees C, yet my dinner cooks at 350 F.

My function is depicted in listing 8.15. The building blocks are eerily similar; there are two main differences noticeable.

1. Instead of udf(), I use pandas_udf(), again, from the pyspark.sql.functions module. The first parameter is the return type of the function (DoubleType()) and the second is an indicator of the type of pandas UDF I am creating, here a PandasUDFType.SCALAR.

2. My code itself could be used as-is for a regular python UDF. I am (ab)using the fact that you can do arithmetic operations with pandas Series. You can use any Series method should you need to.

Listing 8.15. Creating a pandas scalar UDF that transforms Fahrenheit into Celcius. I use the pandas_udf decorator with a UDF type of PandasUDFType.SCALAR.

import pandas as pd

@F.pandas_udf(T.DoubleType(), F.PandasUDFType.SCALAR)  ❶

def f_to_c(degrees):

"""Transforms Farhenheit to Celcius."""

return (degrees - 32) * 5 / 9

f_to_c.func(pd.Series(range(32, 213)))  ❷

# 0        0.000000

# 1        0.555556

# 2        1.111111

# 3        1.666667

# 4        2.222222

#           ...

# 176     97.777778

# 177     98.333333

# 178     98.888889

# 179     99.444444

# 180    100.000000

# Length: 181, dtype: float64

gsod = gsod.withColumn("temp_c", f_to_c(F.col("temp")))

gsod.select("temp", "temp_c").distinct().show(5)

# +-----+-------------------+

# | temp|             temp_c|

# +-----+-------------------+

# | 37.2| 2.8888888888888906|

# | 85.9| 29.944444444444443|

# | 53.5| 11.944444444444445|

# | 71.6| 21.999999999999996|

# |-27.6|-33.111111111111114|

# +-----+-------------------+

# only showing top 5 rows

❶

For scalar UDF, the biggest change happens in the decorator used. I could use the  pandas_udf function directly too.

❷

To test my function, I apply it to every Fahrenheit value from 32 to 212 inclusively (0 to 100 Celcius), using the  func attribute that returns the local version of the UDF.

Scalar UDF, just like Python regular UDF, are very convenient when the record-wise transformation (or "mapping") you want to apply to your data frame is not available within the stock PySpark functions (pyspark.sql.functions). Creating a "Fahrenheit to Celcius" converter as part of core Spark would be a little intense, so using PySpark or (pandas) scalar UDF is a way to extend the core functionality with a minimum of fuss. Next, we see how to gain more control over the split and use the split-apply-combine pattern in PySpark.

Vocabulary matters: partitions vs. chunks

It can be tempting to use the word partitions when talking about how PySpark splits the data for a pandas UDF. Spark already has a concept of partitions, though: they refer to the physical data contained on the worker nodes. When working with pandas UDF, Spark can use the partitions as chunks, but can also decide to split them or move some data around. Because of this, I use chunks or groups (for grouped pandas UDF) instead. Less confusion, more data fun.

8.3.4  Grouped map UDF

Grouped map UDF are PySpark’s answer to the split-apply-combine pattern. At the core, split-apply-combine is just a series of three steps that are frequently used in data analysis.

1. First, you split your data set into logical chunks.

2. You then apply a function to each chunk independently.

3. Finally, you combine the chunks into a unified data set.

To be perfectly honest, I did not know this pattern’s name until somebody pointed at my code one day and said "this is some nice split-apply-combine work you did there". You probably use it intuitively as well. In PySpark’s world, I see it more as a divide and process move, as illustrated in figure 8.6.

Figure 8.6. Split-apply-combine depicted visually. We chunk/group the data frame, process each one with pandas, before unioning them into a (Spark) data frame again.

Before looking at the PySpark plumbing, we focus on the pandas side of the equation. Where scalar UDF were relying on pandas Series, grouped map UDF are using pandas DataFrame. Each logical chunk from step 1 in figure 8.6 becomes a DataFrame ready for action. Our UDF must also return a DataFrame.

Grouped map UDF also take pandas_udf() as a decorator, this time with a type of PandasUDFtype.GROUPED_MAP. The return type is also more verbose: since we have multiple columns in the pandas DataFrame, we have to provide the schema in a StructType(). For a deeper dive into schemas, head to chapter 6.

Listing 8.16. A grouped map UDF that normalizes (min-max) the temperature given a set of values. The code in the scale_temperature() function is regular pandas code.

@F.pandas_udf(

T.StructType(

[

T.StructField("stn", T.StringType()),

T.StructField("year", T.StringType()),

T.StructField("mo", T.StringType()),

T.StructField("da", T.StringType()),

T.StructField("temp", T.DoubleType()),

T.StructField("temp_norm", T.DoubleType()),

]

),

F.PandasUDFType.GROUPED_MAP,

)

def scale_temperature(temp_by_day):

"""Returns a simple normalization of the temperature for a site.

If the temperature is constant for the whole window, defaults to 0.5."""

temp = temp_by_day.temp

answer = temp_by_day[["stn", "year", "mo", "da", "temp"]]

if temp.min() == temp.max():

return answer.assign(temp_norm=0.5)

return answer.assign(temp_norm=(temp - temp.min()) / (temp.max() - temp.min()))

Tip

pandas_udf() will take a SQL-like schema as a string parameter too. For listing 8.16, we could have used stn string, name string, country string, year string, mo string, da string, temp double, temp_norm double as the return type.

Compared with the UDF seen so far in this chapter, the main difference is the return type of the UDF. In both the Python and the scalar UDF, we returned a single column. Here, we return a complete (pandas) DataFrame. In listing 8.16, our return DataFrame contains six columns. My UDF only adds one new column, temp_norm, which scales the temperature column received from a scale from zero to one. Since I have a division in my UDF, I am giving a reasonable value of 0.5 if the minimum temperature in my chunk equals the maximum temperature. By default, pandas will give an infinite value for division by zero: PySpark will interpret this as null.

Now with the "apply" step done, the rest is a piece of cake. I broke the punch in chapter 5: we use groupby() to split a data frame in manageable chunks and then pass our function to the apply() method. You can see the result in listing 8.17.

Listing 8.17. Split-apply-combing in PySpark: we groupby() records in a GroupedData object and apply() our UDF to each group.

gsod = gsod.where(F.col("year") == "2018")  ❶

gsod = gsod.groupby("stn", "year", "mo").apply(scale_temperature)

gsod.show(5, False)

# +------+----+---+---+-------------------+-------------------+

# |stn   |year|mo |da |temp_c             |temp_norm          |

# +------+----+---+---+-------------------+-------------------+

# |010250|2018|12 |08 |-5.666666666666667 |0.06282722513088991|

# |010250|2018|12 |27 |-2.0555555555555554|0.40314136125654443|

# |010250|2018|12 |31 |-1.6111111111111103|0.4450261780104712 |

# |010250|2018|12 |19 |-2.4444444444444438|0.3664921465968586 |

# |010250|2018|12 |04 |2.5555555555555562 |0.8376963350785341 |

# +------+----+---+---+-------------------+-------------------+

# only showing top 5 rows

❶

If you are working locally, keeping a single year worth of data will make sure you don’t wait too long to get your results.

I group by three fields, stn, year, and mo. Unlike the groupby()/agg() combo seen in chapter 5, where the keys are implicitly present in the resulting data frame, the UDF applied needs to return any column that we want in our resulting data frame. My UDF has six columns in its return value, the data frame after apply() has the same six, following the same type equivalence seen in table 8.1. In practice, you create a grouped map UDF with a groupby() pattern in mind, so there is a low risk of a mismatch.

Warning

With great power comes great responsibility: when grouping by your data frame, make sure each chunk is "pandas-size", i.e. it can be loaded comfortably in memory. If one or more chunks is too big, you’ll get an out-of-memory exception.

Grouped map UDF shine when you have distinct groups of data that you can process independently. In the case of listing 8.17, we scale the temperature for each combination of (station, year, month). The moment you feel like your code can process some distinct groups in your data frame, a grouped map UDF is a great choice.

8.3.5  Grouped aggregate UDF

We finish our tour of pandas user-defined functions with the grouped aggregate UDF. They can be thought of a combination of the ones we saw so far, as they take pandas Series as parameters but return a simple scalar value. In that sense, they are akin to Spark aggregate functions: each group is summarized by a single value.

For grouped aggregate UDF, we still rely on the "split" step provided by groupby() — which makes them like grouped map UDF — but this time, we apply our UDF to the agg() method, just aggregate functions. For my grouped aggregate UDF, I wanted to do something a little more complex than reproducing the common offenders (count, min, max, average). In listing 8.18, I compute the linear slope of the (scaled) temperature for a given period, using scikit-learn’s LinearRegression object. You do not need to know scikit-learn or machine learning to follow along: I’m using basic functionality and explain each step.

Note

This is not a machine learning exercise: I am just using scikit-learn’s plumbing to create a feature. Machine learning in Spark is covered in part 3 of this book. Don’t take this code as a robust model training exercise!

Listing 8.18. Creating a grouped aggregate UDF that computes the slope of a set of temperature

from sklearn.linear_model import LinearRegression  ❶

@F.pandas_udf(T.DoubleType(), F.PandasUDFType.GROUPED_AGG)

def rate_of_change_temperature(day, temp):

"""Returns the slope of the daily temperature for a given period of time."""

return (

LinearRegression()  ❷

.fit(X=day.astype("int").values.reshape(-1, 1), y=temp)  ❸

.coef_[0]  ❹

)

❶

I import the linear regression object from  sklearn.linear_model

❷

I initialize the  LinearRegression object.

❸

The  fit method trains the model, using the day Series as a feature and the temp series as the prediction.

❹

Since I have only one feature, I select the first value of the  coef_ attribute as my slope.

To train a model in scikit-learn, we start by initializing the model object. In this case, I use LinearRegression() without any other parameters. I then fit the model, providing X, my feature matrix, and y, my prediction vector. In this case, since I have a single feature, I need to reshape my X matrix or scikit-learn will complain about a shape mismatch.

At the end of the fit method, our LinearRegression object has trained a model and, in the case of a linear regression, keeps its coefficient in a coef_ vector. Since I really just care about the coefficient, I just extract and return it.

It’s easy to apply a grouped aggregate UDF to our data frame. In listing 8.19, I groupby() the station code, name, and country, as well as the year and the month. I pass my newly created grouped aggregate function as a parameter to agg(), passing my Column objects as parameter to the UDF.

Listing 8.19. Applying our grouped aggregate UDF using agg(). Our UDF behaves just like a Spark aggregate function.

result = gsod.groupby("stn", "year", "mo").agg(

rate_of_change_temperature(gsod["da"], gsod["temp_norm"]).alias(  ❶

"rt_chg_temp"

)

)

result.show(5, False)

# +------+----+---+---------------------+

# |stn   |year|mo |rt_chg_temp          |

# +------+----+---+---------------------+

# |010250|2018|12 |-0.01014397905759162 |

# |011120|2018|11 |-0.01704736746691528 |

# |011150|2018|10 |-0.013510329829648423|

# |011510|2018|03 |0.020159116598556657 |

# |011800|2018|06 |0.012645501680677372 |

# +------+----+---+---------------------+

# only showing top 5 rows

result.groupby("stn").agg(

F.sum(F.when(F.col("rt_chg_temp") > 0, 1).otherwise(0)).alias("temp_increasing"),

F.count("rt_chg_temp").alias("count"),

).where(F.col("count") > 6).select(

F.col("stn"),

(F.col("temp_increasing") / F.col("count")).alias("temp_increasing_ratio"),

).orderBy(

"temp_increasing_ratio"

).show(

5, False

)

# +------+---------------------+  ❷

# |stn   |temp_increasing_ratio|

# +------+---------------------+

# |681115|0.0                  |

# |384572|0.0                  |

# |682720|0.0                  |

# |672310|0.0                  |

# |654530|0.08333333333333333  |

# +------+---------------------+

# only showing top 5 rows

❶

Applying a grouped aggregate UDF is the same as using a Spark aggregating function: you add it as an argument to the  agg() method of the GroupedData object.

❷

I am looking at the stations having the lowest proportion of increasing temperature for a month, given that they have data for at least 6 distinct months. That can suggest data quality issues or skew in the data collection.

8.3.6  Going local to troubleshoot pandas UDF

Pandas UDF are quite useful to extend PySpark with transformations that are not included in the pyspark.sql module. I find that they’re also quite easy to understand but pretty hard to get right. I finish this chapter with a few pointers on testing out and debugging your pandas UDF.

The most important aspect of an pandas UDF (and any UDF) is that it need to work on the non-distributed version of your data. For regular UDF, this means passing any argument of the type of values you expect should yield an answer. As an example, our function in section 8.2 took an array of two integers: it needs to work for any arrays of two integers, including a potential zero as a denominator. The same is true for any pandas UDF: you need to be lenient with the input you accept and strict with the output you provide.

To test your pandas UDF, my favorite strategy is always to bring a sample of the data locally (one chunk) and test my function. This way, I can play around in the REPL until I get it just right, then promote it to my script. I show an example of the rate_of_change_temperature() UDF, applied locally, in listing 8.20.

Listing 8.20. Moving one station, one month worth of data in a local pandas DataFrame to test my rate_of_change_temperature() function.

gsod_local = gsod.where("year = '2018' and mo = '08' and stn = '710920'").toPandas()

print(rate_of_change_temperature.func(gsod_local["da"], gsod_local["temp_norm"]))

# -0.007830974115511494

When bringing a sample of your data frame into a pandas data frame for a grouped map or grouped aggregate UDF, you need to ensure you’re getting a full chunk to reproduce the results. In our specific case, since we grouped by "station", "year", "month", I brought one station, one month (one specific year/month, to be precise) of data. Since the grouping of the data happens at PySpark’s level (via groupby()), you need to think the filters for your sample data in the same fashion.

This is a very quick overview of a basic strategy to confirm your code is doing what you’re expecting. In Chapter 14, I cover PySpark code testing, which includes testing UDF, both PySpark and pandas.

User-defined functions are probably the most powerful feature PySpark offers for data manipulation. While the standard data manipulation API provided a lot of functionality out of the box, you have the option to outgrow what’s provided and write your own functions, using Python and pandas. Once written, scaling them to PySpark is as easy as decorating them. Python, pandas, Spark, they all work together now.

8.4  SUMMARY

- The most low level and flexible way of running Python code within the distributed Spark environment is to use the resilient distributed dataset (RDD). With an RDD, you have no structure imposed on your data and need to manage type information into your program, and defensively code against potential exceptions.

- The API for data processing on the RDD is heavily inspired by the MapReduce framework. The same ideas are permeating to the data frame abstraction, which can be seen as a specialized and structured RDD.

- The data frame’s most basic Python code promotion functionality, called the (PySpark) UDF, emulates the "map" part of the RDD. You use it as a scalar function, taking Column objects as parameters and returning a single Column.

- PySpark provides three UDF that leverages pandas serialization and processing (hence their name, "pandas UDF"): the scalar version, which provides similar functionality to the Python UDF, the grouped map, that splits the data frame into chunks and processes them using pandas code on a DataFrame, and the grouped aggregate, which takes Columns and processes them like pandas Series, returning a scalar value.

8.5  EXERCISES

8.5.1  Exercise 8.1

Using the following definitions, create a temp_to_temp(value, from, to) that takes a numerical value in from degrees and converts it to to degrees.

- C = (F - 32) * 5 / 9 (Celcius)

- K = C + 273.15 (Kelvin)

- R = F + 459.67 (Rankine)

8.5.2  Exercise 8.2

Correct the following UDF so it doesn’t generate an error.

@F.udf(T.IntegerType())

def naive_udf(t: str) -> str:

...

return answer * 3.14159

8.5.3  Exercise 8.3

Modify listing 8.16 to use Celcius degrees instead of Fahrenheit. How is the result of the UDF different if applied to the same data frame?

8.5.4  Exercise 8.4

Taking listing 8.17, what happens if we apply our grouped map UDF like so instead?

gsod_exo = gsod.groupby("year", "mo").apply(scale_temperature)

8.5.5  Exercise 8.5

Modify listing 8.18 to return both the intercept of the linear regression, as well as the slope, in an ArrayType. (Hint: the intercept is in the intercept_ attribute of the fitted model.)

A

APPENDIX A. SOLUTIONS TO THE EXERCICES

This appendix contains the solutions to the exercises present in the book.

CHAPTER 4

Exercise 4.1

ANSWER:

sample = spark.read.csv("sample.csv",

sep=",",

header=True",

quote="$",

inferSchema=True)

EXPLANATION:

- sample.csv is the name of the file we want to ingest.

- The record delimiter is the comma. Since we are asked to provide a value there, I pass the comma character , explicitly, knowing it is the default one.

- The file has a header row, so I input header=True

- The quoting character is the dollar sign character, $, so I pass it as an argument to quote.

- Finally, since inferring the schema is nice, I pass True to inferSchema.

Exercise 4.2

ANSWER:

DIRECTORY = "../../data/Ch04"

logs_raw = spark.read.csv(os.path.join(DIRECTORY, "BroadcastLogs_2018_Q3_M8.CSV"),)

logs.printSchema()

# root

#  |-- _c0: string (nullable = true)

logs.show(5)

# +--------------------+

# |                 _c0|

# +--------------------+

# |BroadcastLogID|Lo...|

# |1196192316|3157|2...|

# |1196192317|3157|2...|

# |1196192318|3157|2...|

# |1196192319|3157|2...|

# +--------------------+

# only showing top 5 rows

Two major differences:

1. PySpark globbed everything into a single string column, since it did not encounter the default delimiter (,) consistently in the records.

2. It named the record _c0, the default convention when it has no information about column names.

Exercise 4.3

ANSWER:

logs_clean = logs.select(*[x for x in logs.columns if not x.endswith("ID")])

EXPLANATION:

I use the list comprehension trick on the data frame’s columns, using the filtering clause if not x.endswith("ID") to keep only the columns that do not end with "ID".

Exercise 4.4

ANSWER:

c

EXPLANATION:

Both item and UPC match a columns, while prices doesn’t. PySpark will ignore the non-existent columns passed to drop().

CHAPTER 5

Exercise 5.1

ANSWER:

left

EXPLANATION:

A left_semi join only keep the records in left where the my_column value is also present in the my_column column in right. A left_anti join is the opposite: it keep the recorts not present. Unioning those two together results in the original data frame, left.

Exercise 5.2

ANSWER:

left.join(right, how="left",

on=left["my_column"] == right["my_column"]).where(

right["my_column"].isnull()

).select(left["my_column"]).

EXPLANATION:

When performing an inner join, all the records from the left data frame are kept in the joined data frame. If the predicate in unsuccessful, then the column values from the right table are all set to null for the affected records. We just have to filter to keep only the unmatched records and then select the left["my_column"] column.

Exercise 5.3

ANSWER:

left.alias("l").join(right.select("my_column").distinct().alias("r"), how="left",

on=F.col("l.my_column") == F.col("r.my_column")).where(

~F.col("r.my_column").isnull()

).select(F.col("l.my_column"))

EXPLANATION:

This follows the same pattern as the left anti join, but with a few more tricks.

A left semi join is equivalent to keep only the left records that resolve the predicate. Because a left join would duplicate left records if they are matched more than once in the right table, we have to remove the potentially duplicate values.

We can’t distinct() at the end, since this would remove the duplicate values in the left table that we want to keep.

CHAPTER 8

Exercise 8.1

ANSWER:

I will create a sample data frame before providing the answer, to see it in action.

# I assume spark is initialized and that `spark` exists as a variable

from operator import add

exo_rdd = spark.sparkContext.parallelize(list(range(100)))

# Answer below

exo_rdd.map(lambda _: 1).reduce(add)

EXPLANATION:

I start by mapping each element to the value 1, regardless of the input. The _ in the lambda function doesn’t bind the elements because we don’t process the element: we just care that they exits. After the map operation, we have a RDD containing only the value 1. We can reduce(sum) to get the sum of all the ones, which yields the number of elements in the RDD.

Exercise 8.2

ANSWER:

a

EXPLANATION:

Filter will drop any values when the predicate (the function passed as an argument) returns a falsey value. In Python, 0, None, and empty collections are falsey. Since the predicate returns the value unchanged, 0, None, [] and 0.0 are falsey and filtered out, leaving only [1] as the answer.

B

APPENDIX B. INSTALLING PYSPARK LOCALLY

Last update: 2020-06-21

Python 2 is now officially unsupported as of January 1st, 2020. At the moment, most OSes are in the transitional period between Python 2 and 3, which is why I spend a little time discussing how to install Python 3. I expect this guide to become simpler as time goes.

I am currently targeting the most recent version of each OS as of time of last update.

- Windows 10

- OS.X Catalina

- GNU/Linux Ubuntu 20.04 LTS

This appendix covers the installation of standalone Spark and PySpark on your own computer, whether it’s running Windows, Os.X or Linux.

Having a local PySpark cluster means that you’ll be able to experiment with the syntax, using smaller data sets. You don’t have to acquire multiple computers or spend any money on managed PySpark on the cloud until you’re ready to scale your programs.

Spark is a complex piece of software and most guides out there are over-complicating the installation proces. We’ll take a much simpler approach by installing the bare minimum to start, and building from there. Our goals are as follow:

- Install Java (Spark is written in Scala, which runs on the Java Virtual Machine, or JVM).

- Install Spark

- Install Python 3 and IPython

- Launch a PySpark shell using IPython

- (Optional): Install Jupyter and use it with PySpark.

B.1  WINDOWS

When working on Windows, you either have the option to install Spark directly on Windows, or to use WSL (Windows Subsystem for Linux). If you want to use WSL, follow the instructions at aka.ms/wslinstall and then follow the instructions for GNU/Linux. If you want to install on plain Windows, follow the rest of this section.

B.1.1  Install Java

The easiest way to install Java on Windows is to go on www.java.com and follow the download and installation instructions for downloading Java 11. Make sure to read the installer steps to avoid installing non-useful software!

Warning

If you are installing Spark 2.4.6 or before, you will need to install Java 8 instead of Java 11.

B.1.2  Install 7-zip

Spark is available as a GZIP archive (.tgz) file on their website. By default, windows doesn’t provide a native way to extract those files. The most popular option is 7-zip [8]. Simply go on the website, download the program and follow the installation instructions.

B.1.3  Download and install Apache Spark

Go on the Apache website and download the latest Spark release. You shouldn’t have to change the default options, but Figure-B.1 displays the ones I see when I navigate to the download page. Make sure to download the signatures and checksums if you want to validate the download (step 4 on the page).

Figure B.1. The options to download Spark

Once you have downloaded the file, unzip the file using 7-zip. I recommend putting the directory under C:\Users\[YOUR_USER_NAME]\spark .

Next, we need to download a winutils.exe to prevent some cryptic Hadoop errors. Go on the github.com/cdarlint/winutils repository and download the winutils.exe file in the hadoop-2.7.X/bin directory where X is the highest number. As of the time of writing, it is 2.7.7. Keep the README.md of the repository handy.

Tip

Spark is also available with a more recent Hadoop, but it is not the default option. If you select "Pre-built for Apache Hadoop 3.2 and later" in step 2 of the download form, take the most recent version of winutils.

Place the winutils.exe in the bin directory of your Spark inrtallation. Then, set the environment variables as listed on the winutils repository’s README.md To do so, open the start menu and search for "Edit the system environment variables". Click on the "Environment variables button" (see Figure-B.2) and then add them there. You will also need to set SPARK_HOME to the directory of your Spark installation (C:\Users\[YOUR-USER-NAME]\spark\bin). Finally, add the %SPARK_HOME%\bin directory to your PATH environment variable.

Note

For the  PATH variable, you might already have some in there. If this is the case, double click on the variable and append %HADOOP_HOME%\bin to the list, as well as %SPARK_HOME%\bin.

Figure B.2. Setting environment variables for Hadoop on Windows

B.1.4  Install Python

The easiest way to get Python 3 is to use the Anaconda Distribution. Go to www.anaconda.com/distribution and follow the installation instructions, making sure you’re getting the 64-bits Graphical installer for Python 3.X for your OS.

Once Anaconda is installed, we can activate the Python 3 environment by selecting the "Anaconda Powershell Prompt" in the start menu. If you want to create a dedicated virtual environment for PySpark, use the following command.

$ conda create -n pyspark python=3.8 pandas pyspark=3.0.0

Warning

Python 3.8 is supported only using Spark 3.0. If you use Spark 2.4.X or before, be sure to specify Python 3.7 in you environment creation.

Then, to select your newly created environment, just input conda activate pyspark in the Anaconda Prompt.

B.1.5  Launching an iPython REPL and starting PySpark

If you have configured the SPARK_HOME and PATH variables, your Python REPL will have access to a local instance of pyspark. Follow the next code block to launch iPython.

Tip

If you aren’t comfortable with the Command Line and Powershell, I’ve personally learned to use it using  Learn Powershell in a Month of Lunches by Don Jones and Jeffery D. Hicks (Manning, 2016).

conda activate pyspark ❶

ipython

❶

Only if you have created a  pypark virtual environment.

Then, within the REPL, you can import pyspark and get rolling.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

Note

Spark provides a  pyspark.cmd helper command through the bin directory of your Spark installation. I prefer accessing PySpark through a regular Python REPL when working locally as I find it easier to install libraries and know exactly which Python you’re using. It also interfaces well with your favorite editor.

B.1.6  (Optional) Install and run Jupyter to use Jupyter notebook

Since we have configured PySpark to be discovered from a regular Python process, we don’t have any further configuration to do to use it with a notebook. In your Anaconda Powershell window, install jupyter using the following command.

conda install -c conda-forge notebook

You can now run a Jupyter notebook server using the following command.

jupyter notebook

[8] www.7-zip.org/

B.2  MACOS

With macOS, the easiest option — by far — is to use the HomeBrew apache-spark package. It takes care of all dependencies (I still recommend using Anaconda for managing Python environments, for simplicity).

B.2.1  Install Homebrew

HomeBrew is a package manager for OS.X. It provides a simple command line interface to install many popular software packages and keep them up to date. While you can follow the manual "download and install" steps you’ll find on the Windows OS with little change, Homebrew will simplify our installation process to a few commands.

To install Homebrew, go to brew.sh and follow the installation instructions. You’ll be able to interact with Homebrew through the brew command.

B.2.2  Install Java and Spark

Input the following command in a terminal.

$ brew install apache-spark

You can specify the version you want; I recommend getting the latest by passing no parameters.

B.2.3  Install Anaconda/Python

The easiest way to get Python 3 is to use the Anaconda Distribution. Go to www.anaconda.com/distribution and follow the installation instructions, making sure you’re getting the 64-bits Graphical installer for Python 3.X for your OS.

$ conda create -n pyspark python=3.8 pandas pyspark=3.0.0

If it’s your first time using Anaconda, follow the instructions to register your shell.

Warning

Python 3.8 is supported only using Spark 3.0. If you use Spark 2.4.X or before, be sure to specify Python 3.7 in you environment creation.

Then, to select your newly created environment, just input conda activate pyspark in the Anaconda Prompt.

B.2.4  Launching a iPython REPL and starting PySpark

Homebrew should have the SPARK_HOME and PATH environment variables, so your Python REPL will have access to a local instance of pyspark. You just have to type the following.

conda activate pyspark ❶

ipython

❶

Only if you have created a  pypark virtual environment.

Then, within the REPL, you can import pyspark and get rolling.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

B.2.5  (Optional) Install and run Jupyter to use Jupyter notebook

Since we have configured PySpark to be discovered from a regular Python process, we don’t have any further configuration to do to use it with a notebook. In your Anaconda Powershell window, install jupyter using the following command.

conda install -c conda-forge notebook

You can now run a Jupyter notebook server using the following command.

jupyter notebook

B.3  GNU/LINUX AND WSL

B.3.1  Install Java

Most GNU/Linux distributions provide a package manager. OpenJDK version 11 is available through the software repository.

`sudo apt-get install openjdk-11-jre`

Warning

If you want to install a version of Spark prior to 3.0.0, install  openjdk-8-jre instead.

B.3.2  Installing Spark

Go on the Apache website and download the latest Spark release. You shouldn’t have to change the default options, but Figure-B.1 displays the ones I see when I navigate to the download page. Make sure to download the signatures and checksums if you want to validate the download (step 4 on the page).

Tip

On WSL (and sometimes Linux), you don’t have a graphical user interface really available. The easiest way to download Spark is to go on the website, follow the line, copy the link of the nearest mirror and past it along with wget command.

wget [YOUR_PASTED_DOWNLOAD_URL]

If you want to know more about using the command line on Linux (and Os.X) proficiently, a good free reference is  The Linux Command Line by William Shotts [9]. It is also available as a paper or e-book (No Starch Press, 2019).

Once you have downloaded the file, unzip the file (using 7-zip on Windows). If you are using the command line, the following command will do the trick. Make sure you’re replacing the spark-[…].gz by the name of the file you just downloaded.

tar -xvzf spark-[...].gz

This will unzip the content of the archive into a directory. You can now rename and move the directory to your liking. I usually put it under /home/[MY-USER-NAME]/bin/spark-3.0.0/ (and rename if the name is not identical) and the instructions will use that directory.

Set the following environment variables.

echo 'export SPARK_HOME="$HOME/bin/spark-3.0.0"' >> ~/.bashrc

B.3.3  Install Python 3 and IPython

Python 3 is already provided, you just have to install IPython. Input the following command in a terminal.

sudo apt-get install ipython3

Tip

You can also use Anaconda on GNU/Linux! Follow the instructions on the macOS section.

B.3.4  Launch PySpark with IPython

Launch an iPython shell.

ipython

Then, within the REPL, you can import pyspark and get rolling.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

B.3.5  (Optional) Install and run Jupyter to use Jupyter notebook

Since we have configured PySpark to be discovered from a regular Python process, we don’t have any further configuration to do to use it with a notebook. In your terminal, input the following to install jupyter.

pip install notebook

You can now run a Jupyter notebook server using the following command.

jupyter notebook

[9] linuxcommand.org/

Table of Contents

MEAP VERSION 6

About this MEAP

Welcome

Brief Table of Contents

1 Introduction

1.1 What is PySpark?

1.1.1 You saw it coming: What is Spark?

1.1.2 PySpark = Spark + Python

1.1.3 Why PySpark?

1.1.4 Your very own factory: how PySpark works

1.1.5 Some physical planning with the cluster manager

1.1.6 A factory made efficient through a lazy manager

1.2 What will you learn in this book?

1.3 What do I need to get started?

1.4 Summary

2 Your first data program in PySpark

2.1 Setting up the pyspark shell

2.1.1 The SparkSession entry-point

2.1.2 Configuring how chatty spark is: the log level

2.2 Mapping our program

2.3 Reading and ingesting data into a data frame

2.4 Exploring data in the DataFrame structure

2.4.1 Peeking under the hood: the show() method

2.5 Moving from a sentence to a list of words

2.5.1 Selecting specific columns using select()

2.5.2 Transforming columns: splitting a string into a list of words

2.5.3 Renaming columns: alias and withColumnRenamed

2.6 Reshaping your data: exploding a list into rows

2.7 Working with words: changing case and removing punctuation

2.8 Filtering rows

2.9 Summary

2.10 Exercises

Exercise 2.1

Exercice 2.2

Exercise 2.3

Exercise 2.4

Exercise 2.5

3 Submitting and scaling your first PySpark program

3.1 Grouping records: Counting word frequencies

3.2 Ordering the results on the screen using orderBy

3.3 Writing data from a data frame

3.4 Putting it all together: counting

3.4.1 Simplifying your dependencies with PySpark’s import conventions

3.4.2 Simplifying our program via method chaining

3.5 Your first non-interactive program: using spark-submit

3.5.1 Creating your own SparkSession

3.6 Using spark-submit to launch your program in batch mode

3.7 What didn’t happen in this Chapter

3.8 Scaling up our word frequency program

3.9 Summary

3.10 Exercises

3.10.1 Exercise 3.1

3.10.2 Exercise 3.2

3.10.3 Exercise 3.3

3.10.4 Exercise 3.4

4 Analyzing tabular data with pyspark.sql

4.1 What is tabular data?

4.1.1 How does PySpark represent tabular data?

4.2 PySpark for analyzing and processing tabular data

4.3 Reading delimited data in PySpark

4.3.1 Customizing the SparkReader object to read CSV data files

4.3.2 Exploring the shape of our data universe

4.4 The basics of data manipulation: diagnosing our centre table

4.4.1 Knowing what we want: selecting columns

4.4.2 Keeping what we need: deleting columns

4.4.3 Creating what’s not there: new columns with withColumn()

4.4.4 Tidying our data frame: renaming and re-ordering columns

4.4.5 Summarizing your data frame: describe() and summary()

4.5 Summary

5 The data frame through a new lens: joining and grouping

5.1 From many to one: joining data

5.1.1 What’s what in the world of joins

5.1.2 Knowing our left from our right

5.1.3 The rules to a successful join: the predicates

5.1.4 How do you do it: the join method

5.1.5 Naming conventions in the joining world

5.2 Summarizing the data via: groupby and GroupedData

5.2.1 A simple groupby blueprint

5.2.2 A column is a column: using agg with custom column definitions

5.3 Taking care of null values: drop and fill

5.3.1 Dropping it like it’s hot

5.3.2 Filling values to our heart’s content

5.4 What was our question again: our end-to-end program

5.5 Summary

5.6 Exercises

5.6.1 Exercise 5.1

5.6.2 Exercise 5.2

5.6.3 Exercise 5.3

6 Multi-dimentional data frames: using PySpark with JSON data

6.1 Reading JSON data: getting ready for the schemapocalypse

6.1.1 Starting small: JSON data as Python dictionary

6.1.2 Going bigger: reading JSON data in PySpark

6.2 Breaking the second dimension with complex data types

6.2.1 When you have more than one value: the array

6.2.2 The map type: keys and values within a column

6.3 The struct: nesting columns within colums

6.3.1 Navigating structs as if they were nested columns

6.4 Building and using the data frame schema

6.4.1 Using Spark types as the base blocks of a schema

6.4.2 Reading a JSON document with a strict schema in place

6.4.3 Going full circle: specifying your schemas in JSON

6.5 Putting it all together: reducing duplicate data with complex data types

6.6 Summary

6.7 Exercises

7 Bilingual PySpark: blending Python and SQL code

7.1 Banking on what we know: pyspark.sql vs plain SQL

7.2 Using SQL queries on a data frame

7.2.1 Promoting a data frame to a Spark table

7.2.2 Using the Spark catalog

7.3 SQL and PySpark

7.4 Using SQL-like syntax within data frame methods

7.4.1 Select and where

7.4.2 Group and order by

7.4.3 Having

7.4.4 Create tables/views

7.4.5 Union and join

7.4.6 Subqueries and common table expressions

7.4.7 A quick summary of PySpark vs. SQL syntax

7.5 Simplifying our code: blending SQL and Python together

7.5.1 Reading our data

7.5.2 Using SQL-style expressions in PySpark

7.6 Conclusion

7.7 Summary

7.8 Exercises

7.8.1 Exercise 7.1

7.8.2 Exercise 7.2

7.8.3 Exercise 7.3

7.8.4 Exercise 7.4

8 Extending PySpark with Python: RDD and user-defined-functions

8.1 PySpark, freestyle: the resilient distributed dataset

8.1.1 Manipulating data the RDD way: map, filter and reduce

8.2 Using Python to extend PySpark via user-defined functions

8.2.1 It all starts with plain Python: using typed Python functions

8.2.2 From Python function to UDF: two approaches

8.3 Big data is just a lot of small data: using pandas UDF

8.3.1 Setting our environment: connectors and libraries

8.3.2 Preparing our data

8.3.3 Scalar UDF

8.3.4 Grouped map UDF

8.3.5 Grouped aggregate UDF

8.3.6 Going local to troubleshoot pandas UDF

8.4 Summary

8.5 Exercises

8.5.1 Exercise 8.1

8.5.2 Exercise 8.2

8.5.3 Exercise 8.3

8.5.4 Exercise 8.4

8.5.5 Exercise 8.5

Appendix A. Solutions to the exercices

Chapter 4

Exercise 4.1

Exercise 4.2

Exercise 4.3

Exercise 4.4

Chapter 5

Exercise 5.1

Exercise 5.2

Exercise 5.3

Chapter 8

Exercise 8.1

Exercise 8.2

Appendix B. Installing PySpark locally

B.1 Windows

B.1.1 Install Java

B.1.2 Install 7-zip

B.1.3 Download and install Apache Spark

B.1.4 Install Python

B.1.5 Launching an iPython REPL and starting PySpark

B.1.6 (Optional) Install and run Jupyter to use Jupyter notebook

B.2 macOS

B.2.1 Install Homebrew

B.2.2 Install Java and Spark

B.2.3 Install Anaconda/Python

B.2.4 Launching a iPython REPL and starting PySpark

B.2.5 (Optional) Install and run Jupyter to use Jupyter notebook

B.3 GNU/Linux and WSL

B.3.1 Install Java

B.3.2 Installing Spark

B.3.3 Install Python 3 and IPython

B.3.4 Launch PySpark with IPython

B.3.5 (Optional) Install and run Jupyter to use Jupyter notebook