Building Big Data Pipelines with Apache Beam

Copyright © 2021 Packt Publishing

This is an Early Access product. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the content and extracts of this book may evolve as it is being developed to ensure it is up-to-date.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

The information contained in this book is sold without warranty, either express or implied. Neither the author nor Packt Publishing or its dealers and distributors will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: Building Big Data Pipelines with Apache Beam

Early Access Production Reference: B16918

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80056-493-0

www.packt.com

Table of Contents

1. Building Big Data Pipelines with Apache Beam: Use a single programming model for both batch and streaming use cases

2. 1Introduction to Data Processing with Apache Beam

1. Technical requirements

2. Why Apache Beam?

3. Writing your first Pipeline

4. Running our Pipeline against streaming data

1. Exploring the key properties of Unbounded Data

5. Measuring event-time progress inside data streams

1. State, Triggers and Timers

6. Assigning data to windows

7. Defining the lifecycle of a state in terms of windows

1. Pane accumulation

8. Unifying batch and streaming data processing

9. Summary

1. Cover

2. Table of contents

Building Big Data Pipelines with Apache Beam: Use a single programming model for both batch and streaming use cases

Welcome to Packt Early Access. We’re giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time. You’ll be notified when a new version is ready.

This title is in development, with more chapters still to be written, which means you have the opportunity to have your say about the content. We want to publish books that provide useful information to you and other customers, so we’ll send questionnaires out to you regularly. All feedback is helpful, so please be open about your thoughts and opinions. Our editors will work their magic on the text of the book, so we’d like your input on the technical elements and your experience as a reader. We’ll also provide frequent updates on how our authors have changed their chapters based on your feedback.

You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book. Join the exploration of new topics by contributing your ideas and see them come to life in print.

1. Introduction to Data Processing with Apache Beam

2. Implementing, Testing, and Deploying Basic Pipelines

3. Implementing Pipelines using Stateful Processing

4. Structuring Code for Reusability

5. Using SQL for Pipeline Implementation

6. Using Your Preferred Language with Portability

7. Extending Apache Beam's IO Connectors

8. Understanding How Runners Execute Pipelines

1Introduction to Data Processing with Apache Beam

Data. Big Data. Real-time Data. Data Streams. Many buzzwords to describe many things, and yet they have many common properties. Many mind-blowing applications arise from successful application of theoretically simple logic – take data and produce knowledge. But even such a simple-sounding task turns out to be difficult when the amount of data needed to produce knowledge is ... huge, and still growing. Given the vast volumes of data produced by humanity every day, which tools to choose to turn our simple logic into scalable solutions? Into solutions that protect our investment into creating the data extraction logic, even in the presence of new requirements arising, or changing on a daily basis, and new data processing technologies being created? This book focuses on describing why Apache Beam might be a good solution to these challenges and provides a guide through the learning process.

In this chapter, we will cover the following topics:

- Why Apache Beam?

- Writing your first Pipeline

- Running a Pipeline against streaming data

- Measuring event time progress inside data streams

- Assigning data to windows

- Defining the lifecycle of a state in terms of windows

- Unifying batch and streaming data processing

Technical requirements

In this first chapter we will introduce some elementary Pipelines written using Apache Beam’s Java SDK. We will use the code located on GitHub, at: https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam

We will need the following tools:

- JDK11, possibly OpenJDK11 installed, JAVA_HOME set appropriately

- git

- Bash

Important note

Although it is possible to run many tools in this book using Windows shell, throughout this book we will focus on using Bash scripting only. We hope Windows users will be able to run Bash using virtualization or The Windows Subsystem for Linux or any similar technology.

First of all, we need to clone the repository.

1. We create a suitable directory for that and run:

$ git clone https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam.git

2. This will result in a directory  Building-Big-Data-Pipelines-with-Apache-Beam  being create in the working directory. We then run in this newly created directory:

$ ./mvnw clean install

Throughout this book, we will use the notation, that the character  $  denotes a Bash shell., therefore  $ ./mvnw clean install  would mean to run command  ./mvnw  in the top-level directory of the git clone (named  Building-Big-Data-Pipelines-with-Apache-Beam ). By using  chapter1$ ../mvnw clean install  we mean to run the specified command in subdirectory called  chapter1 .

Why Apache Beam?

There are several basic questions one might ask when considering a new technology to learn and apply in practice. Two fundamental questions might be:

- What problem that I’m struggling with can the new technology help me solve?

- What would be the costs?

Each sound technology has a well-defined selling point, something that can justify its existence in the presence of other competitive technologies. In the case of Apache Beam, this selling point could be reduced to single word: portability. Apache Beam is portable on several layers, namely:

- Portability of Apache Beam Pipelines between multiple Runners (by Runner we mean a technology, that actually executes distributed computation described by a Pipeline’s author).

- Portability of Data Processing Model of Apache Beam to various programming languages.

- Portability of Data Processing logic between Bounded and Unbounded Data.

Each of these points deserves a few words of explanation. By Runner portability, we mean the possibility to run existing Pipelines written in one of the supported programming languages (for instance Java, Python, Go, Scala or even SQL) against a data processing engine that can be chosen at runtime. A typical example of a Runner would be Apache Flink, Apache Spark, or Google Dataflow. But Apache Beam is by no means limited to these; new Runners are created as new technologies arise. And very likely there many more are yet to come.

By portability of Apache Beam’s Data Processing Model to various programming languages we mean the ability to provide support for multiple SDKs, independently of which language, or what technology a particular Runner uses. That way, one can code Beam Pipelines in Go language and run these against Apache Flink Runner, written in Java.

Last but not least, the core of Apache Beam’s Model is designed so that it is portable between Bounded and Unbounded data. Bounded data is what was historically called batch processing, while Unbounded data represents real-time processing, a reliable application crunching live data as it arrives into the system, producing low-latency output

Putting these pieces together, we can describe Apache Beam as a tool that lets you deal with your Big Data architecture with the following vision: Choose your preferred language, write your data processing Pipeline, run this Pipeline using a Runner of your choice, and do all this for both batch and real-time data at the same time.

Because everything comes at a price, you should expect to pay for a flexibility like this – this price would be a somewhat bigger overhead in terms of CPU and/or memory usage. The Apache Beam community works hard to make this overhead as small as possible, but chances are that this overhead will never be nullified.

If all of this sounds compelling to you, then we are ready to start a journey exploring the Apache Beam!

Writing your first Pipeline

Let's jump right into writing our first Pipeline. The first part of this book will focus on Apache Beam’s Java SDK. We assume that you are familiar with programming in Java and building a project using Apache Maven (or any similar). The following code can be found in class  com.packtpub.beam.chapter1.FirstPipeline  in sources for this chapter, that is in the  chapter1  module. We would like to ask you to walk through the code. We will highlight the most important parts here.

1. We need some (demo) input for our Pipeline. We will read this input from resource called  lorem.txt . The code is standard pure Java as follows.

ClassLoader loader = FirstPipeline.class.getClassLoader();

String file = loader.getResource("lorem.txt").getFile();

List<String> lines = Files.readAllLines(

Paths.get(file), StandardCharsets.UTF_8);

2. Next, we need to create an object  Pipeline , which is a container for a Direct Acyclic Graph (DAG) that represents data transformations needed to produce output from input data:

Pipeline pipeline = Pipeline.create();

Important note

There are multiple ways to create a Pipeline, this being the most simplistic. We will see different ways in the Chapter 2, Implementing, Testing, and Deploying Basic Pipelines.

3. After we create a Pipeline, we can start filling it with data. In Beam, data is represented by an object of type  PCollection . Each  PCollection  (a parallel collection) can be imagined as a line (an edge) connecting two vertices ( PTransforms , a parallel transform) in Pipeline's DAG.

4. The following code therefore creates first node in Pipeline. The node is a transform that takes raw input from List and creates new  PCollection :

PCollection<String> input = pipeline.apply(Create.of(lines));

Our DAG will then look as follows:

Figure 1.1 – A Pipeline containing single PTransform

5. Each  PTransform  can have one main output and possibly multiple side output PCollections. Each  PCollection  has to be consumed by another  PTransform , or it might be excluded from the execution. As we can see, our main output  PCollection  of  PTransform  called Create is not presently consumed by any  PTransform . We connect  PTransform  to a  PCollection  by applying this  PTransform  on the  PCollection . We do that by using the following code:

PCollection<String> words = input.apply(Tokenize.of());

This creates a new  PTransform  (Tokenize) and connects it to our input  PCollection  as follows:

Figure 1.2 – A Pipeline with two PTransforms

We’ll skip the details of how the Tokenize  PTransform  is implemented for now (we will return to that later, in Chapter 5, Using SQL for Pipeline Implementation, which describes how to structure code in general). Currently, all we have to remember is that the Tokenize  PTransform  takes input lines of text and splits each line into words, this producing a new  PCollection  that contains all words present on all lines of the input  PCollection .

6. We finish the Pipeline by adding two more  PTransforms . One will produce the famous word-count example, so popular in every Big Data textbook. And the last one will simply print the output  PCollection  to standard output:

PCollection<KV<String, Long>> result =

words.apply(Count.perElement());

result.apply(PrintElements.of());

Details of both the  Count   PTransform  - which is Beam’s built-in  PTransform  – and  PrintElements  – which is a user-defined  PTransform  – will be discussed later. If we currently focus on the Pipeline construction process, we can see that our Pipeline now looks as follows:

Figure 1.3 – Final “Word-count” Pipeline

7. After we define this Pipeline, we must actually run it. This is done by the following line:

pipeline.run().waitUntilFinish();

This causes the Pipeline to be passed to a Runner (configured in the Pipeline; if omitted, it defaults to a runner available on  Classpath ). The standard default Runner is the DirectRunner, which executes the Pipeline in the local JVM only. This runner is mostly suitable for testing, as we will see in the next chapter.

8. We can run this Pipeline by executing:

chapter1$ ../mvnw exec:java \

-Dexec.mainClass=com.packtpub.beam.chapter1.FirstPipeline

in the code examples for module  chapter1 , which will yield the expected output on standard output.

Important note

The ordering of output is not defined and might – and will – vary among multiple runs. That is to be expected and is consequence of the fact that the Pipeline is underneath executed in multiple threads.

9. A very useful feature is that the application of  PTransform  to  PCollection  can be chained, so the preceding code can be simplified to:

ClassLoader loader = ...

FirstPipeline.class.getClassLoader();

String file =

loader.getResource("lorem.txt").getFile();

List<String> lines = Files.readAllLines(

Paths.get(file), StandardCharsets.UTF_8);

Pipeline pipeline = Pipeline.create();

pipeline.apply(Create.of(lines))

.apply(Tokenize.of())

.apply(Count.perElement())

.apply(PrintElements.of());

pipeline.run().waitUntilFinish();

This style is very readable and – when used properly and with care – greatly improves readability of the resulting code.

Now that we have written our first Pipeline, let’s see how to port it from bounded data source to a streaming source!

Running our Pipeline against streaming data

Let’s focus on how we need to change this code to enable it to run against a streaming data source. We first have to define what we mean by a data stream. A stream is a continuous flow of data, without any prior information about the cardinality of the data set. The data set can be either finite or infinite; the common property is that we do not know that in advance. Because of this property, the streaming data is often called Unbounded data, because – as opposed to Bounded data – no prior bounds regarding the cardinality of the data set can be made.

The absence of bounds is one property that makes the processing of data streams trickier (the other is that bounded data sets can be viewed as static, while unbounded data is, by definition, changing over time). We’ll investigate these properties later in this chapter and we’ll see how we can leverage them to define a Beam’s unified model for data processing.

For now, let’s imagine our Pipeline is given a source, which gives one line of text at a time, but does not give any signal of how many more elements is there going to be. How do we need to change our data processing logic, in order to be able to extract information from such a source?

1. We’ll update our Pipeline to use a streaming source. What we need is to change the way we created our input PCollection of lines coming from a List via  Create  PTransform to a streaming input. Apache Beam has a utility for this called TestStream as follows. Create a  TestStream  - a utility that emulates unbounded data source. The  TestStream  needs a  Coder , details of which will be skipped for now and we’ll return to this concept later in the Chapter 2, Implementing, Testing, and Deploying Basic Pipelines:

TestStream.Builder<String> streamBuilder =

TestStream.create(StringUtf8Coder.of());

2. Next, we fill the  TestStream  with data. Note that we need a timestamp for each record, so that the  TestStream  can emulate a real stream, which should have timestamps assigned for every input element:

Instant now = Instant.now();

// add all lines with timestamps to the TestStream

List<TimestampedValue<String>> timestamped =

IntStream.range(0, lines.size())

.mapToObj(i -> TimestampedValue.of(

lines.get(i), now.plus(i)))

.collect(Collectors.toList());

for (TimestampedValue<String> value : timestamped) {

streamBuilder = streamBuilder.addElements(value);

}

3. Last, we will apply this onto the Pipeline:

// create the unbounded PCollection from TestStream

PCollection<String> input =

pipeline.apply(streamBuilder.advanceWatermarkToInfinity());

We also encourage you to investigate the complete source code of class  com.packtpub.beam.chapter1.MissingWindowPipeline  to make sure everything is properly understood.

4. If we run the class with the following command:

chapter1$ ../mvnw exec:java \

-Dexec.mainClass=\

com.packtpub.beam.chapter1.MissingWindowPipeline

It will result in the following exception:

java.lang.IllegalStateException: GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow without a trigger. Use a Window.into or Window.triggering transform prior to GroupByKey.

The cause is that we need a way to identify (at least partial) completeness of data. That is to say – the data need (explicit or implicit) markers that define a condition that (when met) triggers a completion of a computation and outputs data from a PTransform. The computation can then continue from the values already computed or be reset to initial state.There are multiple ways to define such condition, one of them is to define time-constrained intervals called windows. A time-constrained window might be defined as data coming from a specific time-interval – for example between 1pm and 2pm.

5. As the exception suggests, we need to define a window to be applied to input data stream, in order to complete to definition of the Pipeline. The definition of a  Window  is somewhat complex, we will dive into all parameters later in this book, for now, we’ll define that one definition of such a Window would be the following:

PCollection<String> windowed =

words.apply(

Window.<String>into(new GlobalWindows())

.discardingFiredPanes()

.triggering(AfterWatermark.pastEndOfWindow()));

What this code does, is that it applies  Window.into  PTransform using GlobalWindow, which is a specific window, that contains whole data - meaning that it can be viewed as a window containing the whole history and future of the Universe. The complete code can be investigated in class  com.packtpub.beam.chapter1.FirstStreamingPipeline .

6. As usual, we can run this code using the following command:

chapter1$ ../mvnw exec:java \

-Dexec.mainClass=\

com.packtpub.beam.chapter1.FirstStreamingPipeline

This results in the same outcome as in the first example, with the same caveat – the order of output is not defined and will vary among multiple runs of the same code against the same data. The values will be absolutely deterministic, though.

Once we have successfully run our first streaming Pipeline, let’s dive into what exactly this streaming data is, and what to expect when we try to process them!

Exploring the key properties of Unbounded Data

In the previous section, we successfully ran our sample Pipeline against simulated unbounded data. We have seen that only a slight modification had to be made for the Pipeline to produce output in the streaming case. Let’s now dive a little deeper into the understanding, why this modification was necessary and how to code our Pipelines portable from the beginning.

First of all, we need to define a notion of time. In our everyday life, a time is a common thing we don't think that much about. We know what time it is at the moment, and we react to events that happen (more or less) instantly. We can plan for the future but cannot change the past.

When it comes to data processing, things change significantly. Let’s imagine a smart home application that reads data from various sensors and acts based on the values it receives. Such an application is depicted in the following diagram:

Figure 1.4 – A simple sensor data processing application

The application reads a stream of incoming sensor data, reads state associated with each device and/or other settings related to the data being processed, (possibly) updates the state and (possibly) outputs some resulting events or commands (for example, turn on a light if some condition is met).

Now, let’s imagine we want to make modifications to the application logic – we add some new smart features, and we would like to know how the logic would behave if it would have been fed with some historical events that we stored for a purpose like this. We cannot simply exchange the logic and push historical data through it, because that would result in incorrect modifications of the state – the state might have been changed from the time we recorded our historical data. We see that we cannot mix two times – the time at which we process our data and time at which the data originated. We usually call these two the processing time and the event time. The first one is the time we see on our clock when an event arrived to us, while the other is the time at which the event occurred. A beautiful demonstration of these two is depicted on the following table:

Figure 1.5 – Star Wars episodes in processing time and in event time

For those who might not be familiar with the Star Wars saga, the processing time here represents the order in which these movies were released, while the event time represents the actual order of episodes according to the happenings.By defining the event time and the processing time, we are able explain another weirdness of the streaming world – each data stream is inevitable unordered in terms of event time. What do we mean by that? And why that should be inevitable?

The out-of-orderness of a data stream is shown on the following diagram:

Figure 1.6 – Unordered data stream

The circles represent data points (events), on the X axis is processing time and on the Y axis is event time. The upper-left half of the square should be empty under normal circumstances, because that area represents events coming from the future – events with a higher event time than the current processing time. The rest represent some events that arrive with lower or higher delay from the time they occurred (event time). Vast majority of the delay is caused by technical reasons, like queueing in network stack, buffering, machine clocks out of sync, even outages of some parts of a distributed system. But there are even physical reasons why this happens - a vastly delayed data is what you see if you look at the sky during night. Light coming from stars we see by naked eye is delayed by as much as thousand years. Because even our physical reality works like this, the out-of-orderness is to be expected and has to be considered 'normal’ in any stream processing.

So, we defined what event time and what processing time is. We have a clock for measuring processing time. But what about event time? How do we measure that? Let’s find out!

Measuring event-time progress inside data streams

As we have shown, data streams are naturally unordered in terms of event time. Nevertheless, we need a way of measuring the completeness of our computation. Here is where another essential principle appears: a watermark.

Watermark is a (heuristic) algorithm, that gives us an estimation of how far we have got in the event time domain. A perfect watermark gives the highest possible time T, that guarantees that no further data arrives with event time < T. Let’s demonstrate this on the following screenshot:

Figure 1.7 – Watermark and late data

We can see from Figure 1.7 that the watermark is a boundary that moves along with the data points, ideally leaving all of the data on its left side. All data points lying on the right side (with processing time on X axis) are called late data. Such data typically requires special handling that will be described in the following chapter.

There are many ways to implement watermarks. They are typically generated at the source and propagated through the Pipeline. We will discuss the details of implementation of some watermarks in later chapters dedicated to IO connectors. Users typically does not have to generate watermarks themselves, although it is very useful to have a very good understanding of the concept.

State, Triggers and Timers

Each computation on a data stream which takes into account more than a single isolated event needs a state. The state holds (accumulates) values derived from the so-far processed stream elements. Let’s imagine we want to calculate current number of elements in a stream. We would do that as follows:

Figure 1.8 – Counting elements in a stream

The computational logic is straightforward – take the incoming element, read the current number of elements from the state, increment that number by one, store the new count into the state and emit the current value to the output stream.

As simple as this is, the overall picture is quite complex if we realize that what we are building here is an application that is supposed to run for very long time (theoretically, forever). Such a long-running application will necessarily face disruptions caused by failing hardware, software, or necessary upgrades of the application itself. Therefore, each (sane) stream processing application has to be fault-tolerant by design.

Ensuring fault-tolerance puts specific requirements on the state and on the stream itself. Specifically, we must ensure that we:

- Keep the state in a secure fault-tolerant storage.

- Retain the ability to restore both the state and the stream to a defined position.

Both of these requirements dictate that every fault-tolerant stream processing engine must provide state management, state access APIs, and incorporate the state into its core concepts. The same holds true for Apache Beam, and we’ll dive into the state concept in following chapters.

Our element-count example gives motivation for another question: when to output the resulting count. In the example, we output the current count for each input element. That might not be adequate for every application. Other options would be to output current value:

- In fixed periods of processing time (for instance, every 5 seconds).

- When the watermark reaches a certain time (for instance, output count, when our watermark signals that we have processed all data up to 5pm)

- When a specific condition is met in the data.

Such emitting conditions are called triggers. Each of these possibilities represent one option: a processing-time trigger, an event-time trigger, and a data-driven trigger. Apache Beam has full support for processing and event time triggers and supports one data-driven trigger, which is a trigger that outputs after a specific number of elements (for example, after each 10 elements).

We have already seen declaration of a trigger, remember?

PCollection<String> windowed =

words.apply(

Window.<String>into(new GlobalWindows())

.discardingFiredPanes()

.triggering(

AfterWatermark.pastEndOfWindow()));

This is one of many event-time triggers, that specifies, we want to output result, when our watermark reaches time, that is defined as end-of-window. We’ll dive into this when we’ll discuss the concept of windows.

Both event-time and processing-time triggers require an additional stream processing concept. This concept is a timer. A timer is a tool that lets an application specify a moment in either processing time or event time domain and, when that moment is reached, an application-defined callback hook is called. For the same reason as with state, timers also need to be fault tolerant (meaning that they have to be kept in fault-tolerant storage). Beam is purposely designed so that there is actually no way to access watermark directly, and the only way of observing a watermark is by event time timers. We will investigate timers in Chapter 3, Implementing Pipelines using Stateful Processing.

We now know that a streaming data processing engine needs to manage the application’s state for us, but what is the lifecycle of such a state? Let’s find out!

Assigning data to windows

We have already touched, but have not yet defined, the concept of a window. A window is a specific bounded range of data within a data stream. Beam has several types of pre-defined window functions, namely:

- Tumbling windows

- Sliding windows

- Session windows

Tumbling windows are for assigning data elements into single window of pre-defined length as follows:

Figure 1.9 – Tumbling windows

Tumbling windows can each have exactly the same fixed length (for instance 1 hour or 1 day – these are called Fixed Windows), or different length (for example 1 month in case of Calendar Windows). The common property is that event time of each element can be assigned to exactly one window, and that these windows cover a continuous (possibly) infinite time range without any gaps

Sliding windows are windows that assign data elements into multiple windows, shifted by a time period, called a slide as shown in the following figure:

Figure 1.10 – Sliding windows

Sliding windows have the same fixed window length (for instance 1 hour) and the same fixed slide (for instance 10 minutes). A sliding window of 1 hour with slide of 10 minutes assigns each event time into 6 distinct windows, each shifted by 10 minutes.

The last type of windows is called Session windows. This type is special in several ways. Unlike both previous types, session windows are key unaligned. What does that mean? Neither tumbling nor sliding windows depend on the data itself: each data element is assigned to a window (or several windows) based solely on element’s timestamp. The boundary of all windows in the stream is exactly aligned for all data. This is not the case for session windows – session windows split the stream into independent “sub-streams” based on a user provided key for each element in the stream. We can imagine the key as a color representing each stream element. Session windows group only elements having the same color, therefore windows in the stream are no longer aligned on the same boundary. We can illustrate this as follows:

Figure 1.11 – Session windows

We see from Figure 1.11 that different keys (types) of elements are grouped in different windows. There, one other parameter that has to be specified – a session gap duration. This duration is a timeout (in event time), that has to elapse between timestamps of two successive elements with the same key, in order to prevent assigning them in the same window. That is to say, as long as elements for a key arrive with frequency higher than the gap duration, all are placed in the same window. Once there is a delay of at least the gap duration, the window is closed, and another window will be created once new element arrive. This window is used heavily when analyzing user sessions in web click streams, which is what gave this type of windows their name.

There is one more special window type called Global window. This very special type of window assigns all data elements into a single window, regardless of their timestamp. The window therefore spans complete time interval from –infinity to +infinity. This window is used as a default window before any other window is applied. We’ll look into that later in this chapter.

Defining the lifecycle of a state in terms of windows

Windows are actually a way of scoping a state in computation. Each state is valid within a context of a window and each window has its own independent state.

Figure 1.12 illustrates state scoping:

Figure 1.12 – Scoping state within windows

The scoping of states by window brings up another crucial concept of stream processing – late data elements. One such element is shown on Figure 1.7.

We can state the problem as follows: When is the point where we can clear and discard the state that belong to a particular window? Obviously, it is impractical to keep all states of all windows open forever, because each window carries non-zero memory footprint and keeping the window around for unbounded time would cause memory to be depleted over time. On the other hand, deleting the state right after watermark passing the timestamp that marks end-of-window, would mean we need a perfect watermark (watermark that never produces late data). Any possible late data would mean we produce incorrect outputs – the state would be cleared before all data elements belonging to the respective window could be processed and hence would have to be dropped or would produce a completely wrong outcome.

One option would be to define such semantics, which would require the watermark to advance only when the probability of late data is sufficiently low. We would drop all data that arrived after the watermark and pretend that we didn’t see it. If the watermark produces sufficiently low number of such late data, the error introduced by the late data dropping could be made negligible. The crucial problem with this approach is that it necessarily introduces very high latency, due to the out-of-orderness nature of stream processing. We would therefore face a latency vs. correctness dilemma, while our goal ideally is to have both high correctness and low latency.

To resolve this dilemma, stream processing engines introduce an additional concept called allowed lateness. This defines a timeout (in event time) after which the state in a window can be cleared and all remaining data can be cleared. This option gives us the possibility to:

- Enable the watermark heuristic to advance sufficiently quickly not to incur unnecessary latency.

- Enable an independent measure of how many states are to be kept around even after their maximal timestamp has already passed.

We illustrate this concept in Figure 1.13, which shows a simple watermark heuristic which just shifts processing time by a constant duration (which will define minimal latency) and a late data boundary, which shifts the watermark by additional allowed lateness duration. This then might introduce data that will be actually dropped but can now be tuned independently:

Figure 1.13 – Allowed lateness

Important note

Practical watermark implementations do not typically use a fixed shift between watermark and processing time, but rather use statistics inferred from consumed data to produce a watermark that is non-linear in terms of processing time.

The definition of on-time and late data brings up one last technical term that appears in the context of triggers (see State, Trigger, and Timers, as a reminder). When a trigger condition is met and the trigger causes output data to be emitted down-stream, three possible conditions can occur:

- The watermark has not yet reached the end timestamp of a window.

- The watermark has crossed the window end-timestamp, and this is the first activation of a trigger since then.

- The watermark has passed the window end-timestamp, and this is not the first activation of a trigger.

According to these three conditions, we can mark the resulting downstream data element as:

- Early: The data is emitted prior to terminating the respective window's end-timestamp - this means that we output speculative partial results.

- On time: This marks data that was calculated once the window’s end-timestamp was reached.

- Late: This contains any output with possible late data incorporated.

Apache Beam calls data emitted as a result of trigger firings a Pane and puts the information about lateness or earliness of such firing into PaneInfo.

Pane accumulation

When a trigger fires and causes data to be output from the current window(s) to downstream processing, there are several options that can be done with both the state associated with the window and with the resulting value itself.

After a trigger fires and data is output downstream, we have essentially two options:

- Reset the state to empty (initial) state (discard), or

- Keep the state intact (accumulate)

This concept might be a little confusing, so we’ll demonstrate it on an example. Let’s assume that we want to count the number of elements in a stream every minute in processing time. Window functions in general are based on event time, so to get something that would resemble processing time window, we can do the following:

// Window into single window and specify trigger

PCollection<String> windowed =

words.apply(

Window.<String>into(new GlobalWindows())

.triggering(

Repeatedly.forever(

AfterProcessingTime.pastFirstElementInPane()

.plusDelayOf(Duration.standardSeconds(1))))

.discardingFiredPanes());

Please investigate the complete source code in class  com.packtpub.beam.chapter1.ProcessingTimeWindow.

We can run this Pipeline using:

chapter1$ ../mvnw exec:java \

-Dexec.mainClass=com.packtpub.beam.chapter1.ProcessingTimeWindow

Please feel free to experiment with changing  discardingFiredPanes  to  accumulatingFiredPanes  to see how the output differs. In the accumulation mode, the output contains the sum of elements from the beginning, while in discarding mode, it contains only increments from last trigger firing.

Now that we have discussed all the key properties of data streams, let’s see how we can use this knowledge to close the gap between batch processing and real-time streaming processing!

Unifying batch and streaming data processing

One of the core features that Apache Beam offers is portability of data processing Pipelines between batch and streaming processing. This began around 2004, with the famous white paper MapReduce: Simplified Data Processing on Large Clusters. The idea behind MapReduce is quite simple: divide a complex computation into several parts, each of which consists of two functions – Map and Reduce – and apply these functions in large scale using clusters of commodity hardware. The simplicity of the two building blocks gives rise to quite simple requirements in terms of fault tolerance, which is essential for any large distributed system.

Details of this system can be easily found online and are out of the scope of this book. We reference it here to demonstrate how and why data processing systems evolved from this moment on. The greatest benefit – massive parallel processing of data on clusters of computers that fail – is what enabled cost-effectiveness of these large computations and finally led to the great development – applications of deep learning and other computationally intensive approaches.

The approach has two major drawbacks:

- Complex algorithms typically requires a very difficult decomposition into Map and Reduce functions, chaining of multiple these stages, and so on.

- The latency of data processing is very high, due to the fact that all data has to be reprocessed from scratch and no continuous updates are possible.

At first, both of these drawbacks were addressed by different systems. Therefore, batch systems with higher level primitives (like joins and groupings) came out (for example, Apache Spark), while – at the same time – different systems tailored to low-latency processing came out (for example, Apache Storm). The evolution of these systems can be illustrated as follows:

Figure 1.14 – Evolving from Hadoop to Beam

Apache Beam was the first model to unify both of these evolving paths into a single model, targeted at both low latency and advanced programming models. This was enabled by a simple, but very crucial insight: batch semantics can be defined using streaming semantics (this statement is often rephrased as batch is a special case of streaming). Let’s see how exactly this was achieved.

Due to the described simplicity of parallelizing a chain of Map-Reduce operations, practically all batch systems targeting at improving the programming model were defining high-level abstractions, which then in turn translated to low-level map-reduce-like operations. Therefore, we can focus on the simple MapReduce paradigm for the batch case, which works as follows:

Figure 1.15 – Batch data flow

For clarity, we will shortly describe how the processing works. Each input record is fed into the Map function, producing possibly multiple key-value pairs. Each record with the same key is then grouped together and fed into Reduce function which, for given key and list of values, produces final outputs.

As we have seen, for streaming semantics, there are two more things to worry about: an event time of a stream element; and a window function that assigns these elements into windows (tumbling, sliding, session). Therefore, if we extend the batch processing with event time of each key-value pair and define a sensible default window, we get the situation depicted in Figure 1.16.

Each element is now equipped with a default timestamp (ts) and a default window.

Figure 1.16 – Batch data flow with streaming semantics

We have to define sensible defaults for the timestamp and window. Timestamp can be chosen at a fixed value (typically either the timestamp of the start of the batch computation, or –infinity) and default window is the Global Window. The reason why there is no meaningful option other than to put all elements into single window is that we must extends the  GroupByKey  operation in classical batch mode to  GroupByKeyAndWindow  to fulfil the streaming constraint that the state is bound to window. In order to be able to derive the batch semantics from the streaming one, we see that the default window must be to assign the complete input into single window (the Global Window).

Last but not least, we have to deal with how event time moves in our streaming pipeline running batch workflow. As we have seen, streaming processing uses watermarks to mark progress in event time. Batch semantics have no order in the input data set (or in how the input data set is processed) and therefore we are left with simply moving watermark from –infinity at the beginning and throughout the job and then shift it to +infinity once the job completes (more exactly, when the job finishes reducing the last key).

Important note

Under special circumstances, it is possible to smoothly advance event-time watermark even in the case of batch processing. We will learn more about this topic when we speak about stateful processing of time-series data.

To sum this up, we can see, that we can derive batch semantics from streaming semantics by application of the following:

- Assigning a fixed timestamp to all input key-value pairs.

- Assigning all input key-value pairs to Global Window.

- Moving watermark from –inf to +inf in one hop once all the data is processed.

The unified approach of Apache Beam then comes from the following logic: Code your Pipeline as a streaming Pipeline. Run it in both batch and streaming fashion.

There are some cases where a violation of this rule makes sense, but every time you code a batch-only Pipeline in Apache Beam, you should take one step back and think if it is really what you want.

Summary

In this chapter, we went over some of the basic theoretical concepts you will need to understand in order to keep up with the following chapters. Those include the difference between processing time and event time, which is the key insight for being able to define correctness of streaming computation. Processing time is mostly useful for defining the rate of (partial) result emission via triggers, because otherwise one would always have to wait for the end of window to get any result. We have seen how different accumulation modes affect the output of a computation.

We have walked through the lifecycle of states, as needed for aggregations. We have seen that watermarks are a systematic approach for definition of position in event time and, as such, define a relationship between event time and processing time. We also walked through how to write your first pipeline using Apache Beam. We’ll be using this as a foundation for everything we cover throughout the book.

In Chapter 2, Implementing, Testing, and Deploying Basic Pipelines, we’ll be developing our understanding of pipelines even further, covering implementation, testing, and deployment of Pipelines to real distributed runners.