Designing Machine Learning Systems

Iterative Processes for Deployable, Reliable, and Scalable Machine Learning

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Chip Huyen

Designing Machine Learning Systems

by Chip Huyen

Copyright © 2022 Huyen Thi Khanh Nguyen. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: Rebecca Novack

- Development Editor: Jill Leonard

- Production Editor: Kristen Brown

- Copyeditor:

- Proofreader:

- Indexer:

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- March 2022: First Edition

Revision History for the Early Release

- 2021-08-13: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098107963 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Designing Machine Learning Systems, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-10796-3

[LSI]

Chapter 1. Machine Learning in Production

In November 2016, Google announced that it had incorporated its multilingual neural machine translation system into Google Translate, marking one of the first success stories of deep neural artificial neural networks in production at scale1. According to Google, with this update, Google Translate’s quality of translation improved more in a single leap than they had seen in the previous ten years combined.

Since then, more and more companies have turned towards machine learning (ML) for solutions to their most challenging problems. In just five years, ML has found its way into almost every aspect of our lives, from how we access information, how we communicate, how we work, to how we find love. The spread of ML has been so rapid that it’s already hard to imagine life without it. Yet, there are still many more use cases for ML waiting to be explored: in healthcare, in transportation, in farming, even in helping us understand the universe2.

Many people, when they hear “machine learning”, think of ML algorithms such as logistic regression or different types of neural networks. However, the algorithm is only a small part of an ML system in production. The system also includes the interface where users and developers interact with your system, the data stack to manage your data, the infrastructure to execute the required workloads, and the hardware backend your ML algorithm runs on. Figure 1-1 shows you the different components of an ML system.

Figure 1-1. Different components of an ML system. “ML algorithms” is usually what people think of when they say machine learning, but it’s only a small part of the entire system.

There are many excellent books that can give readers a deep understanding of various ML algorithms. This book doesn’t aim to explain any specific algorithms in detail but to help readers understand the entire ML system as a whole. New algorithms are constantly being developed. This book hopes to provide you with a process to develop a solution that best works for your problem, regardless of which algorithm you might end up using. Chapter [TODO] includes a section that helps you evaluate which algorithm is best for your problem.

Because of the scale of many ML systems—they consume a massive amount of data, require heavy computational power, and have the potential to affect the lives of so many people—deploying them in production has many engineering and societal challenges. However, because of the speed at which these applications are being deployed, these challenges are not always properly understood, let alone addressed. In the best case, the failure to address these challenges can lead to a few unhappy users. In the worst case, it can ruin people’s lives and bankrupt companies.

This chapter aims to give you a high-level view of the challenges and requirements for deploying ML systems in production. However, before talking about how to develop ML systems, it’s important to take a step back and ask a fundamental question: when and when not to use machine learning. We’ll cover some of the popular use cases of ML to illustrate this point.

We will then move onto the challenges of deploying ML systems, and it’ll do so by comparing ML in production to ML in research as well as to traditional software. It continues with an overview of ML systems design as well as the iterative process for designing an ML system that is deployable, reliable, scalable, and adaptable.

If you’ve been in the trenches, you might already be familiar with what’s written in this chapter. However, if you have only had experience with ML in an academic setting, this chapter will give an honest view of what it takes to deploy ML in the real world, and, hopefully, set your first application up for success.

When and When not to Use Machine Learning

As its adoption in the industry quickly grows, ML has proven to be a powerful tool for a wide range of problems. Despite an incredible amount of excitement and hype generated by people both inside and outside the field, machine learning (ML) is not a magic tool that can solve all problems. Even for problems that ML can solve, ML solutions might not be the optimal solutions.

Before starting an ML project, you might want to ask whether ML is necessary3 and whether the cost-benefit equation for ML makes sense.

When To Use Machine Learning

We expect that most readers are familiar with ML. However, to understand what ML can do, let’s take a step back and understand what ML is:

Machine learning is an approach to (1) learn (2) complex (3) patterns from (4) existing data and use these patterns to make (5) predictions on (6) unseen data.

We’ll look at each of the underlined keyphrases in the definition to understand its implications to the problems ML can solve.

1. Learn: the system has the capacity to learn

An Excel sheet is an excellent program, but it’s not an ML system because it doesn’t have the capacity to learn. You can explicitly state the relationship between two columns in Excel, but as of writing, Excel doesn’t have the capacity to figure out the relationship between these two columns by itself.

For an ML system to learn, there must be something for it to learn from. In most cases, ML systems learn from data. In supervised learning, based on examples of what inputs and outputs should look like, ML systems learn how to generate outputs for arbitrary inputs. For example, if you want to build an ML system to learn to predict the rental price for Airbnb listings, you need to provide a dataset where each input is a listing with all its characteristics (square footage, number of rooms, neighborhood, amenities, rating of that listing, etc.) and the associated output is the rental price of that listing. Once learned, this ML system can predict the price of a new listing given its characteristics.

Each learning process is guided by an objective function and an update rule. The objective function estimates how good the system is, and the update rule tells the system how to update its parameters to become better.

For example, in the rental price prediction system, the objective function can be the mean absolute error: the average difference between the real prices and the prices generated by the system. The smaller the average difference, the better the system.

The update rule can be vanilla gradient descent: correcting each parameter based on how much it contributes to the mean absolute error.

2. Complex: the patterns are complex

Consider a website like Airbnb with a lot of house listings, each listing comes with a zip code. If you want to sort listings into the states they are located in, you wouldn’t need an ML system. Since the pattern is simple—each zip code corresponds to a known state—you can just use a lookup table.

The relationship between a rental price and all its characteristics follows a much more complex pattern which would be very challenging to explicitly state by hand. ML is a good solution for this. Instead of telling your system how to calculate the price from a list of characteristics, you can provide prices and characteristics, and let your ML system figure out the pattern.

ML has been very successful with tasks with complex patterns such as object detection and speech recognition. Algorithmic complexity is different from complexity in human perception. Many tasks that are hard for humans to do can be easy to express in algorithms, for example, raising a number of the power of 10. Vice versa, many tasks that are easy for humans can be hard to express in algorithms, e.g. deciding whether there’s a cat in a picture.

3. Patterns: there are patterns to learn

ML solutions are only useful when there are patterns to learn. Sane people don’t invest money into building an ML system to predict the next outcome of a fair die because there’s no pattern in how these outcomes are generated4.

However, there are patterns in how stocks are priced, and therefore companies have invested billions of dollars in building ML systems to learn those patterns.

Whether a pattern exists might not be obvious, or if patterns exist, your dataset might not be sufficient to capture them. For example, there might be a pattern in how Elon Musk’s tweets affect Bitcoin prices. However, you wouldn’t know until you’ve rigorously trained and evaluated your ML models on his tweets. Even if all your models fail to make reasonable predictions of Bitcoin prices, it doesn’t mean there’s no pattern.

4. Existing data: data is available, or it’s possible to collect data

Because ML learns from data, there must be data for it to learn from. It’s amusing to think about building a model to predict how much tax a person should pay a year, but it’s not possible unless you have access to tax and income data of a large population.

Existing data can be public, proprietary, or synthesized. Or, you can follow a ‘fake-it-til-you make it’ approach: launching a product that serves predictions made by humans, instead of ML algorithms, with the hope of using the generated data to train ML algorithms.

5. Predictions: it’s a predictive problem

ML algorithms make predictions, so they can only solve problems that require predictions. ML can be especially appealing when you can benefit from a large quantity of cheap but approximate predictions. “Predict” means “estimate a value in the future. For example, what would the weather be like tomorrow? What would win the Super Bowl this year? What movie would a user want to watch next?

As predictive machines (e.g. ML models) are becoming more effective, more and more problems are being reframed as predictive problems. Whatever question you might have, you can always frame it as: “What would the answer to this question be?”

Computing problems are one class of problems that have been very successfully reframed as predictive. Instead of computing the exact outcome of a process, which might be even more computationally costly and time-consuming than ML, you can frame the problem as: “What would the outcome of this process look like?” and approximate it using an ML algorithm. The output will be an approximation of the exact output, but often, it’s good enough. You can see a lot of it in graphic renderings, such as image denoising5, screen-space shading6.

6. Unseen data: Unseen data shares patterns with the training data

The patterns your model learns from existing data are only useful if unseen data also share these patterns. A model to predict whether an app will get downloaded on Christmas 2020 won’t perform very well if it’s trained on data from 2008 when the most popular app on the App Store was Koi Pond. What’s Koi Pond? Exactly.

\

In technical terms, it means your unseen data and training data should come from similar distributions. You might ask: “If the data is unseen, how do we know what distribution it comes from?” We don’t, but we can make assumptions—such as we can assume that users’ behaviors tomorrow won’t be too different from users’ behaviors today—and hope that our assumptions hold. If they don’t, we’ll find out soon enough.

Due to the way most ML algorithms today learn, ML solutions will especially shine if your problem has these additional following characteristics.

7. It’s repetitive

Humans are great at few-shot learning: you can show kids a few pictures of cats and most of them will recognize a cat the next time they see one. Despite exciting progress in few-shots learning research, most ML algorithms still require many examples to learn a pattern. When a task is repetitive, each pattern is repeated multiple times, which makes it easier for machines to learn it.

8. It’s at scale

ML solutions often require non-trivial upfront investment on data, compute, infrastructure, and talent, so it’d make sense if we can use these solutions a lot.

“At scale” means different things for different tasks, but it might mean making a lot of predictions. Examples include sorting through millions of mails a year or predicting which departments thousands of support tickets should be sent to a day.

A problem might appear to be a singular prediction but it’s actually a series of predictions. For example, a model that predicts who will win a US presidential election seems like it only makes one prediction every four years, but it might actually be making a prediction every hour or even less because that prediction has to be updated to new information over time.

Having a problem at scale also means that there’s a lot of data for you to collect, which is useful for training ML models.

9. The patterns are constantly changing

Cultures change. Tastes change. Technologies change. What’s trendy today might be old news tomorrow. Consider the task of email spam classification. Today, an indication of a spam email is a Nigerian prince but tomorrow it might be a distraught Vietnamese writer.

If your problem involves one or more constantly changing patterns, solutions that don’t allow you to learn from changing data might get you stuck in the past.

When not to Use Machine Learning

The list of use cases can go on and on, and it’ll grow even longer as ML adoption matures in the industry. Even though ML can solve a subset of problems very well, it can’t solve and/or shouldn’t be used for a lot of problems. ML shouldn’t be used under any of the following conditions.

1. It’s unethical.

2. Simpler solutions do the trick. In chapter [TODO], we’ll cover how to start with simple solutions first before trying out ML solutions.

3. One single prediction error can cause devastating consequences.

4. Every single decision the system makes must be explainable.

5. It’s not cost-effective.

However, even if ML can’t solve your problem, it might be possible to break your problem into smaller components and ML can solve some of them. For example, if you can’t build a chatbot to answer all your customers’ queries, it might be possible to build an ML model to predict whether a query matches one of the frequently asked questions. If yes, automatically direct the customer to the answer. If not, direct them to customer service.

I’d also want to caution against dismissing a new technology because it’s not as cost-effective as older technologies at the moment. Most technological advances are incremental. A type of technology might not be efficient now, but it might be in the future. If you wait for the technology to prove its worth to the rest of the industry before jumping in, you might be years or decades behind your competitors.

Machine Learning Use Cases

ML has found increasing usage in both enterprise and consumer applications. Since the mid-2010s, there has been an explosion of applications that leverage ML to deliver superior or previously impossible services to the consumers.

With the explosion of information and services, it’d have been very challenging for us to find what we want without the help of ML, manifested in either a search engine or a recommendation system. When you visit a website like Amazon or Netflix, you’re recommended items that are predicted to best match your taste. If you don’t like any of your recommendations, you might want to search for specific items, and your search results are likely to be powered by ML.

If you have a smartphone, ML is likely already assisting you in many of your daily activities. Typing on your phone is made easier with predictive typing, an ML system that gives you suggestions on what you might want to say next. An ML system might run in your photo editing app to suggest how best to enhance your photos. You might authenticate your phone using your fingerprint or your face, which requires an ML system to predict whether a fingerprint or a face matches yours.

The ML use case that drew me into the field was machine translation, automatically translating from one language to another. It has the potential to allow people from different cultures to communicate with each other, erasing the language barrier. My parents don’t speak English, but thanks to Google Translate, now they can read my writing and talk to my friends who don’t speak Vietnamese.

ML is increasingly present in our homes with smart personal assistants such as Alexa and Google Assistant. Smart security cameras can let you know when your pets leave home or if you have an uninvited guest. A friend of mine was worried about his aging mother living by herself -- if she falls, no one is there to help her get up -- so he relied on an at-home health monitoring system that predicts whether someone has fallen in the house.

Even though the market for consumer ML applications is booming, the majority of ML use cases are still in the enterprise world. Enterprise ML applications tend to have vastly different requirements and considerations from consumer applications. There are many exceptions, but for most cases, enterprise applications might have stricter accuracy requirements but be more forgiving with latency requirements. For example, improving a speech recognition system’s accuracy from 95% to 95.5% might not be noticeable to most consumers, but improving a resource allocation system’s efficiency by just 0.1% can help a corporation like Google save millions of dollars. At the same time, latency of a second might get a consumer distracted and open something else, but enterprise users might be more tolerant of that. For people interested in building companies out of ML applications, consumer apps might be easier to distribute but much harder to make money out of. However, most enterprise use cases aren’t obvious unless you’ve encountered them yourself.

According to Algorithmia’s 2020 state of enterprise machine learning survey, ML applications in enterprises are diverse, serving both internal use cases (reducing costs, generating customer insights and intelligence, internal processing automation) and external use cases (improving customer experience, retaining customers, interacting with customers).7

Figure 1-2. 2020 state of enterprise machine learning by Algorithmia.

Fraud detection is among the oldest applications of ML in the industry. If your product or service involves transactions of any value, it’ll be susceptible to fraud. By leveraging ML solutions for anomaly detection, you can have systems that learn from historical fraud transactions and predict whether a future transaction is fraudulent.

Deciding how much to charge for your product or service is probably one of the hardest business decisions, why not let ML do it for you? Price optimization is the process of estimating a price at a certain time period to maximize a defined objective function, e.g. the company’s margin or revenue. ML-based pricing optimization is most suitable for cases with a large number of transactions where demand fluctuates and consumers are willing to pay a dynamic price e.g. Internet ads, flight tickets, accommodation bookings, ride-sharing, events.

To run a business, it’s important to be able to forecast customer demand so that you can prepare a budget, stock inventory, allocate resources, and update pricing strategy. For example, if you run a grocery store, you want to stock enough so that customers find what they’re looking for, but you don’t want to overstock, because if you do, your groceries might go bad and you lose money.

Acquiring a new user is expensive. As of 2019, the average cost for an app to acquire a user who’ll make an in-app purchase is $86.618. The acquisition cost for Lyft is estimated at $158/rider9. This cost is so much higher for enterprise customers. Customer acquisition cost is hailed by investors as a startup killer10. Reducing customer acquisition costs by a small amount can result in a large increase in profit. This can be done through better identifying potential customers, showing better-targeted ads, giving discounts at the right time, etc.—all of which are suitable tasks for ML.

After you’ve spent so much money acquiring a customer, it’d be a shame if they leave. Churn prediction is predicting when a specific customer is about to stop using your products or services so that you can take appropriate actions to win them back. Churn prediction can be used not only for customers but also for employees.

To prevent customers from leaving, it’s important to keep them happy by addressing their concerns as soon as they arise. Automated support ticket classification can help with that. Previously, when a customer opens a support ticket or sends an email, it needs to first be processed then passed around to different departments until it arrives at the inbox of someone who can address it. An ML system can analyze the ticket content and predict where it should go, which can shorten the response time and improve customer satisfaction. It can also be used to classify internal IT tickets.

Another popular use case of ML in enterprise is brand monitoring. The brand is a valuable asset of a business11. It’s important to monitor how the public and how your customers perceive your brand. You might want to know when/where/how it’s mentioned, both explicitly (e.g. when someone mentions “Google”) or implicitly (e.g. when someone says “the search giant”) as well as the sentiment associated with it. If there’s suddenly a surge of negative sentiment in your brand mentions, you might want to do something about it as soon as possible. Sentiment analysis is a typical ML task.

A set of ML use cases that has generated much excitement recently is in health care. There are ML systems that can detect skin cancer and diagnose diabetes. Healthcare prediction systems are technically geared towards consumers, but because of their strict requirements with accuracy and privacy, they might be provided through a healthcare provider such as a hospital or used to assist doctors in providing diagnosis.

Understanding Machine Learning Systems

Before learning how to design machine learning systems, we’ll go over how ML systems are different from both ML in research (or in school) and traditional software, which motivates the need for this framework.

Machine learning in research vs. in production

As ML usage in the industry is still fairly new, most people with ML expertise have gained it through academia: taking courses, doing research, reading papers. If that describes your background, it might be a steep learning curve for you to understand the challenges of deploying ML systems in the wild and navigate an overwhelming set of solutions to these challenges. ML in production is very different from ML in research. Table 1-1 shows five of the major differences.

Table 1-1. Key differences between ML in research and Ml in production

Research

Production

Objectives

Model performance

Different stakeholders have different objectives

Computational priority

Fast training, high throughput

Fast inference, low latency

Data

Static

Constantly shifting

Fairness

Good to have (sadly)

Important

Interpretability

Good to have

Important

Stakeholders and their objectives

Research and leaderboard projects often have one single objective. The most common objective is model performance—develop a model that achieves the state-of-the-art (SOTA) results on benchmark datasets. To edge out a small improvement in performance, researchers often resort to techniques that make models too complex to be useful.

There are many stakeholders involved in bringing an ML system into production. Each stakeholder has their own objective. Consider a project that recommends restaurants to users. The project involves ML engineers, salespeople, product managers, infrastructure engineers, and a manager.

- The ML engineers want a model that recommends restaurants that users will most likely order from, and they believe they can do so by using a more complex model with more data.

- The sales team wants a model that recommends restaurants that pay the highest advertising fee to be shown in-app, since ads bring in more revenue than just service fees.

- The product team notices that every drop in latency leads to drop in orders through the service, so they want a model that can do inference faster than the model that the ML engineers are working on.

- As the traffic grows, the infrastructure team has been woken up in the middle of the night because of problems with scaling their existing system, so they want to hold off the production line so they could update the infrastructure.

- The manager wants to maximize the margin, and one way to achieve it is to let go of the ML team12.

These objectives require different models, yet the stakeholders will have to collaborate to somehow create a model that will satisfy all of them.

Production having different objectives from research is one of the reasons why successful research projects might not always be used in production. Ensembling is a technique popular among the winners of many ML competitions, including the famed $1M Netflix Prize. It combines “multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.13” While it can give you a small improvement, ensembled systems risk being too complex to be useful, e.g. more error-prone to deploy, slower to serve, or harder to interpret.

For many tasks, a small improvement in performance can result in a huge boost in revenue or cost save. For example, a 0.2% improvement in the click-through-rate for a product recommendation system can result in millions of dollars increase in revenue for an ecommerce site. However, for many tasks, a small improvement might not be noticeable for users. From a user’s point of view, a speech recognition app with a 95% accuracy is not that different from an app with a 95.2% accuracy. For the second type of tasks, if a simple model can do a reasonable job, complex models must perform significantly better to justify the complexity.

In recent years, there have been many critics of ML leaderboards, both research leaderboards such as GLUE and competitions such as Kaggle.

An obvious argument is that in these competitions, many hard steps needed for building ML systems are already done for you14.

A less obvious argument is that due to the multiple-hypothesis testing scenario that happens when you have multiple teams testing on the same hold-out test set, a model can do better than the rest just by chance15.

The misalignment of interests between research and production has been noticed by researchers. In an EMNLP 2020 paper, Ethayarajh and Jurafsky argued that benchmarks have helped drive advances in NLP by incentivizing the creation of more accurate models at the expense of other qualities valued by practitioners such as compactness, fairness, and energy efficiency16.

Computational priority

When designing an ML system, people who haven’t deployed an ML system often make the mistake of focusing entirely on the model development part.

During the model development process, you train different iterations of your model multiple times. The trained model then runs inference on the test set once to report the score. This means training is the bottleneck. Once the model has been deployed, however, its job is to do inference, so inference is the bottleneck. Most research prioritizes fast training whereas most production prioritizes fast inference.

Latency vs. throughput

One corollary of this is that research prioritizes high throughput whereas production prioritizes low latency. In case you need a refresh, latency refers to the time it takes from receiving a query to returning the result. Throughput refers to how many queries are processed within a specific period of time.

For example, the average latency of Google Translate is the average time it takes from when a user clicks Translate to when the translation is shown, and the throughput is how many queries it processes and serves a second.

If your system always processes one query at a time, higher latency means lower throughput. If the average latency is 10ms, which means it takes 10ms to process a query, the throughput is 100 queries/second. If the average latency is 100ms, the throughput is 10 queries/second.

However, if your system batches query to process them together, higher latency might mean higher throughput. If you process 10 queries at a time and it takes 10ms to run a batch, the average latency is still 10ms but the throughput is now 10 times higher—1000 queries/second. If you process 100 queries at a time and it takes 50ms to run a batch, the average latency now is 50ms and the throughput is 2000 queries/second. Both latency and throughput have increased!

This is further complicated if you want to batch online queries. Batching requires your system to wait for enough queries to arrive in a batch before processing them, which further increases latency.

In research, you care more about how many samples you can process in a second (throughput) and less about how long it takes for each sample to be processed (latency). You’re willing to increase latency to increase throughput, e.g. with aggressive batching.

However, once you deploy your model into the real world, latency matters a lot. In 2009, Google’s experiments demonstrated that increasing web search latency 100 to 400 ms reduces the daily number of searches per user by 0.2% to 0.6%17. In 2019, Booking.com found that an increase of about 30% in latency cost about 0.5% in conversion rates—“a relevant cost for our business.”18

Reducing latency might reduce the number of queries you can process on the same hardware at a time. If your hardware is capable of processing much more than one sample at a time, using it to process only one sample means making processing one sample more expensive.

Data

During the research phase, the datasets you work with are often clean and well-formatted, freeing you to focus on developing and training models. They are static by nature so that the community can use them to benchmark new architectures and techniques. This means that many people might have used and discussed the same datasets, and quirks of the dataset are known. You might even find open-source scripts to process and feed the data directly into your models.

In production, data, if available, is a lot more messy. It’s noisy, possibly unstructured, constantly shifting. It’s likely biased, and you likely don’t know how it’s biased. Annotated labels, if there are any, are sparse, imbalanced, outdated, or incorrect. Changing project or business requirements might require adding another label class or merging two existing label classes. This can happen even after a model has been trained and deployed. If you work with users’ data, you’ll also have to worry about privacy and regulatory concerns.

In research, since you don’t serve your models to users, you mostly work with historical data, e.g. data that already exists and is stored somewhere. In production, most likely you’ll also have to work with data that is being constantly generated by users, systems, and third-party data.

Figure 1-3 is a great graphic by Andrej Karpathy that illustrates the data problems he encountered during his PhD compared to his time at Tesla.

Research

Production

- Clean

- Static

- Mostly historical data

- Messy

- Constantly shifting

- Historical + streaming data

- Privacy + regulatory concerns

Figure 1-3. Data in research vs. data in production by Andrej Karpathy19

Fairness

During the research phase, a model is not yet used on people, so it’s easy for researchers to put off fairness as an afterthought: “Let’s try to get state-of-the-art first and worry about fairness when we get to production.” When it gets to production, it’s too late.

You or someone in your life might already be a victim of biased mathematical algorithms swithout knowing it. Your loan application might be rejected because the ML algorithm picks on your zip code, which embodies biases about one’s socio-economic background. Your resume might be ranked lower because the ranking system employers use picks on the spelling of your name. Your mortgage might get a higher interest rate because it relies partially on credit scores, which reward the rich and punish the poor. Other examples of ML biases in the real world are in predictive policing algorithms, personality tests administered by potential employers, college ranking. For even more galling examples, I recommend Cathy O’Neil’s Weapon of Math Destruction20.

ML algorithms don’t predict the future, but encode the past, perpetuating the biases in the data and more. When ML algorithms are deployed at scale, they can discriminate against people at scale. If a human operator might only make sweeping judgments about a few individuals at a time, an ML algorithm can make sweeping judgments about millions in split seconds. This can especially hurt members of minority groups because misclassification on them has minor effects on models’ overall performance metrics.

If an algorithm can already make correct predictions on 98% of the population, and improving the predictions on the other 2% would incur multiples of cost, some companies might, unfortunately, choose not to do it. During a McKinsey & Company research in 2019, only 13% of the large companies surveyed said they are taking steps to mitigate risks to equity and fairness, such as algorithmic bias and discrimination21.

Interpretability

In early 2020, the Turing Award winner Dr. Geoffrey Hinton proposed a heatedly debated question about the importance of interpretability in ML systems.

“Suppose you have cancer and you have to choose between a black box AI surgeon that cannot explain how it works but has a 90% cure rate and a human surgeon with an 80% cure rate. Do you want the AI surgeon to be illegal?”22

A couple of weeks later, when I asked this question to a group of 30 technology executives at public non-tech companies, only half of them would want the highly effective but unable to explain AI surgeon to operate on them. The other half wanted the human surgeon.

While most of us are comfortable with using a microwave without understanding how it works, many don’t feel the same way about AI yet, especially if that AI makes important decisions about their lives.

Since most ML research is still evaluated on a single objective, model performance, researchers aren’t incentivized to work on model interpretability. However, interpretability isn’t just optional for most ML use cases in the industry, but a requirement.

First, interpretability is important for users, both business leaders and end-users, to understand why a decision is made so that they can trust a model and detect potential biases mentioned above. Second, it’s important for developers to debug and improve a model.

Just because interpretability is a requirement doesn’t mean everyone is doing it. As of 2019, only 19% of large companies are working to improve the explainability of their algorithms23.

Discussion

Some might argue that it’s okay to know only the academic side of ML because there are plenty of jobs in research. The first part—it’s okay to know only the academic side of ML—is true. The second part is false.

While it’s important to pursue pure research, most companies can’t afford it unless it leads to short-term business applications. This is especially true now that the research community took the “bigger, better” approach, with new models requiring a massive amount of data and tens of millions of dollars in compute alone.

As ML research and off-the-shelf models become more accessible, more people and organizations would want to find applications for them, which increases the demand for ML in production.

The majority of ML-related jobs will be, and already are, in productionizing ML.

Machine learning systems vs. traditional software

Since ML is part of software engineering (SWE), and software has been successfully used in production for more than half a century, some might wonder why we don’t just take tried-and-true best practices in software engineering and apply them to ML.

That’s an excellent idea. In fact, ML production would be a much better place if ML experts were better software engineers. Many traditional SWE tools can be used to develop and deploy ML applications.

However, many challenges are unique to ML applications and require their own tools. In SWE, there’s an underlying assumption that code and data are separated. In fact, in SWE, we want to keep things as modular and separate as possible (see Separation of concerns).

On the contrary, ML systems are part code, part data. The trend in the last decade shows that applications developed with the most/best data win. Instead of focusing on improving ML algorithms, most companies will focus on improving their data. Because data can change quickly, ML applications need to be adaptive to the changing environment which might require faster development and deployment cycles.

In traditional SWE, you only need to focus on testing and versioning your code. With ML, we have to test and version our data too, and that’s the hard part. How to version large datasets? How to know if a data sample is good or bad for your system? Not all data samples are equal -- some are more valuable to your model than others. For example, if your model has already trained on 1M scans of normal lungs and only 1000 scans of cancerous lungs, a scan of a cancerous lung is much more valuable than a scan of a normal lung. Indiscriminately accepting all available data might hurt your model’s performance and even make it susceptible to data poisoning attacks (see Figure 1-4).

Figure 1-4. An example of how a face recognition system can be poisoned, using malicious data, to allow unauthorized people to pose as someone else. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (Chen et al., 2017)

The size of ML models gives another challenge. As of 2020, it’s common for ML models to have hundreds of millions, if not billions, of parameters, which requires GBs of RAM to load them into memory. A few years from now, a billion parameters might seem quaint—like can you believe the computer that sent men to the moon only had 32MB of RAM?

However, for now, getting these large models into production, especially on edge devices, is a massive engineering challenge. Then there is the question of how to get these models to run fast enough to be useful. An autocompletion model is useless if the time it takes to suggest the next character is longer than the time it takes for you to type.

Monitoring and debugging these models in production is also non-trivial. As ML models get more complex, coupled with the lack of visibility into their work, it’s hard to figure out what went wrong or be alerted quickly enough when things go wrong.

The good news is that these engineering challenges are being tackled at a breakneck pace. Back in 2018, when BERT first came out, people were talking about how BERT was too big, too complex, and too slow to be practical. The pretrained large BERT model has 340M parameters and is 1.35GB24. Fast forward two years later, BERT was already used in almost every English search on Google25.

Designing ML Systems in Production

Now that we’ve discussed what it takes to develop and deploy an ML system, let’s get to the fun part of actually designing one. This section aims to give you an overview of machine learning systems design. It starts by explaining what machine learning systems design is and covers the requirements for ML systems. We will then go over the iterative process for designing systems to meet those requirements.

ML systems design is the process of defining all the components of an ML system, including interface, algorithms, data, infrastructure, and hardware, so that the system satisfies specified requirements.

Requirements for ML Systems

Before building a system, it’s essential to defy requirements for that system. Requirements vary from use case to use case. However, most systems should have these four characteristics: reliable, scalable, maintainable, and adaptable.

We’ll walk through each of these concepts in detail. Let’s take a closer look at reliability first.

Reliability

The system should continue to perform the correct function at the desired level of performance even in the face of adversity (hardware or software faults, and even human error).

“Correctness” might be difficult to determine for ML systems. For example, your system might call the function “.predict()” correctly, but the predictions are wrong. How do we know if a prediction is wrong if we don’t have ground truth labels to compare it with?

With traditional software systems, you often get a warning, such as a system crash or runtime error or 404. However, ML systems fail silently. End users don’t even know that the system has failed and might have kept on using it as if it was working. Figure 1-5 shows an example of a silent failure likely caused by an ML system.

Figure 1-5. The awkward message might likely be a bad translation produced by an ML model.

Scalability

As the system grows (in data volume, traffic volume, or complexity), there should be reasonable ways of dealing with that growth.

Scaling isn’t just scaling up -- expanding the resources to handle growth. In ML, it’s also important to scale down -- reducing the resources when not needed. For example, at peak, your system might require 100 GPUs. However, most of the time, your system needs only 10 GPUs. Keeping 100 GPUs up all the time can be costly, so your system should be able to scale down to 10 GPUs.

An indispensable feature in many cloud services is autoscaling: automatically scaling up and down the number of machines depending on usage. This feature can be tricky to implement. Even Amazon fell victim to this when their autoscaling feature failed on Prime Day, causing their system to crash. An hour downtime was estimated to cost it between $72 million and $99 million26.

Maintainability

There are many people who will work on an ML system. They are ML engineers, DevOps engineers, and subject matter experts (SMEs). They might come from very different backgrounds, with very different languages and tools, and might own different parts of the process. It’s important to structure your project and set up your infrastructure in a way such that different contributors can work using tools that they are comfortable with, instead of one group of contributors forcing their tools onto other groups. When a problem occurs, different contributors should be able to work together to identify the problem and implement a solution without finger-pointing. We’ll go more into this in chapter [TODO].

Adaptability

To adapt to changing data distributions and business requirements, the system should have some capacity for both discovering aspects for performance improvement and allowing updates without service interruption.

Because ML systems are part code, part data, and data can change quickly, ML systems need to be able to evolve quickly. This is tightly linked to maintainability. We’ll go more into this in chapter [TODO].

Iterative Process

Developing an ML system is an iterative and, in most cases, never ending process. You do reach the point where you have to put the system into production, but then that system will constantly need to be monitored and updated.

Before deploying their first ML system, many engineers thought their process would be linear and straightforward. They thought all they had to do was to collect data, train a model, deploy that model, and get an employee of the month award. However, in reality, the process looks more like a cycle with a lot of back and forth between different steps.

For example, here is one workflow that you might encounter when building an ML model to predict whether an ad should be shown when users enter a search query27.

1. Choose a metric to optimize. For example, you might want to optimize for impressions -- the number of times an ad is shown.

2. Collect data and obtain labels.

3. Engineer features.

4. Train models.

5. During error analysis, you realize that errors are caused by wrong labels, so you relabel data.

6. Train model again.

7. During error analysis, you realize that your model always predicts that an ad shouldn’t be shown, and the reason is because 99.99% of the data you have is no-show (an ad shouldn’t be shown for most queries). So you have to collect more data of ads that should be shown.

8. Train model again.

9. Model performs well on your existing test data, which is by now two months ago. But it performs poorly on the test data from yesterday. Your model has degraded, so you need to collect more recent data.

10. Train model again.

11. Deploy model.

12. Model seems to be performing well but then the business people come knocking on your door asking why the revenue is decreasing. It turns out the ads are being shown but few people click on them. So you want to change your model to optimize for clickthrough rate instead.

13. Start over.

Here is an oversimplified representation of what the iterative process for developing ML systems in production looks like. We’ll cover the first step, Project Scoping, in this chapter. Each of the following steps will be covered in its own chapter, from [TODO] to chapter [TODO].

Figure 1-6. The process of developing an ML system looks more like a cycle with a lot of back and forth between steps.

While we’ll take a deeper dive into what each of these steps mean in practice in later chapters, let’s take a brief look at what happens during each of the steps.

Step 1. Project scoping

A project starts with scoping the project, laying out goals & objectives, constraints, and evaluation criteria. Stakeholders should be identified and involved. Resources should be estimated and allocated.

Step 2. Data engineering

Data used and generated by ML systems can be large and diverse, which requires scalable infrastructure to process and access it fast and reliably. Data engineering covers data sources, data formats, data processing, and data manipulation to create training data.

Step 3. ML model development

From raw data, you need to create training datasets and possibly label them, then generate features, train models, optimize models, and evaluate them. This is the stage that requires the most ML knowledge and is most often covered in ML courses.

Step 4. Deployment

After a model is developed, it needs to be made accessible to users. Developing an ML system is like writing—you will never reach the point when your system is done. But you do reach the point when you have to put your system out there.

Step 5. Monitoring and continual learning

Once in production, models need to be monitored for performance decay and maintained to be adaptive to changing environments and changing requirements.

Step 6. Business analysis

Model performance needs to be evaluated against business goals and analyzed to generate business insights. These insights can then be used to eliminate unproductive projects or scope out new projects.

Summary

This chapter set out to achieve an ambitious goal: to give readers an understanding of what it takes to bring an ML system into production, how they differ from ML projects in a research setting, as well as how they differ from traditional software engineering systems.

It’s ambitious because, as we’ve covered in this chapter, ML systems are complex, consisting of many different components and involving many different stakeholders. They can be deployed to solve a wide range of tasks, both for consumers and enterprises. Each task also comes with its own challenges and requirements. The effort is further complicated by the fact that as ML adoption matured, tools and best practices for ML systems will also evolve.

It’s impossible to cover every aspect of ML systems in production, but I hope that this chapter has covered what I believe to be most applicable to ML systems in a wide range of tasks. I hope that this chapter can help mitigate surprises and help you to become better prepared when evaluating and planning the use of ML in your projects. If you believe that there’s something I’ve missed, please let me know.

Fortunately, complex ML systems are made up of simpler building blocks. Now that we’ve covered the high-level overview of an ML system in production, we’ll zoom into its building blocks in the following chapters, starting with the fundamentals of data engineering in the next chapter. If any of the challenges mentioned in this chapter seems abstract to you, I hope that specific examples in the following chapters will make them more concrete.

1   Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System (Schuster et al., Google AI Blog 2016)

2   A method to image black holes (MIT News 2019)

3   I didn’t ask whether ML is sufficient because the answer is always no.

4   Patterns are different from distributions. We know the distribution of the outcomes of a fair die, but there are no patterns in the way the outcomes are generated.

5   Kernel-predicting convolutional networks for denoising Monte Carlo renderings (Bako et al., ACM Transactions on Graphics 2017)

6   Deep Shading: Convolutional Neural Networks for Screen-Space Shading (Nalbach et al., 2016)

7   2020 state of enterprise machine learning (Algorithmia, 2020)

8   Average mobile app user acquisition costs worldwide from September 2018 to August 2019, by user action and operating system (Statista, 2019)

9   Valuing Lyft Requires A Deep Look Into Unit Economics (Forbes, 2019)

10   Startup Killer: the Cost of Customer Acquisition (David Skok, 2018)

11   Apple, Google, Microsoft, Amazon each has a brand estimated to be worth in the order of hundreds of millions dollars (Forbes, 2020)

12   It’s common for the ML and data science teams to be among the first to go during a company’s mass layoff. See IBM, Uber, Airbnb, and this analysis on How Data Scientists Are Also Susceptible To The Layoffs Amid Crisis (AIM, 2020).

13   Ensemble learning (Wikipedia)

14   Machine learning isn’t Kaggle competitions (Julia Evans, 2014)

15   AI competitions don’t produce useful models (Luke Oakden-Rayner, 2019)

16   Utility is in the Eye of the User: A Critique of NLP Leaderboards (Ethayarajh and Jurafsky, EMNLP 2020)

17   Speed Matters for Google Web Search (Jake Brutlag, Google 2009)

18   150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com (Bernardi et al., KDD 2019)

19   Building the Software 2.0 Stack (Andrei Karpathy, Spark+AI Summit 2018)

20   Weapon of Math Destruction (Cathy O’Neil, Crown Books 2016)

21   AI Index 2019 (Stanford HAI, 2019)

22   https://twitter.com/geoffreyhinton/status/1230592238490615816

23   AI Index 2019 (Stanford HAI, 2019)

24   BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)

25   Google SearchOn 2020.

26   Wolfe, Sean. 2018. “Amazon’s one hour of downtime on Prime Day may have cost it up to $100 million in lost sales.” Business Insider. https://www.businessinsider.com/amazon-prime-day-website-issues-cost-it-millions-in-lost-sales-2018-7.

27   Praying and crying not featured but present through the entire process.

Chapter 2. Data Engineering: Fundamentals

The rise of machine learning in recent decades is tightly coupled with the rise of big data. Big data systems, even without machine learning, are complex. If you haven’t spent years and years working with them, it’s easy to get lost in acronyms. There are many challenges and possible solutions that these systems generate. Industry standards, if there are any, evolve quickly as new tools come out and the needs of the industry expand, creating a dynamic and ever-changing environment. If you look into the data stack for different tech companies, it might seem like each is doing its own thing.

In this chapter, we’ll cover the basics of data engineering that will, hopefully, give you a steady piece of land to stand on as you explore the landscape for your own needs. It will start with the question: how important data is for building intelligent systems? It will then cover the basics of data engineering. Knowing how to collect, handle, and process an increasingly growing amount of data is essential to people who want to build ML systems in production. If you’re already familiar with data engineering fundamentals, you might want to move directly to Chapter 3 to learn more about how to sample and generate labels to create training data.

Mind vs. Data

Progress in the last decade shows that the success of an ML system depends largely on the data it was trained on. Instead of focusing on improving ML algorithms, most companies focus on managing and improving their data1.

Despite the success of models using massive amounts of data, many are skeptical of the emphasis on data as the way forward. In the last three years, at every academic conference I attended, there were always some debates among famous academics on the power of mind vs. data. Mind might be disguised as inductive biases or intelligent architectural designs. Data might be grouped together with computation since more computation is usually required because more data is involved, and more data tends to lead to more computation.

In theory, you can both pursue intelligent design and leverage large data and computation, but spending time on one often takes time away from another2.

On the mind over data camp, there’s Dr. Judea Pearl, a Turing Award winner best known for his work on causal inference and Bayesian networks. The introduction to his book, “The book of why”, is entitled “Mind over data,” in which he emphasizes: “Data is profoundly dumb.” In one of his more controversial posts on Twitter, he expressed his strong opinion against ML approaches that rely heavily on data and warned that data-centric ML people might be out of job in 3-5 years.

“ML will not be the same in 3-5 years, and ML folks who continue to follow the current data-centric paradigm will find themselves outdated, if not jobless. Take note.”3

There’s also a milder opinion from Dr. Christopher Manning, Director of the Stanford Artificial Intelligence Laboratory, who argued that huge computation and a massive amount of data with a simple learning device create incredibly bad learners. The structure allows us to design systems that can learn more from fewer data4.

Many people in ML today are on the data over mind camp. Dr. Richard Sutton, a distinguished research scientist at DeepMind and a professor of computing science at the University of Alberta, wrote a great blog post in which he claimed that researchers who chose to pursue intelligent designs over methods that leverage computation will eventually learn a bitter lesson.

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. … Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.”5

When asked how Google search was doing so well, Peter Norvig, Google’s Director of Search, emphasized the importance of having a large amount of data over intelligent algorithms in their success: “We don’t have better algorithms. We just have more data.”6

Dr. Monica Rogati, Former VP of Data at Jawbone, argued that data lies at the foundation of data science. If you want to use data science, a discipline of which machine learning is a part of, to improve your products or processes, you need to start with building out your data, both in terms of quality and quantity. Without data, there’s no data science.

Figure 2-1. The data science hierarchy of needs (Monica Rogati, 20177)

The debate isn’t about whether finite data is necessary, but whether it’s sufficient. The term finite here is important, because if we had infinite data, we can just look up the answer. Having a lot of data is different from having infinite data.

Regardless of which camp will prove to be right eventually, no one can deny that data is essential, for now. Both the research and industry trends in the recent decades show the success of machine learning relies more and more on the quality and quantity of data. Models are getting bigger and using more data. Back in 2013, people were getting excited when the One Billion Words Benchmark for Language Modeling was released, which contains 0.8 billion tokens8. Six years later, OpenAI’s GPT-2 used a dataset of 10 billion tokens. And another year later, GPT-3 used 500 billion tokens.

Table 2-1. The size of the datasets used for language models over time

Dataset

Year

Tokens (M)

Penn Treebank

1993

1

Text8

2011

17

One Billion

2013

800

BookCorpus

2015

985

GPT-2 (OpenAI)

2019

10,000

GPT-3 (OpenAI)

2020

500,000

Figure 2-2. The size of the datasets used for language models over time (log scale)

Even though much of the progress in deep learning in the last decade was fueled by an increasingly large amount of data, more data doesn’t always lead to better performance for your model. More data at lower quality might even hurt your model’s performance.

Data Sources

An ML system works with data from many different sources. They have different characteristics with different access patterns, can be used for different purposes, and require different processing methods. Understanding the sources your data comes from can help you use your data more efficiently. This section aims to give a quick overview of different data sources to those unfamiliar with data in production. If you’ve already worked with ML in production for a while, feel free to skip this section.

One source is user input data, data explicitly input by users, which is often the input on which ML models can make predictions. User input can be texts, images, videos, uploaded files, etc. If there is a wrong way for humans to input data, humans are going to do it, and as a result, user input data can be easily mal-formatted. If user input is supposed to be texts, they might be too long or too short. If it’s supposed to be numerical values, users might accidentally enter texts. If you expect users to upload files, they might upload files in the wrong formats. User input data requires more heavy-duty checking and processing. Users also have little patience. In most cases, when we input data, we expect to get results back immediately. Therefore, user input data tends to require fast processing.

Another source is system-generated data. This is the data generated by different components of your systems, which include various types of logs and system outputs such as model predictions.

Logs can record the state of the system and significant events in the system, such as memory usage, number of instances, services called, packages used, etc. It can record the results of different jobs, including large batch jobs for data processing and model training. These types of logs provide visibility into how the system is doing, and the main purpose of this visibility is for debugging and possibly improving the application. Most of the time, you don’t have to look at this type of log, but they are essential when something is on fire.

Because logs are system generated, they are much less likely to be mal-formatted the way users input data is. Most logs don’t require to be processed as fast as user input data. However, you might still want to process your logs fast to be notified whenever something interesting happens9.

Because debugging ML systems is hard, it’s a common practice to log everything you can. This means that your volume of logs can grow very, very quickly. This leads to two problems. The first is that it can be hard to know where to look because signals are lost in the noise. There have been many services that process and analyze logs, such as Logstash, DataDog, Logz, etc. Logs are used to monitor ML systems, but ML is also used to process and analyze logs. In some use cases, logs are used as data to train ML systems to optimize systems’ resource usage.

The second problem is how to store a rapidly growing amount of logs. Luckily, in most cases, you only have to store logs for as long as they are useful, and can discard them when they are no longer relevant for you to debug your current system. If you don’t have to access your logs frequently, they can also be stored in low-access storage that costs much less than higher-frequency-access storage.

System also generates data to record users’ behaviors, such as clicking, choosing a suggestion, scrolling, zooming, ignoring a popup, or spending an unusual amount of time on certain pages. Even though this is system-generated data, it’s still considered part of user data10 and might be subject to privacy regulations. This kind of data can also be used for ML systems to make predictions and to train their future versions.

There are also internal databases, generated by various services and enterprise applications in a company. These databases manage their assets such as inventory, customer relationship, users, and more. This kind of data can be used by ML models directly or by various components of an ML system. For example, when users enter a search query on Airbnb, one or more ML models will return a list of properties that match that query, then Airbnb will need to check with their internal databases for ratings of these properties to rank the results.

Then there’s the wonderfully weird word of third-party data that, to many, is riddled with privacy concerns. First-party data is the data that your company already collects about your users or customers. Second-party data is the data collected by another company on their own customers. Third-party data companies collect data on the public who aren’t their customers.

The rise of the Internet and smartphones has made it much easier for all types of data to be collected. It’s especially easy with smartphones since each phone has a Mobile Advertiser ID, which acts as a unique ID to aggregate all activities on a phone. Data from apps, websites, check-in services, etc. are collected and (hopefully) anonymized to generate activity history for each person.

You can buy all types of data such as social media activities, purchase history, web browsing habits, car rentals, political leaning for different demographic groups getting as granular as men, age 25-34, working in tech, living in the Bay Area. From this data, you can infer information such as people who like brand A also like brand B. This data can be especially helpful for systems such as recommendation systems to generate results relevant to users’ interests. Third-party data is usually sold as structured data after being cleaned and processed by vendors.

Data Formats

Once you have data, you might want to store it (or “persist” it, in technical terms). Since your data comes from multiple sources with different access patterns, storing your data isn’t always straightforward and can be costly. Some of the questions you might want to consider are: How do I store multimodal data? When each sample might contain both images and texts? Where to store your data so that it’s cheap and still fast to access? How to store complex models so that they can be loaded and run correctly on different hardware?

The process of converting a data structure or object state into a format that can be stored or transmitted and reconstructed later is data serialization. There are many, many data serialization formats. Table 2-2 consists of just a few of the common formats that you might work with. For a more comprehensive list, check out the wonderful Wikipedia page Comparison of data-serialization formats.

Table 2-2. Common data formats and where they are used

Format

Binary/Text

Human-readable?

Example use cases

JSON

Text

Yes

Everywhere

CSV

Text

Yes

Everywhere

Parquet

Binary

No

Hadoop, Amazon Redshift

Avro

Binary primary

No

Hadoop

Protobuf

Binary primary

No

Google, TensorFlow (TFRecord)

Pickle

Text, binary

No

Python, PyTorch serialization

We’ll go over a few of these formats, starting with JSON.

JSON

JSON, JavaScript Object Notation, is everywhere. Even though it was derived from JavaScript, it’s language-independent—most modern programming languages can generate and parse JSON. It’s human-readable. Its key-value pair paradigm is simple but powerful, capable of handling data of different levels of structuredness. For example, your data can be stored in a structured format like the following.

{

"firstName": "Boatie",

"lastName": "McBoatFace",

"isVibing": true,

"age": 12,

"address": {

"streetAddress": "12 Ocean Drive",

"city": "Port Royal",

"postalCode": "10021-3100"

}

}

The same data can also be stored in an unstructured blob of text like the following.

{

"text": "Boatie McBoatFace, aged 12, is vibing, at 12 Ocean Drive,

Port Royal, 10021-3100"

}

Row-major vs. Column-major Format

The two formats that are common and represent two distinct paradigms are CSV and Parquet. CSV is row-major, which means consecutive elements in a row are stored next to each other in memory. Parquet is column-major, which means consecutive elements in a column are stored next to each other.

Because modern computers process sequential data more efficiently than non-sequential data, if a table is row-major, accessing its rows will be faster than accessing its columns in expectation. This means that for row-major formats, accessing data by rows is expected to be faster than accessing data by columns.

Imagine we have a dataset of 1000 examples, each example has 10 features. If we consider each example as a row and each feature as a column, then the row-major formats like CSV are better for accessing examples, e.g. accessing all the examples collected last week. Column-major formats like Parquet are better for accessing features, e.g. accessing the timestamps of all your examples.

Figure 2-3. Row-major vs. column-major formats

Column-major formats allow flexible data reads, especially if your data is large with thousands, if not millions, of features. Consider if you have data about ride-sharing transactions that has 1000 features but you only want 4 features: time, location, distance, price. With column-major formats, you can read the 4 columns corresponding to these 4 features directly. However, with row-major formats, if you don’t know the sizes of the rows, you will have to read in all columns then filtering down to these 4 columns. Even if you know the sizes of the rows, it can still be slow as you’ll have to jump around the memory, unable to take advantage of caching.

Row-major formats allow faster data writes. Consider the situation when you have to keep adding new individual examples to your data. For each individual example, it’d be much faster to write it to a file that your data is already in a row-major format.

Overall, row-major formats are better when you have to do a lot of writes, whereas column-major ones are better when you have to do a lot of reads.

NUMPY VS. PANDAS

One subtle point that a lot of people don’t pay attention to, which leads to misuses of Pandas, is that this library is built around the columnar format.

Pandas is built around DataFrame, a concept inspired by R’s Data Frame, which is column-major. A DataFrame is a two-dimensional table with rows and columns.

In NumPy, the major order can be specified. When an ndarray is created, it’s row-major by default if you don’t specify the order. People coming to pandas from NumPy tend to treat DataFrame the way they would ndarray, e.g. trying to access data by rows, and find DataFrame slow.

For example, in the example below, you can see that a row of a DataFrame is so much slower than accessing a column.

However, if you convert your DataFrame to NumPy ndarray, accessing a row becomes much faster.11

Text vs. Binary Format

CSV and JSON are text files whereas Parquet files are binary files. Text files are files that are in plain texts, which usually mean they are human-readable. Binary files, as the name suggests, are files that contain 0’s and 1’s, and meant to be read or used by programs that know how to interpret the raw bytes. A program has to know exactly how the data inside the binary file is laid out to make use of the file. If you open text files in your text editors (e.g. VSCode, Notepad), you’ll be able to read the texts in them. If you open a binary file in your text editors, you’ll see blocks of numbers, likely in hexadecimal values, for corresponding bytes of the file.

Binary files are more compact. Here’s a simple example to show how binary files can save space compared to text files. Consider you want to store the number 1000000. If you store it in a text file, it’ll require 7 characters, and if each character is 1 byte, it’ll require 7 bytes. If you store it in a binary file as int32, it’ll take only 32 bits or 4 bytes.

As an illustration, I use interviews.csv, which is a CSV file (text format) of 17,654 rows and 10 columns. When I converted it to a binary format (Parquet), the file size went from 14MB to 6MB.

AWS recommends using the Parquet format because “the Parquet format is up to 2x faster to unload and consumes up to 6x less storage in Amazon S3, compared to text formats.”12

Data Processing

In this section, we will cover the basics of data processing, starting with two major types of processing: transaction processing and analytical processing, and their uses. We will then cover the basics of the ETL process that you will inevitably encounter when building an ML system in production. When dealing with a large amount of data, a question that often comes up is whether you want to store your data as structured or unstructured. In the last part of this section, we will discuss the pros and cons of both formats.

OLTP vs. OLAP

Systems in production generate data. To process online data, there are two types of queries: OnLine Transaction Processing (OLTP) and OnLine Analytical Processing (OLAP). Even though their acronyms look similar, they have distinct characteristics and have distinct underlying architectures. This section gives a shallow overview of OLTP and OLAP. If you’re already familiar with this, feel free to skip it.

Imagine you’re running a consumer application that generates many short transactions within a short amount of time, such as food ordering, online shopping, ridesharing, money transferring. You want to process and store these transactions as they are generated. They need to be processed fast, in the order of milliseconds. The processing method needs to have extremely high availability, because, without a way to record transactions, you won’t be able to serve your users. On top of that, the processing needs to satisfy the ACID (Atomicity, Consistency, Isolation, Durability) requirements:

- Atomicity: to guarantee that all the steps in a transaction are completed successfully as a group. If any steps between the transaction fail, all other steps must fail also. For example, if a user’s payment fails, you don’t want to still assign a driver to that user.

- Consistency: to guarantee that all the transactions coming through must follow predefined rules. For example, a transaction must be made by a valid user.

- Isolation: to guarantee that two transactions happen at the same time as if they were isolated. Two users accessing the same data won’t change it at the same time. For example, you don’t want two users to book the same driver at the same time.

- Durability: to guarantee that once a transaction has been committed, it will remain committed even in the case of a system failure. For example, after you’ve ordered a ride and your phone dies, you still want your ride to come.

OLTP databases are designed to process online transactions and satisfy all those requirements. Most of the operations they do will be inserting, deleting, and updating an existing transaction. This means that most OLTP databases are more row-oriented.

Because OLTP databases are more row-oriented, they aren’t good for questions such as “What’s the average price for all the rides in September in San Francisco?”. This kind of analytical question requires aggregating data in columns across multiple rows of data. OLAP databases are designed for this purpose. They are efficient with queries that allow you to look at data from different viewpoints.

ETL: Extract, Transform, Load

OLTP databases can be processed and aggregated to generate OLAP databases through a process called ETL (extract, transform, load).

Extract is extracting the data you want from data source(s). Your data will likely come from multiple sources in different formats. Some of them will be corrupted or malformatted. In the extracting phase, you need to validate your data and reject the data that doesn’t meet your requirements. For rejected data, you might have to notify the sources. Since this is the first step of the process, doing it correctly can save you a lot of time downstream.

Transform is the meaty part of the process, where most of the data processing is done. You might want to join data from multiple sources and clean it. You might want to standardize the value ranges (e.g. one data source might use “Male” and “Female” for genders, but another uses “M” and “F” or “1” and “2”). You can apply operations such as transposing, deduplicating, sorting, aggregating, deriving new features, more data validating, etc..

Load is deciding how and how often to load your transformed data into the target destination, which can be a file, a database, or a data warehouse.

The idea of ETL sounds simple but powerful, and it’s the underlying structure of the data layer at many organizations.

Figure 2-4. An overview of the ETL process

Structured vs. unstructured data

Structured data is data that follows a predefined data model, also known as a data schema. For example, the data model might specify that each data item consists of two values: the first value, “name”, is a string at most 50 characters, and the second value, “age”, is an 8-bit integer in the range between 0 and 200. The predefined structure makes your data easier to analyze. If you want to know the average age of people in the database, all you have to do is to extract all the age values and get their mean.

The disadvantage of structured data is that you have to commit your data to a predefined schema. If your schema changes, you’ll have to retrospectively update all your data and/or the changes will cause mysterious bugs. For example, you’ve never kept your users’ email addresses before but now you do, so you have to retrospectively update email information to all previous users. One of the strangest bugs one of my colleagues encountered was when they could no longer use users’ age with their transactions, and their data schema replaced all the null age with 0, and their ML model thought the transactions were made by people of 0 years old.

Because business requirements change over time, committing to a predefined data schema can become too restricting. Or you might have data from multiple data sources, many of the sources are beyond your control, and it’s impossible to make them follow the same schema. This is where unstructured data becomes appealing. Unstructured data is data that doesn’t adhere to a predefined data schema. It’s usually text but can also be numbers, dates, etc. For example, a text file of logs generated by your ML model is unstructured data.

Even though unstructured data doesn’t adhere to a schema, it might still contain intrinsic patterns that help you extract structures. For example, the following text is unstructured, but you can notice the pattern that each line contains two values separated by a comma, the first value is textual and the second value is numerical. However, there is no guarantee that all lines must follow this format. You can add a new line to that text even if that line doesn’t follow this format.

“Lisa, 43

Jack, 23

Nguyen, 59”

Unstructured data also allows for more flexible storage options. For example, if your storage follows a schema, you can only store data following that schema. But if your storage doesn’t follow a schema, you can store any type of data. You can convert all your data, regardless of types and formats into bytestrings and store them together.

A repository for storing structured data is called a data warehouse. A repository for storing unstructured data is called a data lake. Data lakes are usually used to store raw data before processing. Data warehouses are used to store data that have been processed into formats ready to be used.

ETL to ELT

When the Internet first became ubiquitous and hardware had just become so much more powerful, collecting data suddenly became so much easier. The amount of data grew rapidly. Not only that, but the nature of data also changed. The number of data sources expanded, and data schemas evolved.

Finding it difficult to keep data structured, some companies had this idea: “Why not just store all data in a data lake so we don’t have to deal with schema changes? Whichever application needs data can just pull out raw data from there and process it.” This process of loading data into storage first then processing it later is sometimes called ELT (extract, load, transform). This paradigm allows for the fast arrival of data since there’s little processing needed before data is stored.

However, as data keeps on growing, this idea becomes less attractive. It’s expensive to store everything, and it’s inefficient to search through a massive amount of raw data for the piece of data that you want. At the same time, as companies switch to running applications on the cloud and infrastructures become standardized, data structures also become standardized. Committing data to a predefined schema becomes more feasible.

Here is a summary of the key differences between structured and unstructured data.

Table 2-3. The key differences between structured and unstructured data

Structured data

Unstructured data

Schema clearly defined

Data doesn’t have to follow a schema

Easy to search and analyze

Fast arrival

Can only handle data with a specific schema

Can handle data from any source

Schema changes will cause a lot of troubles

No need to worry about schema changes

Stored in data warehouses

Stored in data lakes

Summary

In this chapter, we started with the question about the role of data in building intelligent systems. There are still many people who believe that having intelligent algorithms will eventually trump having a large amount of data. However, the success of systems including AlexNet, BERT, GPT showed that the progress of ML in the last decade relies on having access to a large amount of data.

Therefore, it’s important for ML practitioners to know how to manage and process a large amount of data. This chapter covered the fundamentals of data engineering that I wish I knew when I started my ML career, from handling data from different data sources, choosing the right data format, to processing structured and unstructured data. These fundamentals will hopefully help readers become better prepared when facing seemingly overwhelming data in production.

1   More data usually beats better algorithms (Anand Rajaraman, Datawocky 2008)

2   The Bitter Lesson (Richard Sutton, 2019)

3   Tweet by Dr. Judea Pearl (2020)

4   Deep Learning and Innate Priors (Chris Manning vs. Yann LeCun debate).

5   The Bitter Lesson (Richard Sutton, 2019)

6   The Unreasonable Effectiveness of Data (Alon Halevy, Peter Norvig, and Fernando Pereira, Google 2009)

7   The AI Hierarchy of Needs (Monica Rogati, 2017)

8   1 Billion Word Language Model Benchmark (Chelba et al., 2013)

9   “Interesting” in production usually means catastrophic, such as a crash or when your cloud bill hits an astronomical amount.

10   An ML engineer once mentioned to me that his team only used users’ historical product browsing and purchases to make recommendations on what they might like to see next. I responded: “So you don’t use personal data at all?” He looked at me, confused. “If you meant demographic data like users’ age, location then no, we don’t. But I’d say that a person’s browsing and purchasing activities are extremely personal.”

11   For more Pandas quirks, check out just-pandas-things (Chip Huyen, GitHub 2020).

12   Announcing Amazon Redshift data lake export: share data in Apache Parquet format (Amazon AWS 2019).

Chapter 3. Data Engineering: Training Data

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at chip@huyenchip.com.

Despite the importance of training data in developing and improving ML models, the conversation is heavily skewed towards modeling, which is considered by many researchers and engineers as the “fun” part of the process. Building a state-of-the-art model is interesting. Spending days wrangling with a massive amount of malformatted data that doesn’t even fit into memory is frustrating.

Data is messy, complex, unpredictable, and potentially treacherous. If in school, training data is a cute little puppy then in production, it’s a kraken that, if not tamed, can easily sink your entire ML operation. But this is precisely the reason why ML engineers should learn how to handle data well, whipping it into the shape and form that we want, saving us time and headache down the road.

In this chapter, we will go over techniques to obtain or create good training data. It starts with different sampling techniques to select data for training. We’ll then address common challenges in creating training data including the label multiplicity problem, the lack of labels problem, and the class imbalance problem.

We use the term “training data” instead of “training datasets”, because “datasets” denote a set that is finite and stationery. Data in production is neither finite nor stationery, a phenomenon that we will cover in Chapter [TODO].

Before we move forward, I just want to echo a word of caution that has been said many times yet still not enough. Data is full of potential biases. These biases have many possible sauces. There are biases caused during collecting, sampling, or labeling. Historical data might be embedded with human biases and ML models, trained on this data, can perpetuate them. Use data but don’t trust it too much!

Sampling

Unless we have access to all possible data in the real-world, all the data that we use are subsets of real-world data, created by one sampling method or another. Sampling is an integral part of the ML workflow that is, unfortunately, often overlooked in typical ML coursework. Understanding different sampling methods and how they are being used in our workflow can, first, help us avoid potential sampling biases, and second, help us choose the methods that improve the efficiency of the data we sample.

There are two families of sampling: non-probability sampling and random sampling. We will start with non-probability sampling methods, followed by several common random methods. We’ll analyze the pros and cons of each method.

Non-Probability Sampling

Non-probability sampling is when selection of data isn’t based on any probability criteria. Here are some of the criteria for non-probability sampling.

Convenience sampling

Samples of data are selected based on their availability. This sampling method is popular because, well, it’s convenient.

Snowball sampling

Future samples are selected based on existing samples. For example, to scrape legitimate Twitter accounts without having access to Twitter databases, you start with a small number of accounts then you scrape all the accounts in their following, and so on.

Judgment sampling

Experts decide what samples to include.

Quota sampling

You select samples based on quotas for certain slices of data without any randomization.

The samples selected by non-probability criteria are not representative of the real-world data, and therefore, are ridden with selection biases. Because of these biases, you might think that it’s a bad idea to select data to train ML models using this family of sampling methods. You’re right. Unfortunately, in many cases, the selection of data for ML models is still driven by convenience.

One example of these cases is language modeling. Language models are often trained not with data that is representative of all possible texts but with data that can be easily collected—Wikipedia, CommonCrawl, Reddit.

Another example is data for sentiment analysis or recommendation systems. Many of this data are collected from sources with natural labels (ratings)—IMDB reviews, Amazon reviews. These sources don’t include people who don’t have access to the Internet and aren’t willing to put reviews online.

The third example is data for training self-driving cars. Data collected for self-driving cars come largely from two areas: Phoenix in Arizona (because of its lax regulations) and the Bay Area in California (because many companies that build self-driving cars are located here). Both areas have generally sunny weather, which means that there’s a lot more self-driving car data for sunny weather than for rainy or snowy weather.

Simple Random Sampling

In the simplest form of random sampling, you give all samples in the population equal probabilities of being selected. For example, you randomly select 10% of all samples, giving all samples an equal 10% chance of being selected.

The advantage of this method is that it’s easy to implement. The drawback is that rare categories of data might not appear in your selection. Consider the case where a class appears only in 0.01% of your data population. If you randomly select 1% of your data, samples of this rare class will unlikely be selected. Models trained on this selection might think that this rare class doesn’t exist.

Stratified Sampling

To avoid the drawback of simple random sampling listed above, you can first divide your population into the groups that you care about and sample from each group separately. For example, to sample 1% of data that has two classes A and B, you can sample 1% of class A and 1% of class B. This way, no matter how rare class A or B is, you’ll ensure that samples from it will be included in the selection. Each group is called a strata, and this method is called stratified sampling.

One drawback of this sampling method is that it isn’t always possible, such as when it’s impossible to divide all samples into groups. This is especially challenging when one sample might belong to multiple groups as in the case of multilabel tasks, e.g. a sample can be both class A and class B.

Weighted Sampling

In weighted sampling, each sample is given a weight, which determines the probability of it being selected. For example, if you have three samples A, B, C and want them to be selected with the probabilities of 50%, 30%, 20% respectively, you can give them the weights 0.5, 0.3, 0.2.

This method allows you to leverage domain expertise. For example, if you know that a certain subpopulation of data, such as more recent data, is more valuable to your model and want it to have a higher chance of being selected, you can give it higher weight.

This also helps with the case when the data you have comes from a different distribution compared to the true data. For example, if in your data, red samples account for 25% and blue samples account for 75%, but you know that in the real world, red and blue have equal probability to happen, you can give red samples the weights three times higher than blue samples.

In Python, you can do weighted sampling with random.choices as following:

# Choose two items from the list such that 1, 2, 3, 4 each has

# 20% chance of being selected, while 100 and 1000 each have only 10% chance.

random.choices(population=[1, 2, 3, 4, 100, 1000],

weights=[0.2, 0.2, 0.2, 0.2, 0.1, 0.1],

k=2)

# This is equivalent to the following

random.choices(population=[1, 1, 2, 2, 3, 3, 4, 4, 100, 1000],

k=2)

A concept common in ML that is closely related to weighted sampling is sample weights. Weighted sampling is used to select samples to train your model with, whereas sample weights are used to assign “weights” or “importance” to training samples. Samples with higher weights affect the loss function more. Changing sample weights can change your model’s decision boundaries significantly.

Figure 3-1. How sample weights can affect the decision boundary. On the left is when all samples are given equal weights. On the right is when samples are given different weights. Source: SVM: Weighted samples (sklearn), BSD License.

Importance Sampling

Importance sampling is one of the most important sampling methods not just in ML. It allows us to sample from a distribution when we only have access to another distribution.

Imagine you have to sample from a distribution , but is really expensive, slow, or infeasible to sample from. However, you have a distribution that is a lot easier to sample from. So you sample from instead and weight this sample by is called the proposal distribution or the importance distribution. can be any distribution as long as whenever . The equation below shows that in expectation, sampled from is equal to sampled from weighted by .

One example where importance sampling is used in ML is with policy-based reinforcement learning. Consider the case when you want to update your policy. You want to estimate the value functions of the new policy, but calculating the total rewards of taking an action can be costly because it requires considering all possible outcomes until the end of the time horizon after that action. However, if the new policy is relatively close to the old policy, you can calculate the total rewards based on the old policy instead and reweight them according to the new policy. The rewards from the old policy make up the proposal distribution. Figure 3-2 is an example to show how to use importance sampling to do policy gradient.

Figure 3-2. An example of how to use importance sampling to do policy gradient

Reservoir Sampling

Reservoir sampling is a fascinating algorithm that is especially useful when you have to deal with continually incoming data, which is usually what you have in production.

Imagine you have an incoming stream of tweets and you want to sample a certain number k of tweets to do analysis or training a model on. You don’t know how many tweets there are but you know you can’t fit them all in memory, which means you don’t know the probability at which a tweet should be selected. You want to ensure that one, every tweet has an equal probability of being selected, and two, you can stop the algorithm at any time and the tweets are sampled with the correct probability.

One solution for this problem is reservoir sampling. The algorithm involves a reservoir, which can be an array, and consists of the three following steps.

1. Put the first k elements into the reservoir.

2. For each incoming nth element, generate a random number i such that 1 ≤ i ≤ n.

3. If 1 ≤ i ≤ k: replace the ith element in the reservoir with the nth element. Else, do nothing.

This means that each incoming element has probability of being in the reservoir. You can also prove that each element in the reservoir has probability of being there. This means that all samples have an equal chance of being selected. If we stop the algorithm at any time, all samples in the reservoir have been sampled with the correct probability.

Figure 3-3. A visualization of how reservoir sampling works

Labeling

Despite the promise of unsupervised ML, most ML models in production today are supervised, which means that they need labels to learn. The performance of an ML model still depends heavily on the quality and quantity of the labeled data it’s trained on.

There are tasks where data has natural labels or it’s possible to collect natural labels on the fly. For example, for predicting click-through-rate on an ad, labels are whether users click on an ad or not. Similarly, for recommendation systems, labels are whether users click on a recommended item or not. However, for most tasks, natural labels are not available or not accessible, and you will need to obtain labels by other means.

In a talk to my students, Andrej Karpathy, Director of AI at Tesla, shared an anecdote about when he decided to have an in-house labeling team, his recruiter asked how long he’d need this team for. He responded: “How long do we need an engineering team for?” Data labeling has gone from being an auxiliary task to being a core function of many ML teams in production.

In this section, we will discuss the challenges of obtaining labels for your data, and their proposed solutions.

Hand Labels

Anyone who has ever had to work with data in production has probably felt this at a visceral level: acquiring hand labels for your data is difficult for many, many reasons. First, hand-labeling data can be expensive, especially if subject matter expertise is required. To classify whether a comment is spam, you might be able to find 200 annotators on a crowdsourcing platform and train them in 15 minutes to label your data. However, if you want to label chest X-rays, you’d need to find board-certified radiologists, whose time is limited and expensive.

Second, hand labeling poses a threat to data privacy. Hand labeling means that someone has to look at your data, which isn’t always possible if your data has strict privacy requirements. For example, you can’t just ship your patient’s medical records or your company’s confidential financial information to a third party service for labeling. In many cases, your data might not even be allowed to leave your organization, and you might have to hire or contract annotators to label your data on premise.

Third, hand labeling is slow. While the more data a person labels, the faster their annotation speed will become, the improvement isn’t in orders of magnitude. Labeling 1000 samples takes approximately 10 times longer than labeling 100 samples. For example, accurate transcription of speech utterance at phonetic level can take 400 times longer than the utterance duration1. So if you want to annotate 1 hour of speech, it’ll take 400 hours or almost 3 working months to do so. In a study to use ML to help classify lung cancers from X-rays, my colleagues had to wait almost a year to obtain sufficient labels.

Slow labeling leads to slow iteration speed and makes your model less adaptive to changing environments and requirements. If the task changes or data changes, you’ll have to wait for your data to be relabeled before updating your model. Imagine the scenario when you have a sentiment analysis model to analyze the sentiment of every tweet that mentions your brand. It has only two classes: NEGATIVE and POSITIVE. However, after deployment, your PR team realizes that the most damage comes from angry tweets and they want to attend to angry messages faster. So you have to update your sentiment analysis model to have three classes: NEGATIVE, POSITIVE, and ANGRY. To do so, you will need to look at your data again to see which existing training examples should be relabeled ANGRY. If you don’t have enough ANGRY examples, you will have to collect more data. The longer the process takes, the more your existing model performance will degrade.

Label Multiplicity

Often, to obtain enough labeled data, companies have to use data from multiple sources and rely on multiple annotators who have different levels of expertises. These different data sources and annotators have different levels of accuracy. This leads to the problem of label ambiguity or label multiplicity: what to do when there are multiple possible labels for a data instance.

Consider this simple task of entity recognition. You give three annotators the following sample and ask them to annotate all entities they can find.

Darth Sidious, known simply as the Emperor, was a Dark Lord of the Sith who reigned over the galaxy as Galactic Emperor of the First Galactic Empire.

You receive back three different solutions, as follows. Three annotators have identified different entities. Which one should your model train on? A model trained on data labeled mostly by annotator 1 will perform very differently from a model trained on data labeled mostly by annotator 2.

Table 3-1. Identities identified by different annotators might be very different

Annotator

# entities

Annotation

1

3

[Darth Sidious], known simply as the Emperor, was a [Dark Lord of the Sith] who reigned over the galaxy as [Galactic Emperor of the First Galactic Empire ]

2

6

[Darth Sidious], known simply as the [Emperor], was a [Dark Lord] of the [Sith] who reigned over the galaxy as [Galactic Emperor] of the [First Galactic Empire ].

3

4

[Darth Sidious], known simply as the [Emperor], was a [Dark Lord of the Sith] who reigned over the galaxy as [Galactic Emperor of the First Galactic Empire ].

Disagreements among annotators are extremely common. The higher level of domain expertise required, the higher the potential for annotating disagreement2. If one human-expert thinks the label should be A while another believes it should be B, how do we resolve this conflict to obtain one single ground truth? If human experts can’t agree on a label, what does human-level performance even mean?

To minimize the disagreement among annotators, it’s important to, first, have a clear problem definition. For example, in the entity recognition task above, some disagreements could have been eliminated if we clarify that in case of multiple possible entities, pick the entity that comprises the longest substring. This means Galactic Emperor of the First Galactic Empire instead of Galactic Emperor and First Galactic Empire. Second, you need to incorporate that definition into training to make sure that all annotators understand the rules.

Using data from multiple sources indiscriminately without examining their quality can cause your model to fail mysteriously. Consider a case when you’ve trained a moderately good model with 100K data samples. Your ML engineers are confident that more data will improve the model performance, so you spend a lot of money to hire annotators to label another million data samples.

However, the model performance actually decreases after being trained on the new data. The reason is that the new million samples were crowdsourced to annotators who labeled data with much less accuracy than the original data. It can be especially difficult to remedy this if you’ve already mixed your data and can’t differentiate new data from old data.

On top of that, it’s good practice to keep track of the origin of each of our data samples as well as its labels, a technique known as data lineage. Data lineage helps us both flag potential biases in our data as well as debug our models. For example, if our model fails mostly on the recently acquired data samples, you might want to look into how the new data was acquired. On more than one occasion, we’ve discovered that the problem wasn’t with our model, but because of the unusually high number of wrong labels in the data that we’d acquired recently.

Handling the Lack of Hand Labels

Because of the challenges in acquiring sufficient high-quality labels, many techniques have been developed to address the lack of labels problem. In this section, we will cover four of them: weak supervision, semi supervision, transfer learning, and active learning.

Table 3-2. Summaries for four techniques for handling the lack of hand labeling data

Method

How

Ground truths required?

Weak supervision

Leverages structural assumptions to generate labels

A small amount of initial labels as seeds to generate more labels

Semi supervision

Leverages (often noisy) heuristics to generate labels

A small amount of labels are recommended to guide the development of heuristics

Transfer learning

Leverages models pretrained on another task for your new task

No for zero-shot learning

Yes for fine-tuning

Active learning

Labels data samples that are most useful to your model

Yes

Weak supervision

If hand labeling is so problematic, what if we don’t use hand labels altogether? One approach that has gained popularity is weak supervision. One of the most popular open source tools for weak supervision is Snorkel, developed at Stanford AI Lab3. The insight behind weak supervision is that people rely on heuristics, which can be developed with subject matter expertise, to label data. For example, a doctor might use the following heuristics to decide whether a patient’s case should be prioritized as emergent.

If the nurse’s note mentions a serious condition like pneumonia, the patient’s case should be given priority consideration.

Libraries like Snorkel are built around the concept of labeling function: a function that encodes heuristics. The above heuristics can be expressed by the following function.

def labeling_function(note):

if "pneumonia" in note:

return "EMERGENT"

Labeling functions can encode many different types of heuristics, such as keyword heuristic (e.g. the example above), regular expressions (e.g. if the note matches or not matches a certain regular expression), database lookup (e.g. if the note contains the disease listed in the dangerous disease list), even the outputs of other models (e.g. if an existing system classifies this as EMERGENT). After you’ve written labeling functions, you can apply them to the samples you want to label.

Because labeling functions encode heuristics, and heuristics are noisy, labeling functions are noisy. Multiple label functions might apply to the same data examples, and they might give conflicting labels. One function might think a note is EMERGENT but another function might think it’s not. One heuristic might be much more accurate than another heuristic, which you might not know because you don’t have ground truth labels to compare them to. It’s important to combine, denoise, and reweight all labeling functions to get a set of most likely-to-be-correct labels.

Figure 3-4. A high level overview of how labeling functions are combined.4

In theory, you don’t need any hand labels for weak supervision. However, to get a sense of how accurate your labeling functions are, a small amount of hand labels is recommended. These hand labels can help you discover patterns in your data to write better labeling functions.

Weak supervision can be especially useful when your data has strict privacy requirements. You only need to see a small, cleared subset of data to write labeling functions, which can be applied to the rest of your data without looking at it.

With labeling functions, subject matter expertise can be versioned, reused, and shared. Expertise owned by one team can be encoded and used by another team. If your data changes or your requirements change, you can just reapply labeling functions to your data samples.

Table 3-3. The advantages of programmatic labeling over hand labeling

Hand labeling

Programmatic labeling

Expensive : esp. when subject matter expertise required

Cost saving : Expertise can be versioned, shared, reused across organization

Non-private : Need to ship data to human annotators

Privacy : Create LFs using a cleared data subsample then apply LFs to other data without looking at individual samples.

Slow : Time required scales linearly with # labels needed

Fast : Easily scale from 1K to 1M samples

Non-adaptive : Every change requires re-labeling the data

Adaptive : When changes happen, just reapply LFs!

Here is a case study to show how well weak supervision works in practice. In a case study with Stanford Medicine, models trained with weakly-supervised labels obtained by a single radiologist after 8 hours of writing labeling functions had comparable performance with models trained on data obtained through almost a year of hand labeling. There are two interesting facts about the results of the experiment. First, the models continued improving with more unlabeled data even without more labeling functions. Second, labeling functions were being reused across tasks. The researchers were able to reuse 6 labeling functions between the CXR (Chest X-Rays) task and EXR (Extremity X-Rays) task.5

Figure 3-5. Comparison of the performance of a model trained on fully-supervised labels (FS) and a model trained with programmatic labels (DP) on CXR and EXR tasks.

My students often ask that if heuristics work so well to label data, why do we need machine learning models? One reason is that your labeling functions might not cover all your data samples, so you need to train ML models to generalize to samples that aren’t covered by any labeling function.

Weak supervision is a simple but powerful paradigm. However, it’s not perfect. In some cases, the labels obtained by weak supervision might be too noisy to be useful. But it’s often a good method to get your started when you want to explore the effectiveness of ML without wanting to invest too much in hand labeling upfront.

Semi supervision

If weak supervision leverages heuristics to obtain noisy labels, semi supervision leverages structural assumptions to generate new labels based on a small set of initial labels. Unlike weak supervision, semi supervision requires an initial set of labels.

Semi supervised learning is a technique that was used back in the 90s6, and since then, many semi supervision methods have been developed. A comprehensive review of semi-supervised learning is out of the scope of this book. We’ll go over a small subset of these methods to give readers a sense of how they are used. For a comprehensive review, we recommend Semi-Supervised Learning Literature Survey (Xiaojin Zhu, 2008) and A survey on semi-supervised learning (Engelen and Hoos, 2018).

A classic semi-supervision method is self-training. You start by training a model on your existing set of labeled data, and use this model to make predictions for unlabeled samples. Assuming that predictions with high raw probability scores are correct, you add the labels predicted with high probability to your training set, and train a new model on this expanded training set. This goes on until you’re happy with your model performance.

Another semi supervision method assumes that data samples that share similar characteristics share the same labels. The similarity might be obvious, such as in the task of classifying the topic of Twitter hashtags as follows. You can start by labeling the hashtag “#AI” as Computer Science. Assuming that hashtags that appear in the same tweet or profile are likely about the same topic, given the profile of MIT CSAIL below, you can also label the hashtags “#ML” and “#BigData” as Computer Science.

Figure 3-6. Because #ML and #BigData appears in the same Twitter profile as #AI, we can assume that they belong to the same topic.

In most cases, the similarity can only be discovered by more complex methods. For example, you might need to use a clustering method or a K-nearest neighbor method to discover samples that belong to the same cluster.

A semi supervision method that has gained popularity in recent years is perturbation-based method. It’s based on the assumption that small perturbations to a sample shouldn’t change its label. So you apply small perturbations to your training samples to obtain new training samples. The perturbations might be applied directly to the samples (e..g adding white noise to images) or to their representations (e.g. adding small values to embeddings of words). The perturbed samples have the same labels with the unperturbed samples. This method is closely related to adversarial training7.

In some cases, SSL approaches have reached the performance of purely supervised learning, even when a substantial portion of the labels in a given dataset has been discarded8. Semi supervision is the most useful when the number of training labels is limited. One thing to consider when doing semi supervision is how much of this limited amount should be used for evaluation. If you evaluate multiple model candidates on the same test set and choose the one that performs best on the test set, you might have chosen a model that overfits the most on the test set. On the contrary, if you choose models on a validation set, the value gained by having a validation set might be less than the value gained by adding the validation set to the limited training set.

Transfer learning

Transfer learning refers to the family of methods where a model developed for a task is reused as the starting point for a model on a second task. You (or someone else) first train a base model for a base task such as language modeling. The base task is usually a task that has cheap and abundant training data. Language modeling is a great candidate because it doesn’t require labeled data.

You then fine-tune this pretrained base model on the task that you’re interested in, such as sentiment analysis, intent detection, question answering, etc.. This task is called a downstream task, and we say that this model is fine-tuned for this downstream task.

Transfer learning is especially appealing for tasks that don’t have a lot of labeled data. Even for tasks that have a lot of labeled data, using a pretrained model as the starting point can often boost the performance significantly compared to training from scratch.

Transfer learning has gained a lot of interest in recent years for the right reasons. It has enabled many applications previously impossible due to the lack of training samples. A non-trivial portion of ML models in production today are the results of transfer learning, including object detection models that leverage models pretrained on ImageNet and text classification models that leverage pretrained language models such as BERT9 or GPT-310. It also lowers the entry barriers into ML, as it helps reduce the upfront cost needed for labeling data to build ML applications.

A trend that has emerged in the last five years is that the larger the pretrained base model, the better its performance on downstream tasks. Large models are expensive to train. Based on the configuration of GPT-3, it’s estimated that the cost of training this model is in the tens of million USD. Many have hypothesized that in the future, only a handful of companies can afford to train large pretrained models. The rest of the industry will use these pretrained models directly or finetune them for their specific needs.

Active learning

Active learning is a method for improving the efficiency of data labels. The hope here is that ML models can achieve greater accuracy with fewer training labels if they can choose which data samples to learn from. Active learning is sometimes called query learning, though this term is getting increasingly unpopular, because a model (active learner) sends back queries in the form of unlabeled samples to be labeled by annotators (usually humans).

Instead of randomly labeling data samples, you label the samples that are most helpful to your models according to some heuristics. The most straightforward heuristic is uncertainty measurement—label the examples that your model is the least certain about hoping that they will help your model learn the decision boundary better. For example, in the case of classification problems where your model outputs raw probabilities for different classes, it might choose the data examples with the lowest probabilities for the predicted class. Figure 4-7 illustrates how well this method works on a toy example.

Figure 3-7. How uncertainty-based active learning works. (a) A toy data set of 400 instances, evenly sampled from two class Gaussians. (b) A model trained on 30 examples randomly labeled gives the accuracy of 70%. (c) A model trained on 30 examples chosen by active learning gives the accuracy of 90%. Image by Burr Settles.11

Another common heuristic is based on disagreement among multiple candidate models. This method is called query-by-committee. You need a committee of several candidate models, which are usually the same model trained with different sets of hyperparameters. Each model can make one vote for which examples to label next, which it might vote based on how uncertain it is about the prediction. You then label the examples that the committee disagrees on the most.

There are other heuristics such as choosing examples that, if trained on them, will give the highest gradient updates, or will reduce the loss the most. For a comprehensive review of active learning methods, check out Active Learning Literature Survey (Burr Settles, 2010).

The examples to be labeled can come from different data regimes. They can be synthesized: your model generates examples in the region of the input space that it’s most uncertain about to be labeled12. They can come from a stationary distribution: you’ve already collected a lot of unlabeled data and your model chooses examples from this pool to label. They can come from the real-world distribution: you have a stream of data coming in, as in production, and your model chooses examples from this stream of data to label.

I’m the most excited about active learning in the stream data regime. Data changes all the time, a phenomenon we briefly touched on in Chapter 1 and will go more in detail in Chapter [TODO]. Active learning in this data regime will allow your model to learn more effectively in real-time and adapt faster to changing environments.

Class Imbalance

Class imbalance typically refers to a problem in classification tasks where there is a substantial difference in the number of samples in each class of the training data. For example, in a training dataset for the task of detecting lung cancer from X-Ray images, 99.99% of the X-Rays might be of normal lungs, and only 0.01% might contain cancerous cells.

Challenges of Class Imbalance

ML works well in situations when the data distribution is more balanced, and not so well when the classes are heavily imbalanced. Class imbalance can make learning difficult for the three following reasons.

Figure 3-8. ML works well in situations where the classes are balanced. Image by Andrew Ng13.

The first reason is that class imbalance often means that there’s insufficient signal for your model to learn to detect the minority classes. In the case where there is a small number of instances in the minority class, the problem becomes a few-shot learning problem where your model only gets to see the minority class a few times before having to make a decision on it. In the case where there is no instance of the rare classes in your training set, your model might assume that these rare classes don’t exist.

The second reason is that class imbalance makes it easier for your model to get stuck in a local minima by learning a simple heuristic instead of learning anything useful about the underlying structure of the data. Consider the lung cancer detection example above. If your model learns to always output the majority class, its accuracy is already 99.99%. This local minima can be very hard for gradient-descent algorithms to get out, because a small amount of randomness added to this heuristic might lead to worse accuracy.

The third reason is that class imbalance leads to asymmetric cost of error—the cost of a wrong prediction on an example of the rare class might be much higher than a wrong prediction on an example of the majority class. For example, misclassification on an X-Ray with cancerous cells is much more dangerous than misclassification on an X-Ray of a normal lung. If your loss function isn’t configured to address this asymmetry, your model will treat all examples the same way. As a result, you might obtain a model that performs equally well on both majority and minority classes, while you much prefer a model that performs less well on the majority class but much better on the minority one.

When I was in school, most datasets I was given had more or less balanced classes, because I imagined that it would be easier for me to learn about neural networks if I was free from the huge roadblock caused by class imbalance. It was a shock for me to start working and realize that class imbalance is the norm. In the real-world settings, rare events are often more interesting (or more dangerous) than regular events, and many tasks focus on detecting those rare events.

The classical example of tasks with class imbalance is fraud detection. Most credit card transactions are not fraudulent. As of 2018, 6.8¢ for every $100 in cardholder spending is fraudulent14. Another is churn prediction. The majority of your customers are not planning on cancelling their subscription. If they are, your business has more to worry about than churn prediction algorithms. Other examples include disease screening—most people, fortunately, don’t have terminal illness, and resume screening—98% of job seekers are eliminated at the initial resume screening15. A less obvious example of a task with class imbalance is object detection. Object detection algorithms currently work by generating a large number of bounding boxes over an image then predicting which boxes are most likely to have objects in them. Most bounding boxes do not contain a relevant object.

Outside the cases where class imbalance is inherent in the problem, class imbalance can also be caused by biases during the sampling process. Consider the case when you want to create training data to detect whether an email is spam or not. You decide to use all the anonymized emails from your company’s email database. According to Talos Intelligence, as of May 2021, nearly 85% of all emails are spam16. But most spam emails were filtered out before they reached your company’s database, so in your dataset, only a small percentage is spam.

Another cause for class imbalance, though less common, is due to labeling errors. Your annotators might have read the instructions wrong or followed the wrong instructions (thinking there are only two classes POSITIVE and NEGATIVE while there are actually three), or simply made errors. Whenever faced with the problem of class imbalance, it’s important to examine your data to understand the causes of it.

Handling Class Imbalance

Because of its prevalence in real-world applications, class imbalance has been thoroughly studied over the last two decades17. Class imbalance affects tasks differently based on the level of imbalance. Some tasks are more sensitive to class imbalance than others. Japkowicz showed that sensitivity to imbalance increases, and that non-complex, linearly separable problems are unaffected by all levels of class imbalance18. Class imbalance in binary classification problems is a much easier problem than class imbalance in multiclass classification problems. Ding et al. showed that very-deep neural networks—with “very deep” meaning over 10 layers back in 2017—performed much better on imbalanced data than shallower neural networks19.

There have been many techniques suggested to mitigate the effect of class imbalance. However, as neural networks have grown to be much larger and much deeper, with more learning capacity, some might argue that you shouldn’t try to “fix” class imbalance if that’s how the data looks like in the real world. A good model should learn to model that class imbalance. However, developing a model good enough for that can be challenging, so we still have to rely on special training techniques.

In this section, we will cover three aspects: choosing the right metrics for your problem, data-level methods, which means changing the data distribution to make it less imbalanced, and algorithm-level methods, which means changing your learning method to make it more robust to class imbalance.

These techniques might be necessary but not sufficient. For a comprehensive survey of , we recommend Survey on deep learning with class imbalance (Johnson and Khoshgoftaar, Journal of Big Data 2019).

Note that in practice, ensembles have shown to help with the class imbalance problem, but I haven’t been able to find literature to support this, so we won’t discuss ensembles here. Ensemble techniques are, however, important to know on their own and will be covered in Chapter 5: Model Development and Evaluation.

Using the right evaluation metrics

The most important thing to do when facing a task with class imbalance is to choose the appropriate evaluation metrics. Wrong metrics will give you the wrong ideas of how your models are doing, and subsequently, won’t be able to help you develop or choose models good enough for your task.

The overall accuracy and error rate are the most frequently used metrics to report the performance of ML models. However, they are insufficient metrics for tasks with class imbalance because they treat all classes equally, which means the performance of your model on the majority class will dominate the accuracy. This is especially bad when the majority class isn’t what you care about.

Consider a task with two labels: CANCER (positive) and NORMAL, where 90% of the labeled data is NORMAL. Consider two models A and B with the following confusion matrices.

Table 3-4. Model A’s confusion matrix

Model A

Actual CANCER

Actual NORMAL

Predicted CANCER

10

10

Predicted NORMAL

90

890

Model A can detect 10 out of 100 CANCER cases.

Table 3-5. Model B’s confusion matrix

Model B

Actual CANCER

Actual NORMAL

Predicted CANCER

90

90

Predicted NORMAL

10

810

Model B can detect 90 out of 100 CANCER cases.

If you’re like most people, you’d probably prefer model B to make predictions for you since it has a better chance of telling you if you actually have cancer. However, they both have the same accuracy of 0.9.

Metrics that help you understand your model’s performance with respect to specific classes would be better choices. Accuracy can still be a good metric if you use it for each class individually. The accuracy of Model A on the CANCER is 10% and the accuracy of model B on the CANCER class is 90%.

F1 and recall are metrics that measure your model’s performance with respect to the positive class in binary classification problems20. F1 and recall are asymmetric metrics, which means that their values change depending on which class is considered the positive class. In our case, if we consider CANCER the positive class, model A’s F1 is 0.17. However, if we consider NORMAL the positive class, model A’s F1 is 0.95.

In multiclass classification problems, you can calculate F1 for each individual class.

Table 3-6. Model A and model B have the same accuracy even though one model is clearly superior to another

CANCER (1)

NORMAL (0)

Accuracy

Precision

Recall

F1

Model A

10/100

890/900

0.9

0.5

0.1

0.17

Model B

90/100

810/900

0.9

0.5

0.9

0.64

Many classification problems can be modeled as regression problems. Your model can output a value, and based on that value, you classify the example. For example, if the value is greater than 0.5, it’s a positive label, and if it’s less than or equal to 0.5, it’s a negative label. This means that you can tune the threshold to increase the true positive rate (recall) while decreasing the false positive rate (also known as the probability of false alarm), and vice versa. The idea is that we can plot the true positive rate against the false positive rate for different thresholds. This plot is known as the ROC (Receiver Operating Characteristics). When your model is perfect, the recall is 1.0, and the curve is just a line at the top. This curve shows you how your model’s performance changes depending on the threshold, and helps you choose the threshold that works best for you. The closer to the perfect line the better your model’s performance.

The area under the curve (AUC) measures the area under the ROC curve. Since the closer to the perfect line the better, the larger this area the better.

Figure 3-9. ROC curve

Like F1 and recall, the ROC curve focuses only on the positive class and doesn’t show how well your model does on the nagive class. Davis and Goadrich suggested that we should plot precision against recall instead, in what they termed the Precision-Recall Curve. They argued that this curve s give a more informative picture of an algorithm’s performance on tasks with heavy class imbalance21.

Data-level methods: Resampling

Data-level methods modify the distribution of the training data to reduce the level of imbalance to make it easier for the model to learn. A common family of techniques is resampling. Resampling includes oversampling, adding more examples from the minority classes and undersampling, removing examples of the majority classes. The simplest way to undersample is to randomly remove instances from the majority class, while the simplest way to oversample is to randomly make copies of the minority class until you have a ratio that you’re happy with.

Figure 3-10. Illustrations of how undersampling and oversampling works. Image by Rafael Alencar 22

A popular method of undersampling low-dimensional data that was developed back in 1976 is Tomek links23. With this technique, you find pairs of samples from opposite classes that are close in proximity, and remove the sample of the majority class in each pair.

While this makes the decision boundary more clear and arguably helps models learn the boundary better, it may make the model less robust by removing some of the subtleties of the true decision boundary.

A popular method of oversampling low-dimensional data is SMOTE. It synthesizes novel samples of the minority class through sampling convex24 combinations of existing data points within the minority class.

Both SMOTE and Tomek Links have only been proven effective in low-dimensional data. Many of the sophisticated resampling techniques, such as Near-Miss25 and one-sided selection26, require calculating the distance between instances or between instances and the decision boundaries, which can be expensive or infeasible for high-dimensional data or in high-dimensional feature space, such as the case with large neural networks.

When you resample your training data, never evaluate your model on resampled data, since it’ll cause your model to overfit to that resampled distribution.

Undersampling runs the risk of losing important data from removing data. Oversampling runs the risk of overfitting on training data, especially if the added copies of the minority class are replicas of existing data. Many sophisticated sampling techniques have been developed to mitigate these risks.

One such technique is two-phase learning27. You first train your model on the resampled data. This resampled data can be achieved by randomly undersampling large classes until each class has only N instances each. You then finetune your model on the original data.

Another technique is dynamic sampling: oversample the low performing classes and undersample the high performing classes during the training process. Introduced by Pouyanfar et al.28, the method aims to show the model less of what it has already learned and more of what it has not.

Algorithm-level methods

If data-level methods mitigate the challenge of class imbalance by altering the distribution of your training data, algorithm-level methods keep the training data distribution intact but alter the algorithm to make it more robust to class imbalance.

Because the loss function (or the cost function) guides the learning process, many algorithm-level methods involve adjustment to the loss function. The key idea is that if there are two instances and and the loss resulting from making the wrong prediction on higher than , the model will prioritize making the correct prediction on over making the correct prediction on . By giving the training instances we care about higher weight, we can make the model focus more on learning these instances.

Let be the loss caused by the instance for the model with the parameter set . The model’s loss is often defined as the average loss caused by all instances.

This loss function values the loss caused by all instances equally, even though wrong predictions on some instances might be much costlier than wrong predictions on other instances. There are many ways to modify this cost function. In this section, we will focus on three of them, starting with cost-sensitive learning.

Cost-sensitive learning

Back in 2001, based on the insight that misclassification of different classes incur different cost, Elkan proposed cost-sensitive learning where the individual loss function is modified to take into account this varying cost29. The method started by using a cost matrix to specify : the cost if class is classified as class . If , it’s a correct classification, and the cost is usually 0. If not, it’s a misclassification. If classifying POSITIVE examples as NEGATIVE is twice as costly as the other way around, you can make twice as high as .

For example, if you have two classes: POSITIVE and NEGATIVE, the cost matrix can look like this.

Table 3-7. Example of a cost matrix

Actual NEGATIVE

Actual POSITIVE

Predicted NEGATIVE

Predicted POSITIVE

The loss caused by instance x of class i will become the weighted average of all possible classifications of instance x.

The problem with this loss function is that you have to manually define the cost matrix, which is different for different tasks at different scales.

Class-balanced loss

What might happen with a model trained on an imbalance dataset is that it’ll bias toward majority classes and make wrong predictions on minority classes. What if we punish the model for making wrong predictions on minority classes to correct this bias?

In its vanilla form, we can make the weight of each class inversely proportional to the number of samples in that class, so that the rarer classes have higher weights.

The loss caused by instance x of class i will become as follows, with Loss(x, j) being the loss when x is classified as class j. It can be cross entropy or any other loss function.

A more sophisticated version of this loss can take in account the overlapping among existing samples, such as Class-Balanced Loss Based on Effective Number of Samples (Cui et al., CVPR 2019).

Focal loss

In our data, some examples are easier to classify than others, and our model might learn to classify them quickly. We want to incentivize our model to focus on learning the samples they still have difficulty classifying. What if we adjust the loss so that if a sample has a lower probability of being right, it’ll have a higher weight? This is exactly what Focal Loss does30.

Figure 3-11. Model trained with focal loss (FL) shows reduced loss values compared to models trained with cross entropy loss (CE). Image by Lin et al. 31

1   Semi-Supervised Learning with Graphs (Xu, 2005)

2   If something is so obvious to label, you wouldn’t need domain expertise.

3   Snorkel: Rapid Training Data Creation with Weak Supervision (Ratner et al., 2017, Proceedings of the VLDB Endowment, Vol. 11, No. 3)

4   Snorkel: Rapid Training Data Creation with Weak Supervision (Ratner et al., 2017)

5   Cross-Modal Data Programming Enables Rapid Medical Machine Learning (Dunnmon et al., 2020)

6   Combining Labeled and Unlabeled Data with Co-Training (Blum and Mitchell, 1998)

7   Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (Miyato et al., 2017)

8   Realistic Evaluation of Deep Semi-Supervised Learning Algorithms (Oliver et al., NeurIPS 2018)

9   BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)

10   Language Models are Few-Shot Learners (OpenAI 2020)

11   Active Learning Literature Survey (Burr Settles, 2010)

12   Queries and Concept Learning (Dana Angluin, 1988)

13   Bridging AI’s Proof-of-Concept to Production Gap (Andrew Ng, Stanford HAI 2020)

14   Payment Card Fraud Losses Reach $27.85 Billion (Nilson Report, 2019)

15   Job Market Expert Explains Why Only 2% of Job Seekers Get Interviewed (WebWire, 2014)

16   Email & Spam Data (Talos Intelligence, 2021)

17   The Class Imbalance Problem: A Systematic Study (Nathalie Japkowciz and Shaju Stephen, 2002)

18   The Class Imbalance Problem: Significance and Strategies (Nathalie Japkowicz, 2000)

19   Facial action recognition using very deep networks for highly imbalanced class distribution (Ding et al., 2017)

20   As of July 2021, when you use scikit-learn.metrics.f1_score, pos_label is set to 1 by default, but you can change to 0 if you want 0 to be your positive label.

21   The Relationship Between Precision-Recall and ROC Curves (Davis and Goadrich, 2006).

22   Resampling strategies for imbalanced datasets (Rafael Alencar, Kaggle 2018)

23   An Experiment with the Edited Nearest-Neighbor Rule (Ivan Tomek, IEEE 1876)

24   “Convex” here approximately means “linear”.

25   KNN Approach to Unbalanced Data Distributions: A Case Study Involving Information Extraction (Zhang and Mani, 2003)

26   Addressing the curse of imbalanced training sets: one-sided selection (Kubat and Matwin, 2000)

27   Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning (Lee et al., 2016)

28   Dynamic sampling in convolutional neural networks for imbalanced data classification (Pouyanfar et al., 2018)

29   The foundations of cost-sensitive learning (Elkan, IJCAI 2001)

30   Focal Loss for Dense Object Detection (Lin et al., 2017)

31   Focal Loss for Dense Object Detection (Lin et al., 2017)

About the Author

Chip Huyen (https://huyenchip.com) is a best-selling author and engineer who develops tools and best practices for bringing AI research into production. Through her work at Netflix, NVIDIA, and Snorkel AI, she has helped some of the world’s largest organizations develop and deploy machine learning systems. She is the founder of a startup that focuses on real-time machine learning.

In 2017, she created and taught the Stanford course TensorFlow for Deep Learning Research. She is currently teaching CS 329S: Machine Learning Systems Design at Stanford. This book is based on the course’s lecture notes.

She is also the author of four Vietnamese books that have sold more than 100,000 copies. The first two books belong to the series Xách ba lô lên và Đi (Quảng Văn 2012, 2013). The first book in the series was the #1 best-selling book of 2012 on Tiki.vn. The series was among FAHASA’s Top 10 Readers Choice Books in 2014.

Chip’s expertise is in the intersection of software engineering and machine learning. LinkedIn included her among the 10 Top Voices in Software Development in 2019, and Top Voices in Data Science & AI in 2020.

1. 1. Machine Learning in Production

1. When and When not to Use Machine Learning

1. When To Use Machine Learning

2. When not to Use Machine Learning

3. Machine Learning Use Cases

2. Understanding Machine Learning Systems

1. Machine learning in research vs. in production

2. Machine learning systems vs. traditional software

3. Designing ML Systems in Production

1. Requirements for ML Systems

2. Iterative Process

4. Summary

2. 2. Data Engineering: Fundamentals

1. Mind vs. Data

2. Data Sources

3. Data Formats

1. JSON

2. Row-major vs. Column-major Format

3. Text vs. Binary Format

4. Data Processing

1. OLTP vs. OLAP

2. ETL: Extract, Transform, Load

5. Summary

3. 3. Data Engineering: Training Data

1. Sampling

1. Non-Probability Sampling

2. Simple Random Sampling

3. Stratified Sampling

4. Weighted Sampling

5. Importance Sampling

6. Reservoir Sampling

2. Labeling

1. Hand Labels

2. Handling the Lack of Hand Labels

3. Class Imbalance

1. Challenges of Class Imbalance

2. Handling Class Imbalance

Table of Contents

1. Machine Learning in Production

When and When not to Use Machine Learning

When To Use Machine Learning

When not to Use Machine Learning

Machine Learning Use Cases

Understanding Machine Learning Systems

Machine learning in research vs. in production

Machine learning systems vs. traditional software

Designing ML Systems in Production

Requirements for ML Systems

Iterative Process

Summary

2. Data Engineering: Fundamentals

Mind vs. Data

Data Sources

Data Formats

JSON

Row-major vs. Column-major Format

Text vs. Binary Format

Data Processing

OLTP vs. OLAP

ETL: Extract, Transform, Load

Summary

3. Data Engineering: Training Data

Sampling

Non-Probability Sampling

Simple Random Sampling

Stratified Sampling

Weighted Sampling

Importance Sampling

Reservoir Sampling

Labeling

Hand Labels

Handling the Lack of Hand Labels

Class Imbalance

Challenges of Class Imbalance

Handling Class Imbalance