Date: 2023-07-25T07:00:00+00:00
Description: Between major privacy regulations like the GDPR and CCPA and expensive and notorious data breaches, there has never been so much pressure for data scientists to ensure data privacy. Unfortunately, integrating privacy into your data science workflow is still complicated. This essential guide will give you solid advice and best practices on breakthrough privacy-enhancing technologies such as encrypted learning and differential privacy--as well as a look at emerging technologies and techniques in the field. Practical Data Privacy answers important questions such as: - What do privacy regulations
Publisher: O'Reilly Media, Inc.

Practical Data Privacy

Solving Privacy and Security Problems in Your Data Science Workflow

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Katharine Jarmul

Practical Data Privacy
by Katharine
Jarmul

Copyright © 2023 Kjamistan Inc. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Editors: Andy Kwan and Rita Fernando
- Production Editor: Katherine Tozer
- Interior Designer: David Futato
- Cover Designer: Karen Montgomery
- Illustrator: Kate Dullea

- July 2023: First Edition

Revision History for the Early Release

- 2022-07-20: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098129460 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc.
Practical Data Privacy, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-12817-3

Preface

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at katharine@kjamistan.com.

Welcome to the wonderful world of data privacy! You might have some preconceived notions around privacy — that it is a nuisance, that it is administrative and therefore boring, or that it’s a topic that only interests lawyers. What this book will show you is just how technically challenging and interesting data privacy problems are and will continue to be for years to come. If you entered the field of data science because you liked challenging mathematical and statistical problems, you will love exploring data privacy in data science. The topics you’ll learn in this book will expand your understanding of probability theory, modeling and even cryptography, and also I’ll also clarify when you do need assistance from legal professionals.

Learning how to solve data privacy problems is increasingly critical for data science practitioners today. You’ll be able to solve real world problems in the fields like cybersecurity, health care and finance, and to advance your career in a patchwork world of privacy laws, policies and protections. Since 2018 when the General Data Protection Regulation went into effect in Europe, the landscape has become more complicated, but it is likely to only increase as regulatory agencies and lawmakers continue to change the rules about how, where, why and when we store data. Building up your data privacy and data security skill set now is an investment in your future career.

Finally, taking the time to learn new privacy skills means you are contributing to our field in terms of trust, accountability, understanding and social responsibility. Currently, we see fear-mongering and a backlash against the use of machine learning to solve real world problems. Some of that negativity is based in real issues and actual deployments, where data, models and systems were not used in a trustworthy manner. For example, Clearview AI scrapes faces from social media sites and sells the facial recognition model built from those faces to law enforcement, raising questions regarding data ownership, privacy and fairness. To help counter this reputational damage, we need data scientists and machine learning engineers who understand the tasks at hand, the risks involved and who can competently solve the problems we all face. These problems are not impossible to overcome, but they are difficult. Use this book as you navigate these challenges, finding ways forward with practical, hands-on guidance.

In fact, I hope this book can contribute to new data science not currently available due to data privacy issues. Worldwide, concerns about digitizing personl data — even for responsible government use — are so prevaltent hat they obstruct the use of machine learning to provide assistance with social problems like climate change, financial auditing and global health crises. If privacy is build into our data science practice, it will enable more beneficial data science and help answer important questions for our countries, our societies and our world.

Who Should Read this Book

This book is for data scientists who want to upskill themselves with a focus on data privacy and security. You could have many reasons, such as:

- You’d like to pursue a specialization (data privacy) that you care about, which has a long future in the industry

- You want to move into a more regulated industry like finance or health care and these skills will set you up as a promising candidate in these sectors

- You work with research data and you’d like to get faster approval from ethics board reviews and publications

- You are a data science freelancer or consultant and want to expand your customer base by ensuring that you know how to manage sensitive data

- You manage a data team and want to be able to design products and architect solutions with attention to data privacy

- You would like to use “AI for good”, and think privacy is an important human right

- Your team has been told that privacy is important, but you aren’t sure of what that means or how to go about implementing it

- You work with sensitive data and want to ensure you are following best practices

- You’d like to become a privacy engineer and focus on engineering privacy into data products

- Privacy and Security are cool topics, and you just enjoy nerding out about them

I could go on and on, actually, and I have met different folks from all of these backgrounds. One thing I can tell you with certainty — demand for these skills is increasing rapidly, driven by much more than new regulation. Companies are investing in these skills so they can build a secure future for data management, without worrying that the company’s brand or reputation will be dragged through the headlines due to lack of attention to data privacy.

Note

To follow along the notebooks and exercises, you’ll need to have familiarity with Python and Jupyter notebooks. Statistical knowledge and understanding will also help, as well as familiarity with probability theory. When possible, I’ve added some basic explanations and references to guide you in case some ideas are new.

As you read this book and work through exercises, you’ll see how aspects of data privacy highlight the wonders of data science you already know and love. As with other challenging areas of data science, this book will take you from simple methods for solving privacy into more difficult ones, some of which we haven’t quite solved yet. I will share simpler approaches and methods for applying data privacy to everyday data management. Just like linear regression, we want to start simple and are elated when simple “just works” and you can move onto the next problem. That said, experienced practitioners know that the simple approach will solve the problem 50-60% of the time. For the rest, you will need to ask deeper questions that have mathematical, ethical and technical implications. These questions and the answers you’ll find as you explore them, will make you a better data scientist, technologist and mathematician/statistician.

This book may be all you require and you’ll be a data scientist with some extra skills around data privacy. That’s fine! You might also decide this book is the first of several in a path that takes you further into the field. In case that’s enticing now or later, let me introduce you to the concept of privacy engineering.

Privacy Engineering

In the next 10 years [1], I foresee that privacy engineering as a field will continue to grow in importance. The skills you gain by reading this book, working through the exercises and then applying this new knowledge to your work, will prepare you for this type of role. Let’s talk about privacy engineering is and what it isn’t, so you can explore your level of interest in this growing area.

At companies where data science is an important product, a privacy engineer is equal parts data scientist and engineer. This means that, unlike some roles in data science, you are actively engineering solutions rather than exploring data or testing an idea in a lab setting. This could mean working directly with the data engineering teams, the software or applications teams or even architects at your company to ensure privacy is actually built into the product itself as well as all the internal applications. You’ll need to understand engineering and architecture basics as a part of this work, especially as it pertains to designing systems and integrating systems with one another. Some related titles you can pick up on these topics are:

- Software Architecture in Practice, 4th Edition

- Fundamentals of Software Architecture

- Head First: Design Patterns

- Designing Data Intensive Applications

- Practical MLOps

Blindly implementing privacy policy by finding appropriate vendors to plug-and-play the problem away is a naive approach to these problems, but the growing space of privacy technology companies mean that you become a decision maker and leader at your organization for evaluating technologies to use, build and buy for data privacy management. In doing so, you’ll be using concepts learned from this book to put together evaluation criteria, ask probing questions on implementation and analyze the flexibility, support and product road maps. In this role, you will determine how well potential vendors can meet your company needs as the dependence on private, sensitive and confidential data grows.

A privacy engineer is also not just another data scientist who cares about privacy but is given no authority, time or budget to make decisions with regard to privacy. Although it is great that advocacy has become a part of the data science role, privacy engineering is about the building and application of privacy techniques as data is ingested, collected, transformed, stored and then used in data science applications. Advocacy is a nice side job, but implementation proves these theories work.

Nor is a privacy engineer just a data engineer who thinks about privacy. While privacy engineers can work alongside data engineers and often might embed in a team for a project or a Proof of Concept, they must work with different parts of the organization and will be pulled into many projects where their expertise is relevant. They are specialists — not locked into a single team for too long. Instead, their knowledge is a tremendously valuable resource that should be applied to the most pressing business problems affected by data privacy.

The position of a privacy engineer is still being defined and continues to evolve. Although there are several companies who actively hire for these roles now, it reminds me of the era when the term “machine learning engineer” first came into use in 2018-2019. It’s a relatively new skill set in the data science space that is emerging because of the industry needs and demands. I am excited to see how privacy engineers 2 or 10 years from now. I am confident that there will be many of them and that, if you wish, you can be one of them!

Why I Wrote This Book

When I first became interested in data privacy, it felt like a maze. Most of the material was beyond my comprehension and the introductory guides were often written by folks trying to sell me software. I was lucky to know a few folks in the data community, who helped shepard me to a deeper and broader understanding of privacy. It took a lot of study, time, attention, effort, willpower and several helping hands to get me from curious data scientist to someone who had command of the topics you’ll find in this book.

I strongly believe the skills I teach you in this book are absolutely essential for data scientists today and in the future. The steep learning curve I experienced is unnecessary, and that’s what this book will hep you avoid it. I wrote this book to provide a welcoming, fast-paced and practical environment for you to learn, ask questions, find helpful advice and begin to dive deeper into the challenging topics.

This book is meant to be a useful overview — leading you from zero knowledge to being informed and prepared on data privacy as you perform your work. You’ll learn popular strategies, like pseudonymization and anonymization methods, for bringing privacy into your regular workflow and newer approaches, like encrypted computation and federated data science. If this book acts as a springboard for your academic career or lead you to a research role, that would be terrific. We need intelligent and curious folks working on the unsolved problems in this space. That said, this book is a practical-minded overview, but I will identify ways you can dive into deeper research tpics, and you will then need to navigate that path with the skills from this book as your stepping stones.

Finally, data scientists who need to consider data privacy and security topics as part of their daily work will find this book helpful. There are several chapters that can be quick references for you as you navigate data privacy at your workplace. While a cover-to-cover read will help you build your base knowledge and teach you how to solve new and unknown data privacy challenges, a quick search will provide you straightforward advice on how to manage specific data privacy emergencies that come up in your day-to-day activities.

Navigating This Book

This book is organized into chapters with a practical approach to data privacy and a mixture of theory, exercises and use cases:

- Chapter 1 focuses on data governance and simple privacy approaches, answering questions you might have around how to manage data, track consent, pseudonymize data for internal use cases and interact with remote data.

- Chapter 2 dives into anonymization, covering state-of-the-art approaches that you can use today and exploring the rise of differential privacy as a tool for data scientists at the US Census Bureau.

- Chapter 3 covers how to begin automating privacy in data pipelines and workflows, documenting several use cases around consent, anonymization and data engineering.

- Chapter 4 outlines well-known privacy attacks, including the de-anonymization of the Netflix prize data set and introduces ways you can reason about potential breaches and attacks when working with sensitive data.

- Chapter 5 explores privacy-aware data science and machine learning, introducing practical applications of data privacy for data science projects. This chapter should be used as a quick reference for exploring particular approaches in a project- or product-based data science team.

- Chapter 6 describes how federated techniques work in machine learning and data science and compares those approaches to other privacy-preserving and data minimization techniques.

- Chapter 7 covers encrypted learning and encrypted computation for data privacy in data science, diving into multi-party computation and homomorphic encryption protocols and libraries.

- Chapter 8 works through designing secure machine learning systems, with considerations around privacy, data security and operations. This is another chapter that serves as a quick reference for architects and data science management.

- Chapter 9 navigates how to read and apply data privacy regulation and policies, looking at the GDPR, the California Consumer Privacy Act (CCPA) and internal policy examples to support your path through the legal side of privacy.

- Chapter 10 summarizes frequently asked questions and use cases as a handy reference for Data Privacy emergencies, allowing you to confidently move forward and ensure data privacy is baked into each project and your normal workflow.

The book’s conclusion provides support and motivation for using your newly acquired data privacy skills to push the field and your own path forward!

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/title_title.

If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Book Title by Some Author (O’Reilly). Copyright 2012 Some Copyright Holder, 978-0-596-xxxx-x.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

O’Reilly Online Learning

Note

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

- O’Reilly Media, Inc.
- 1005 Gravenstein Highway North
- Sebastopol, CA 95472

- 800-998-9938 (in the United States or Canada)

- 707-829-0515 (international or local)

- 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/.

Email bookquestions@oreilly.com to comment or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

[1] Disclaimer: I generally avoid predictions, as they are often wrong; however, I am offering this one, based on hard-won experience in industry during the past 6 years.

Chapter 1. Data Governance and Simple Privacy Approaches

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at katharine@kjamistan.com.

Data privacy is a large and long-living field — I want you to picture it like an old road, packed with interesting side streets and diversions, but hard to navigate if you don’t know the way. This chapter is your initial orientation to this road, that has many disciplines and practitioners and a long history. I’ll help you map the important parts of the privacy landscape, you’ll be able to find areas where you want to focus on and reach out to experts for help. With this foundation, you can then start finding the small alleys that interest you which we might consider specializations within data privacy. Applying this map within your organization means uncovering who is doing what, what their responsibilities are and what level of data privacy needs exist in your organization.

It all starts with asking questions: so, what exactly is data governance? You might have heard the word only once or hundreds of times, but it is often left unexplained or open for interpretation. In this chapter, we’ll define places where data governance overlaps with data privacy for practical data science purposes and dive into simpler approaches for solving privacy problems with data, like pseudonymization and remote access. These are very useful, as they will cover the basics you need for working with sensitive data on the job. Think of these tools as the training wheels for your bike. Understanding these topics will help you learn to ride the bike without them later, when more advanced requirements like anonymization or encryption are required.

Being able to speak about governance with privacy professionals is also an unexpected and powerful skill for any data scientist. Understanding why certain data is considered sensitive, knowing some of the legal and technical jargon around data privacy and being able to think of new avenues to address privacy concerns is a superpower when operating at larger enterprises. By the end of this chapter, you will have a better understanding of the needs of the privacy professionals at your workplace, so that you can address their concerns in a meaningful way. Your next compliance meeting will be a piece of cake, so let’s dive in!

Data Governance: What is it?

What do we actually mean when we say data governance? Usually this term is used as a sort of “all encompassing” way to think about our data decisions. But what does the phrase really refer to and how can we make it actionable?

Data governance is literally governing data. But what would that actually look like? Governing is a transfer of rights we individually and communally possess, that we pass onto our elected officials so they can manage things for which we have no time, expertise or interest. Things like running the subway system, ensuring taxes are paid, making a budget, speaking with their international and national counterparts, etc. For data, we also transfer some rights when we hand over our data. We agreed to whatever terms and conditions explained on the website, service or application at the time. We helped produce the data by interacting with an application, site or service. Just as with the legal forms of governance, in data governance, we are residents of a particular state and jurisdiction, and the laws (legal governance) of that area also apply.

Data governance applies to whose data we collect, how we collect and manage it, what we do with it after collection and under what conditions it was handed to us. Figure 1-1 illustrates a map to guide your thinking about the privacy and security part of data governance. In this diagram, we can see the sensitive data inside a tower. The security initiatives are supported by using concepts like Privacy by Design, which we will cover later in this chapter. Regulations and compliance provide a moat which helps us keep our sensitive data separate. Privacy technologies that you will learn in this book are bridges for users of the data, allowing them to gather insights and make decisions with sensitive data without violating data privacy.

Figure 1-1. Mapping Data Governance

Governance is often described as a mix of people, process and technology. Several of these areas won’t be as applicable to your work as a data scientist, but I want you to be aware of their existence. There are likely folks at your organization now (or in the future), who specialize in managing people in relation to governance, who create standards, which turn into policies and procedures; those then need to be implemented through technology. Let’s zoom into the technology section, as this is likely were you’ll be asked to help.

What elements of taking governance standards and policies and implementing them in technology are important for data scientists? In Table 1-2, we dive into the significant areas and related questions within data governance that we grapple with as data scientists.

Table 1-1. Data Governance in Data Science

Data Lineage / Origin | Policies and Controls

Where did the data come from?

What laws or internal policies apply to this data?

Whose data is it?

Where, when and how was it collected?

How did we change it?

What privacy or security concerns do we need to address when using this data?

Did we buy this data or acquire it from someone else and if so, did they document the above?

What was the state of our privacy policy and terms at collection time?

Is the metadata for lineage information easily accessible?

Did this come from a third party and what are their regulations for this data?

Data Reliability / Knowledge

Data Privacy and Security

What are our concerns around understanding our data and our data systems (i.e. including collection, transformation and downstream systems)?

How is access to sensitive data given and managed?

Can we understand how data got somewhere and document it?

Do we understand when we’ve been breached? How so?

When we have data quality problems, do we know how to diagnose them?

Who is responsible for managing privacy controls? Security controls?

Is our dark data non-existent or at least known and understood?

When someone invokes one of their rights (i.e. GDPR), we have a system for this that is well documented and understood.

Is our data well documented and understood? (Know your data)

What Data Loss Prevention technologies and privacy technologies do we use and how?

As you can see, several of these topics are things we are concerned with in our job as data scientists. In actuality, we are already focused on many of these questions since data is a major part of our job. Many of us have suffered from lack of data documentation, incomplete understanding of how a certain database came to be and issues with data labeling and quality. Now you have a new word to call it: governance!

Working on the governance side of data administration or management is really about focusing on how we collect information about our data as we collect it and how we modify and change that information as we transform and use the data. The regulatory, privacy and security concerns help shape those decisions and ensure we are attentive to taking care of the rights of the individuals who created that data. Of course, if your data does not come from individuals, there may be other concerns with regard to proprietary data or related security issues in your area of work.

When we think about governing data in a concrete way, we begin to look at focus on tasks like documenting the ever-changing data flows at our organization? This is where the technical work can get pretty interesting.

Let’s say you have a huge data lake that gets fed from at least 10 different sources, some external, some internal. How can you actually begin to govern that data? What would a solution look like? What happens when the way the company manages those data flows changes? It may be enough to just document the code or the workflows that are active, and to leave the rest for future work. If a workflow stops running, you might want to look at it later and save the documentation somewhere. What do you do with regard to data from partners or other external data collection systems (like Salesforce)? You’ll likely need to coordinate this documentation so that legal and risk departments can use it for their auditing processes.

Documenting this information and related topics, such as quality control and data testing are the important parts of a holistic governance approach. Good news is that there should be other teams and persons interested in helping you!

But first, let’s identify which data is the most important to protect. How can we identify sensitive data? What exactly is sensitive data?

Identifying Sensitive Data

When we talk about sensitive data, we normally mention person-related data. Even then, we focus on things we would call identifiable. Your full name, your email address, your gender, your mailing address, your IP address, your Facebook or Instagram profile, your phone number, your social security number or other national identification number, your credit card number, your birthday, your health records or biometric data (i.e. fingerprints, iris scan, even your gait!).

All of these fall under the category of personally identifiable data, or what many refer to as PII (personally identifiable information). This data is specific to you, it can be used alone or in combination with other sources of information about you to identify you specifically or you alone. This is the most sensitive and frequently regulated data because it provides a nearly-or completely-unique identifier.

As defined in this book, sensitive data includes:

- Personally Identifiable Information (PII): Data that is unique or close to unique for a particular individual

- Person-Related Data: Data that relates to a person but doesn’t fall under PII. This could be anything related to their personhood, including interests, locations, online and offline behaviors and activities—even their beliefs.

- Proprietary and Confidential Data: Data that is for contractual or business-related purposes deemed as sensitive. Its release would endanger a business or other legal relationship or agreement.

One thing I hope this book will do is expand your definition of sensitive data to include a broader range of data. For example, do you think your phone location is sensitive? Or does it depend on where you are? If you are sitting at home, your phone location also reveals your mailing address or residence, which again is personally identifiable. What about if you are at work? Or in a movie theater? Or at a close friend’s house?

And how about your political affiliation and areas of interest? Voting history? Religious beliefs and practices? Your sexual orientation and preferences? How you meet new partners or what is the nature of your relationships with partners you have? What about your daily routines, the news you consume, your music and entertainment choices, the devices you own?

These questions begin to reveal the range of privacy preferences. Some individuals might be comfortable sharing location at work or might even be required to do so. Others might see this as an invasion of their privacy. One person might be quite comfortable sharing location at work (or even required to do so!). Others might see this as an invasion of privacy. Whereas one person might be quite open about their personal relationships, political choices and religion, another person might see these topics as deeply personal and sensitive. This is where regulations can also step in, allowing individuals more choice regarding their privacy preferences and the ability to communication that sensitivity to data collectors.

It also means, however, that when we work with data, we recognize the range of what is considered sensitive. We should be aware of the additional privacy risk we create when we combine a variety of sensitive data that is not personally identifiable but inadvertently link it in a way where it becomes personally identifiable. For example, if I track your location throughout the day, I would likely learn where you work, where you eat, your daily personal activities and where you live. Even if I were to just collect data while you move (i.e. driving data), I would be able to answer some of these questions. Even if I only collected data when you were in the presence of others moving, I could likely still infer things about you, such as if you jog with a friend in the park every morning or if you like to shop at a particular grocery store and at what times.

Similarly, it’s been shown by researchers that a series of Facebook likes can give us a range of probabilites to infer things like gender, sexual orientation, political beliefs—even the marital status of your parents[1]. Of course, these are ranges of probability (we cannot be certain), but the fact that online behavior and social network behavior end up creating both unique breadcrumbs as well as reveal patterns that allow us to infer more personal and private traits of an individual, means we need to treat most person-generated data as potentially identifiable.

As we also know, with the power of inference, we can end up identifying individuals even when that is not our intention. There have been several studies of advertisement targeting, which ultimately sub-segmented groups until only a few persons remained in the target group. Let’s say I am trying to build a Facebook ad campaign and would like to target native Spanish speakers in a particular postal code who also like a particular TV show. As you can likely imagine, the more of these factors I specify, the more I could target a particular individual, even if that isn’t my intention. And if it is my intention (for whatever reason), the information I have that isn’t personally identifiable in theory, but is sensitive can help me figure out how to target that exact person or get fairly close.

For this reason, the term “sensitive data” could mean any person-related data, regardless of if it is personally identifiable or not. Person-related data, particularly in large quantities or in aggregate has a tendency to become identifiable over time. In this book, when I say sensitive data, I am not only referring to PII, but instead a broader range of person-related data, which could be used in combination with other information to identify a person or small group of persons.

A final category of sensitive data is data that is proprietary or confidential for non-person-related reasons. This can be trade secrets, proprietary information for a particular business or product or information that falls under confidentiality clauses either when or after it was collected. This could be data that is shared between parent companies and their subsidiaries, which must be kept inaccessible due to internal policies or confidentiality agreements. Or it could be sensitive internal data that, if leaked, would give competitors an edge or compromise the company in another manner. This type of sensitive information would also benefit from several of the approaches and treatments we’ll learn in this book.

Note

Note that I am a supporter of whistle-blowing and that if you have data that you believe would do the public good to release, that you think about techniques you have learned in this book as ways to enable the release of that data in a responsible and thoughtful manner.

As we’ve seen from these examples, any person-related data can also be seen as falling under need for privacy protections.

Identifying PII

PII falls under a particular legal category for most data protection regulations, which means it requires the most attention when identifying sensitive data as data governance is implemented within an organization. If your organization collects any personal data—even for employees—then this data often has special governance requirements. Frequently, there is a lack of documentation or categorization of PII because it is contained within text files or other unstructured data and, therefore, poorly documented—or not documented at all.

If this is the case, there are several tools built explicitly for PII discovery using a variety of methods. I’ve also seen teams successfully build their own tools and systems for PII discovery. These tools often use fairly brittle methods like regular expressions (which are strings used to match patterns) or tracking string entropy (for finding things like Application Programming Interface (API) keys or cryptographic keys or passwords). I have also built performant deep learning models to identify PII in message text, with some success.

Warning

PII Discovery is never perfect, and likely never will be perfect. It’s important to talk with your risk teams (privacy, legal, security) about the fact that there is no silver bullet, whether you purchase a PII discovery toolkit or build your own. For this reason, it’s always a good idea to treat human-generated data as sensitive, regardless of what “cleaning processes” it has undergone.

If you are working with a backlog of undocumented data and you fear there is a lot of PII contained therein, you suspect it contains a lot of PII, take a look at an easy-to-use open source tool. After you see how far that will take you, then determine whether you need to invest in a more advanced or expensive approach. OAs I write this book, I can recommend one library—Microsoft’s Presidio, which also includes some basic pseudonymization techniques covered later in this chapter.

The best approach to managing and keeping track of PII is to actually track the PII as it comes into your system and to have data expertise at your organization help label and manage this data as it comes in, so that you don’t need to aggressively search for it or discover it later in the process. One of the ways you can start the habit and culture of detecting PII early and often is to build a culture of documentation around data collection and data use.

Documenting Data for Use

As discussed, documenting your data—sensitive or otherwise—is an essential part of data governance. Use a multi-faceted approach to documentation. This allows you to cover the range of data quickly—especially important if you are starting from scratch.

In this section, we’ll walk through ways to think about data documentation—how to find undocumented data, how to begin basic data documentation, adding documentation for lineage, collection and processing information and finally data version control.

Finding and Documenting Unknown Data

Unknown data (sometimes referred to as “dark data” or “data blindspots”) are gaps in data documentation or even basic knowledge and understanding of the data. These occur at large organizations due to lack of best practices, usually over the course of many years. Data from long ago applications or products that may or may not even exist along with data that was amassed via acquisitions, partnerships or even purchsed data are often found in undocumented databases or files. Sometimes this data is in active use, but no one knows how it got there or when it was collected. Other times the data is discovered and the company would like to figure out if they might use it, but are unclear on the origin. These types of undocumented data can also be the result of experimentation or workflows from data engineering teams or data management teams where the folks who knew about the origin are no longer at the company or where the data was never documented and now no one can remember where it came from.

When dealing with unknown data situations, it’s important to establish a process and routine, so as to prevent the data from sitting around even longer without documentation. Here is an outlined process you can use to investigate, document and determine a decision when you find dark data. Feel free to modify it to fit your needs by adding steps or clearer requirements for each stage, such as the specific location and technology that should be used for each stage.

1. Investigate potential provenance. Does the data look like any already documented data at the company? Is the data easily found by searching a search engine (i.e. publicly available data)? Does anyone on a related team know the data or where it came from?

2. Dive into Discovery. Can you find out more about the data via investigation and inquiry? Investigate all potential sources and connect with other teams and departments. Someone might know the origin of the data. Look at the contents of the data for potential clues and document your findings as you go.

3. Determine sensitivity. When looking at the data, does it have person-related or personally identifiable information? Can you determine the date it might have been collected by looking at any related dates in the database or document? If person-related, what would your company’s privacy policy say in relation to the data you found?

4. Start Documentation for Consumption. How can you build documentation people will read and use? To start, document where the data was found, what it includes, what you were able to find when asking about provenance and the sensitivity levels. Documentation should follow standards your organization has and should be discoverable and accessible. At this point, you may want to involve data management decision makers to determine next steps. Continue to inform them as you move through the process.

5. Delete, Archive or Maintain? What are the next steps for the now known data? To decide, involve interested parties, including compliance and audit departments if they exist at your organization. If the data isn’t sensitive, doesn’t relate to any proprietary details or persons, and is useful, you can likely just integrate it. Ensure the documentation is shared and updated and move on. Otherwise, you might choose to archive the data until more details become available. This is also a good option if the data isn’t useful, but you would like to wait a given time before deleting it. Finally, data minimization standards for data privacy recommend deleting the data, particularly if it contains personal details and you were not able to ascertain under what circumstances it was acquired. You should still document this decision and the data before you delete it, so if someone later figures out the data existed, there is an audit trail. I would almost always suggest deletion if the data appears too old to be valuable for the data science or business teams.

There are several products that help teams find unknown data, if you are particularly concerned about its existence at your company. These services often provide scanning software that looks at servers and attempts to find data where you expect none. That said, if you are properly documenting your data processes and are working closely with other data teams, it’s unlikely that data will just sit around unused and undiscovered.

One common issue with unknown data is that it is historical reporting data, often collected and used by business decisions makers before data science was established at the company. If the data science team or focus at your company is fairly new, then it might be worth investigating reporting data used by business units that falls outside your normal data collection mechanisms. For example, there could be many spreadsheets or other types of document-based reports that were used for years before there was better data available on customers or products. There could also be integrations with tools like Salesforce or other external software that pull information and place it into internal servers or file servers. Familiarize yourself with these if you join a new team concerned about undocumented data sources.

Sometimes these practices have developed due to Shadow IT, where more sensitive data is copied to many locations because of access restrictions. Shadow IT is a term used to describe processes outside of IT leadership or purview—often created as helpful shortcuts—that frequently lead to security and auditing nightmares. Sadly, this process is commonplace, because humans spend lots of hours, days and weeks waiting for access to be granted. Then, they immediately copy the data or develop some automation to do so, so that they don’t ever have to wait again. Part of your job as a privacy-focused data person will be to identify these practices and build better privacy technologies to automate future access. Replace Shadow IT with transparent, easy-to-use, well-documented and privacy-preserving access systems!

You may encounter a variety of reactions to your search for unknown data as well as your recommendations to delete it. Some folks have an idea that data should be saved, no matter whether it is useful or helpful. Data hoarding is a big problem in a variety of institutions, and your company might be one of them. Rather than tackle the cultural and communication problem alone, educate decision makers about the risks of harboring unknown data. Spreadsheets full of customer information are valuable assets, that should either be documented and managed by capable data and security teams following best practices, or that should be deleted so as to not become a target for internal or external security threats. If the reports or documents are useful and answering good questions for the teams that use them, move the data into the light, keep it documented and audited. You may get better insights by doing so!

Basic Data Documentation

Data needs to be documented, based on how it will be consumed. When you document data, think of your readers, as you would if you were documenting code. What is this reader going to understand? How will they find and access the documentation in their normal workflow? How will they search through the documentation? What words will they use? What is the most important information? How can you make it concise enough that they will find it? How do you make data more self-documenting and easy to update?

Although the topic of data documentation is not new, it’s also not a widely used practice, especially within data science teams. Data science practices have shifted from research-oriented teams and analysis-driven dashboards to experimentation, failures, agile development and deployment standards. This makes it even more important that we understand the reason and the processes to document data.

Like data and experiment version control, documenting data enables other teams to discover and utilize data sources that might have been obtuse or hard-to-find if not properly documented. In many organizations, however, data sources are often disjointed across departments. It can be challenging to get the fundamentals right, yet make make the data consumable.

Here are some ways to convince data management or business units that data documentation is worth the extra time and effort. Data documentation:

- Speeds up data experiments, which can lead to new data-driven insights and discoveries

- Enables cross-department and team collaboration

- Accelerates data access and use for all stakeholders

- Helps eliminate unknown or undocumented data

- Empowers data teams when deciding which data to use for new ventures, products and models

- Signals to product teams what data is available for new ideas

- Shows analysts disparate datasets that could be used for new insights and reports

- Gives compliance and audit teams proper oversight and assists in new data security and privacy controls

Data governance within your organization will only work if there is functional and effective data documentation. This might even integrate into whatever the identity management and access systems to help data administrators, data owners and data management folks to properly allow or deny access to data based on the documentation.

Tip

To empower a responsible AI driven organization, you’ll need some extra documentation to communicate any stereotypes or known biases in the data itself. In this case, I recommend taking time to read about Data Cards, which is a system developed by several researchers as a way to document data for AI use and consumption by non-technical users. If you’ve heard of Model Cards, then Data Cards is what came first and inspired several of the recommendations for model documentation. If your team is focused on consumer-facing machine learning systems, I recommend using both to ensure you are providing accurate, fair and reliable systems!

The documentation can collect information on any or all of the following:

Documenting Data Collection

Explanation and Related Questions: Who, where, when, why and how was data collected? A description of when each dataset was collected, what team or software managed the collection, any post-collection processing that was performed as well as why the collection happened (i.e. under what circumstances and legality)

Example: A diagram showing the data flow, with documentation on when it was implemented, by whom and a data snapshot of the data at that point in time. The diagram should also contain consent information or other information regarding legal and compliance questions.

Documenting Data Quality

Explanation and Related Questions: What standardization has the data undergone (if any)? What quality controls were established? How many null values or extreme values are present in the data? Has the data been checked or processed for duplicates or inconsistencies?

Example: An analysis of the data quality in the dataset, including the frequency of null values, the standardization and harmonization of values (i.e. values that are changed to percentages or that are converted to a standard unit), as well as the data bias and variance. A deeper investigation here could show histograms across numerous dimensions, visualize data, show covariance or correlation between attributes as well as information on outliers or extreme values. Should the data shift significantly in a short period of time, there could be monitoring for analysis and alerting or exploration into the cause of these shifts.

Documenting Data Security

Explanation and Related Questions: What level of security risk applies to this data? What measures should be taken in use or access of this data? A risk analysis on the sensitivity of the data and its infrastructure, architecture, storage details to help other teams (i.e. security and operations teams) assess risk modeling and make suggestions on access restrictions. This can also include privacy-preserving mechanisms or functions that the data has been subject to during collection and processing as well as access and query restrictions on the data itself. If you’ve already treated the data or removed identifiers, you should have a description of what was done and in what order, so data consumers know how to work with the data intelligently. I’ll discuss this more in later chapters-including my recommendation to never drop identifiers unless other measures are also taken!

Example: A risk analysis chart for the possibility of sensitive data leakage due to dataset variables, analyzing the likelihood of the privacy risk and protections taken should the likelihood be medium or high.

Documenting Data Privacy

Explanation and Related Questions: Does the data contain person-related information? If so, what privacy by design and other legal or data protection requirements have been performed to ensure the data is properly handled? If PII or person-related data is stored, the documentation should include information about the jurisdiction of the data and what privacy policies or other documents were shared with the users at the time. There should also be clear documentation on any and all privacy-preserving mechanisms performed on the data with the ability to look at the processes (in code if possible, with a commit hash!). Finally, there might need to be particular access restrictions or other types of functionality available depending on the privacy policy or regulations in that jurisdiction. Ensure that this data is treated as you would other sensitive data and that these measures are adequately documented. This section should include related consent information or where to locate consent information.

Example: A list of columns that contain person-related data along with a timestamp column (even better, an automation) declaring when data should be deleted.

Documenting Data Descriptions

Explanation and Related Questions: What is the organization of the data? If tabular, what do the column names mean? What data types are there? What units of measurement are used? Clarify the meaning of jargon or codes in the data (i.e. shorthand or internal mappings). A description of the data fields, column names (if any), keys and values, codes, units of measure and other details that help a new consumer of the data understand what they are using. This is typical documentation that you might already have stored somewhere, either in a sheet or in the database or data document itself. If particular formatting standards are chosen (such as ISO date representations, 24 hour time or other standard units, especially to standardize the data across multiple jurisdictions, please include details on how this processing is done in case others need to find potential errors or if processing has changed at any point in time!

Example: A list of columns and data types for each column, with categorical column codes listed in a tabular format for easy lookup.

Documenting Data Statistics

Explanation and Related Questions: What are the standard descriptive statistics of the dataset? How is the data distributed across particular features of importance? Are there particular areas of note or concern around data biases or unequal classes? Summarized statistical descriptions of the data can be included in written and/or graphical forms to allow others to quickly assess if the data sample will properly address their questions. Here, an interactive experience goes a long way in answering the questions, but some basic summaries can help new users get started. These properties can sometimes be quite sensitive, so you should expose them only after assessing the security and privacy risks of giving access to that data. One great tool to have a look at for inspiration is Google’s Facets.

Example: A chart showing the percentiles of numerical columns in comparison with one another (using boxcharts with outliers removed, for example). These could be toggleable based on other features which allows you to analyze encoded biases in the dataset itself.

This list is not all encompassing, but it can guide you as you begin to address data documentation at your organization. Remember, this documentation is not for you, but instead for the data consumers (presuming you are not also the data consumer!). Figure out language, visualizations and descriptions that work for your team(s) and organization(s) so the end users are able to properly find what they are looking for and easily use it. Feel free to experiment with what works by doing smaller tests and documentation and getting feedback. Pair with a user experience or product expert for help. Only standardize and roll out full data documentation when you have a system that is working and one which you maintain and improve over time. Remember: documentation that isn’t accurate is often worse than no documentation at all, because it gives people a false sense because your users think they understand it, but they don’t.

At the end of the day, just like with code and architecture documentation, your organization and data users will reap the rewards of well-documented data in languages and terms that make sense to them. This will expedite projects, standardize how the organization works with data, help clarify data lineage and quality for easier decision-making. It will also bring privacy and security questions to the forefront of data decisions, where they are most effective!

Tracking Data Lineage

Data lineage (sometimes called data provenance) is a way of tracking where the data came from, how it got there and what actions have been performed on the data since entering the system. As you can imagine, this information is extremely useful for data scientists clarify data quality and the content of the data, as well as the utility of the dataset for our needs.

Lineage information helps you answer questions like:

- When was this data collected? From where?

- How am I allowed to use this data? What consent information is documented?

- What processing or cleaning has this data undergone (i.e. removal of nulls, standardization of units, etc.)?

- Where else is this data or related data (i.e. data with more or less transformations and processing) stored?

- What should I keep in mind about the quality and origin of the data?

Unfortunately when companies originally developed their data infrastructure, adequate systems to track data lineage were not often embedded. The focus tended to be on ingesting the data and storing it away as efficiently as possible—not on tracing it and determining how it was processed. Sometimes this data was transformed so it could be used, sometimes not. Technological debt builds up around data systems. However, just because there isn’t data lineage information for older data, that does not mean you cannot start now.

Depending on how advanced your data infrastructure and engineering systems are, there might already be great places to begin documenting lineage. This information can be pulled from systems like Apache Spark (or Beam, Flink, Kafka, Airflow) or other pipeline automation systems and integrated into whatever documentation or tracking systems you are using. If you aren’t sure if tracking is going on, it’s a good idea to connect with the teams who manage data catalogs and data schema, as they are often stored in those or similar systems. If your company doesn’t currently catalog data or track lineage information, then you should coordinate a group of data folks and determine the most light weight approach to getting started.

Sometimes we face a problem as data scientists that makes us question the validity of the input data. Only via proper lineage or provenance tracking can we ascertain if errors were made in collection or processing. These errors could arise from a simple application change that changed the units of measure or inserted a bug into the data collected. Knowing when and where it happened can help us (and other technical folks we work with) determine what might have caused it and whether anything should be fixed. Tracking origin and processing is essential when maintaining streaming or more real-time systems, as errors in the data collection and transformation can quickly propagate into models or other downstream products.

Note

There are a lot of cool tools around data documentation, lineage and version control. This means that, hopefully, you can find a good set of tools or even single tool to help manage a lot of the governance concerns that your team will have. There are always newcomers, but some tools I’ve seen teams enjoy are DBT, CKAN and Tableau (with their Data Catalog).

To manage changes in data flows, I recommend using tools like Great Expectations, which can be used to test data mid-stream and determine if anything has markedly changed. Great Expectations allows you to write what I like to think of as “Data Unit Tests”, to assert that the data meets your expectations. For example, you may want to test if a particular value is not null, or that a value is above or below a certain number, that a date string has been properly standardized, or even that a value is a string or an integer. These tests, when used properly by data teams, can immediately alert you to the aforementioned bugs. This allows you to understand these changes as you work with the data.

In addition, compiling and documenting consent as a part of your data collection is a great way to ensure you are following the best privacy practices and incorporating Privacy by Design standards [2]. If you are working across multiple jurisdictions, you might also want to treat data from these areas differently. For example, if you have a group of application users from Europe, and you’d like to track GDPR consent management as you collect their data, you could incorporate that data into the data itself. This would then be an optional field in your data structures that allows you to see the opt-in and opt-out controls set by the user (should you have those in your application!). Data coming in from other jurisdictions would simply be missing these fields—you might want to standardize passing it in and recording it as null. This allows you to sort your data later and ensures you are using only data with proper opt-in for experiments that require consent.

In addition, there are more data sovereignty legislations and policies in development and use than ever before, and my guess is this will increase in coming years. These laws focus on keeping residents’ data inside particular jurisdications (such as keeping EU Resident data housed within European data infrastructure). Ensuring those sovereignty controls are properly tracked and in place can also assist your legal, compliance and security teams in validating that the data infrastucture is doing The Right Thing.

A full tracking pipeline could look like Figure 1-2. First, the user is presented with an interface with fine-grained privacy preferences explained in clear language. As the data is collected, provenance details like where, when and how the data was collected are added as normal fields (not attached in a separate JSON dictionary that no one looks at). The data enters the normal cleaning or transformation pipelines. That data is then ready for data quality analysis and ensuring the data meets other governance standards. Sensitivity of the data is also analyzed at this point in time in a semi-automated manner[3]. At that point the data hits storage and the attached data is placed in the appropriate linkable data structures. This additional governance data is easily referable and remains current and attached to the user data until the user requests deletion or the data expires and is removed from the system.

Figure 1-2. Lineage and Consent Tracking Pipeline

Tracking data lineage and consent adds overhead now, but helps debugging and speeds up access to data later. If you work closely with a data engineering or infrastructure team, talk with them about sharing the workload for setting up and maintaining your lineage information. And just like your data documentation, ensure it’s readable, interpretable and usable for the data consumers. If it sits in a file that no one ever looks at, it’s not worth producing and maintaining in the first place! Meet your data users where they are and take time to train and upskill folks so they can independently use the information for their daily work.

Data Version Control

When we work with data daily, we understand incremental data changes occur. If we have a very large dataset, then, usually, these changes are not noticeable and they don’t affect our workflow very often. However, when we makesignificant changes to our data collection and transformation—or when there are external forces at play (like a global pandemic)--these incremental changes have larger effect and can change normal workflows, analytics and modeling.

This is when data versioning comes into play for all data science. Data versioning is the ability to create versions or checkpoints for your data tied to a particular time. Similar to code version control systems like git, data version control allows you to monitor changes by creating a snapshot or “commit” at a particular point in time. This can then be compared with earlier or later versions to understand how the data changed.

As we’ve seen with agile software development practices like testing, continuous integration (CI), continuous delivery (CD) and software version control, data version control benefit data scientists. One major win is the ability to debug how a change in data might have affected a particular downstream process or model. Being able to pinpoint when a change happened can help solve bigger problems, like diagnosing why a model isn’t performing as expected or why a particular report or analytics tool stopped working. In addition to understanding and debugging errors, data version control offers important information for modeling and understanding your data.

Knowing how the data you collect changes over time can be a superpower for understanding the behavior or systems you are modeling. How are the users of your application changing over time? How are your experiment results shifting? What assumptions did you have about the data originally and how have they held up. Essentially, a lot of questions we ask about the data can benefit from a periodic review to understand the data itself and changes over a given period. Finally, data versioning is great for data privacy and trustworthy AI practices. When we can answer what data was used for a particular model exactly, when we can pin a before and after point to when a person requested their data be removed, when we can transparently demonstrate the changes we have made to preserve privacy, we can ensure the processes we are implementing are working. Data versioning supports each of these actions and helps ensure that we are clear about the outcomes of the data privacy measures we are implementing. Then we can debug them should anything go wrong!

So how do you start versioning data? There are several tools available now and sure to be many more launched in the coming years. Here are some questions you might use to help you evaluate what versioning tools are appropriate for you:

1. How does the tool manage snapshots and checkpoints for the data? This should be done in a way that is both programmable and well understood. You don’t want to be reading through documentation when you need to recover data after a broken pipeline!

2. Are they memory efficient? How will you manage space? A fairly naive way to manage data would be to copy your entire database every day and save an old copy. This is great if you have unlimited memory and space and only a few hundred rows. This is not the situation for most of us, so you’ll want to evaluate how many snapshots you save, how much extra space and processing power those will take and when to delete old snapshots.

3. Will this integrate well with other teams and their workflows? As with all software choices, you should ensure that what works for your team is a good choice, also for other teams that work closely with the data. Checking with the data engineers and software engineers can help ensure they also know how to “recover” data after deployment problems and answer questions should anything on their end go awry. You should ensure that they understand both the API and programming language and how the tool is used. You may have to update documentation or even provide training to the other teams.

4. Can I create or save common workflows or debugging steps? A tool is only as good as your understanding and use of it! For this reason, I would recommend saving a sprint (if you work in an agile method) or a small chunk of time and energy to have your team play with the tool (or tools if you are evaluating a few) and write up a few example stories to see if you can pre-program some of the use cases you foresee. How would a recovery of data before a schema change work? How could you answer what data was used for a particular model that was trained and deployed? What if a GDPR request for data deletion comes in and we want to prove the data was removed? These are some examples, but spend time with the data team to make a comprehensive list and ensure that the use cases you want to use it for are well understood and maybe even pre-scripted!

5. Can this be used in conjunction with data lineage efforts to better select data for particular use cases? As we talked about earlier in this chapter, understanding lineage can help determine if a particular dataset is well suited for the task at hand. This information, combined with version awareness can speed up the type of modeling and experimentation your team works on by increasing the data understanding and allowing for shortcuts to get the data that fits best as soon as possible.

Data Versioning is closely related with model versioning. The prior questions can also be asked in relation to model versioning tools, and some available open-source libraries today do both. Whatever your plan is and however your team works best, think about incorporating versioning for data and models into your workflows. These practices are well understood in software and can help make the data science work in your organization more predictable, error free and better understood across the engineering department. Setting them up now and having them evolve via usage and growing data governance is something I would recommend any and every team, even if the first few years are a learning process!

At the time of this book publishing, one of the prominent libraries in data version control is DVC (Data Version Control). It currently supports a git-like interface for data and model versioning and has its own workflow interface for running and managing data pipelines.

There is also growing support for version control of datasets directly in data lake and data warehousing tools. If your organization is already using large data management tools, it’d be a good idea to first check if there is some support or integration before introducing another library.

As you are likely aware of, the data tooling landscape changes quickly! Having a look around at what new tools are available and evaluating several before deciding what is best is always a good idea! Use these questions to guide your evaluation and selection criteria and don’t be afraid to do a few proof-of-concept implementations before setting up one as the standard.

Basic Privacy: Pseudonymization for Privacy by Design

You’ve learned about what data governance is, working with experts who help manage data protection and security policies as well as ways to document data, ask useful questions, determine what is sensitive and reasoning about privacy and security needs of the data. Combine data knowledge with contextual knowledge to deploy governance initiatives. It’s time to use contextual knowledge to fashion privacy technologies and mechanisms for sensitive data.

Implement privacy mechanisms by starting with the basics. As you’ll learn throughout this book, the simplest approach can solve quite a few problems and address numerous internal and external concerns. In evaluating risk and reward, you’ll be looking at which privacy technologies and techniques are appropriate for the risks at hand and what are the rewards (for example, getting access or expanded availability of the data).

For those moments, pseudonymization is a great match for privacy basics. Pseudonymization is a technique that allows us to use “Pseudonyms” instead of real names and data. There are several approaches to pseudonymization, outlined below.

Table 1-2. Approaches to Pseudonymization

Pseudonymization Approach

Description

Example

Masking

Applying a “mask” to the data that often replaces values with a standard series of values.

888-23-5322 → <ID-NUMBER> or <XXX-XX-5322>

Tokenization (Table-Based)

Replacing identifiable tokens via a look-up table that allows a 1-to-1 replacement.

Freddo Bamber → Fiona Molyn

Hashing

Using a hash mechanism to make the data less interpretable, but still linkable.

foo@bar.com → 32dz22945nzow

Format-Preserving Encryption

Using a cipher or other cryptographic technique to replace the data with similar data. Often this is also linkable.

(0)30 4344 3333 → (0)44 4627 1111

As you may already notice in Table 1-3, these approaches can significantly vary the quality of your data as well as the privacy of the individuals. For example, the hashing mechanism takes what is easily interpreted as an email address and turns it into something that is no longer interpretable. This provides some minimal privacy but also destroys our ability to extract useful information (such as linking accounts that share domains). The masking mechanism can often remove all identifying information, or sometimes leave enough data that it can be linked with other datasets and the identity of the individual is revealed (if, let’s say the full ID numbers are also exposed another way). The table-based tokenization means maintaining a solution that may or may not scale with your data, but allows linking which is a benefit to data scientists who might need to connect disparate datasets.

Linking, however, is a primary attack vector to determine the identity of an individual. The more data we are able to link, the easier it is to use that linked information to infer who the person is or to learn enough about them to make a good guess. The format-preserving method allows this linkability, but is more scalable due to using a standard two-way mechanism (often based in cryptographic techniques).

Based on this analysis, we have several benefits and drawbacks of using pseudonymization in our data science illustrated in Table 1-4.

Table 1-3. Benefits and Drawbacks of Pseudonymization

Benefit

Drawback

Linkable: Pseudonymization techniques often retain the ability to link data, which helps when we are connecting datasets using personal identifiers or other sensitive columns.

Linkage Attacks are a prominent and consistent threat to privacy protection, and become easier the more data is available (read more on these in Chapter 4)

Format-Preserving Utility: Several pseudonymization methods allow you to preserve format or learn the original intention for that data (i.e. is it an email?). This can be helpful if you have questions about the original data source and makeup.

Any information included in the pseudonymized data is already more information and more risk should the data get released or accidentally exposed. Would data documentation be a better way to understand the underlying schema?

Privacy-By-Design Technique: Pseudonymization is a technique proposed by frameworks like Privacy by Design to support more privacy than using the raw data, especially in contexts where the raw data is sensitive.

Basic techniques like pseudonymization can create a false sense of security, making persons more likely to share the data more widely or to claim it is “anonymized” because personal identifiers are removed or pseudonymized.

In my experience, the biggest argument against using pseudonymization would be that it creates a false sense of security and privacy. I have seen teams struggle with privacy solutions and determine that pseudonymization is good enough because it “anonymizes” the data. This is unfortunately a widespread myth. We’ll review what anonymization is and isn’t in Chapter 2, but I can guarantee you that no amount of pseudonymization will bring you the anonymization you seek, should that be the recommended method.

However, if you can guarantee the data will only be used internally by a small group of individuals who may require privileged access, then pseudonymization might be a good fit for your needs. One use case might be for internal sales or customer support teams, who need to see some details for the customers, but likely don’t need access to all of the information. Another could be for internal business intelligence (BI) and analytics dashboards, which might want the ability to link data but should not have direct access to the sensitive values. However, some teams suggest alternatives to pseudonymization that allow aggregate values at the SQL level which might address the same privacy concerns. For example, you could imagine that a Business Intelligence Dashboard reporting orders across geographies would only need aggregated queries for large-enough geographies to ensure some privacy is preserved. Or you can imagine a customer support system that doesn’t expose fields that are not necessary to perform the task at hand.

I’ve often seen pseudonymization used as a mechanism to extract data from production systems and use it in testing environments (for software development and analytics tools) or to log potential errors in a secure system. This has the benefit that the data is “close” to the data seen in production, but is not the actual raw production data. You’ll want to talk to security professionals at your organization to determine what fits for your risk model (more on this in Chapter 4). As you can imagine, having this production data in a testing environment is risky, since the data is only weakly protected and test environments are often insecure. That said, pseudonymization is one small privacy measure better than taking an actual production data dump and using the raw production data as test data (or for analytics data).

If pseudonymization is recommended by your internal stakeholders and you have also deemed the risk low enough for its use, there are several tools and libraries to evaluate as you get started.

I’ll walk you through an example workflow using Hashicorp Vault. Hashicorp Vault is a microservice used by infrastructure teams to manage secrets across applications. This is a common pattern for Kubernetes and Docker-based setups, where many applications and services are deployed in containers and need to access sensitive data like API keys, encryption keys or identity details in a scalable and secure way. After Vault is installed and in use on your infrastructure and properly set up to allow for transformation of secrets, you can use it to safely hold secrets and allow access to them with proper identity and verification services.

To use the format-preserving mechanism, you would first build a regular expression pattern for the format you need. Here is an example of a regular expression for a credit card number:

\d{4}-\d{2}(\d{2})-(\d{4})-(\d{4})

You would register this pattern as a template for a particular transformation in Hashicorp with a set of roles attached (i.e. who can use this transformation). Hashicorp already supports certain types of format-preserving encryption. Note: some of them are reversible and some are not!

You would then test your transformation by using the command line interface (CLI) and seeing that a fake generated number comes back as a different, but still valid credit card number.

$ vault write transform/encode/payments value=1111-2222-3333-4444

Key Value
--- -----
encoded_value 1111-2200-1452-4879

If you chose a method that is reversible, you can also check that you can properly decode with the proper role and permissions.

$ vault write transform/decode/payments value=1111-2200-1452-4879

Key Value
--- -----
decoded_value 1111-2222-3333-4444

Note

The API calls may change and need to be cleared with the infrastructure support running Hashicorp for your organization. I recommend taking a look at the latest Hashicorp documentation to review if they are the right solution for your pseudonymization needs!

There are several open-source libraries with format-preserving encryption support as well as other pseudonymization methods (such as hashing, masking or tokenization). Here are a few that have useful documentation and functionality at the writing of this book:

- KIProtect’s Kodex provides an open-source community edition that has several pseudonymization methods.[4]

- Format-Preserving Encryption Python library by Mysto allows you to set up several format-preserving algorithms with an easy-to-use Python interface.

- Microsoft’s Presidio allows for several masking and tokenization options as well as measures to discover PII in text.

- Private Input Masked Output (PIMO) uses a Go-based engine and has many templates to pseudonymize and mask data.

As you explore more of the advanced techniques and potential threats and attacks in this book, you’ll develop a better understanding of what risk level and ease-of-use fits your team and your work. In doing so, it will become clearer to you when pseudonymization fits and when it’s best to approach the problem with a more advanced and protective technique.

Summary

This chapter outlined what data governance is and how you can and should use it as a data scientist. You learned more about how to work with legal and security professionals, who will likely have their own ideas of what data privacy is and what concerns the organization has with regard to privacy. I highlighted important avenues that link governance to data science and data privacy: like managing data documentation and tracking lineage. You also learned how to identify sensitive data and some basic privacy-aware approaches for working with it, like pseudonymization and remote access.

You should now feel like you are getting your foot in the door. You are building your mental map of privacy, who can help you and what is important. You may already have some questions about how you will practically identify and manage privacy risk or how you will ensure that you are eliminating or mitigating them effectively. I have some good news, the next chapter helps you think about these questions using math and probabilities! Onto differential privacy!

[1] This work was published by some of the researchers who later worked on Cambridge Analytica. See: Kosinki et al., Private traits and attributes are predictable from digital records of human behavior, 2013.

[2] Privacy by Design is a set or principles outlining measures that technologists can use to ensure that systems are architected and software is designed with privacy in mind from the beginning. You will hear it used quite often in conversations with governance experts who have worked in the field for a long time. I recommend taking time to read and explore these principles, which are quite high-level and determine how they fit your data work. For the original publication, please see the IAPP copy of the principles.

[3] This may need to occur as the first step depending on the sensitivity of the data and the logging included in the transformation steps. Please make sure you are never logging sensitive data!

[4] Disclaimer: I co-founded KI Protect and worked on the initial implementations of this library; however, the company now runs without my involvement and I am no longer a contributor to the library.

Chapter 2. Anonymization

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at katharine@kjamistan.com.

In this chapter, we’ll dive into anonymization: what it is, how to use it and what privacy technologies exist for data science use cases. You may already know about anonymization but this chapter might challenge or contradict what you think you know! The topic of anonymization is one that has plagued researchers and scientists for decades. In this chapter, we’ll use strong, rigorous, scientific definitions of anonymization. This will help you approach the problem with state-of-the-art technologies and give you tools that empower you to meet strong privacy protections while still performing accurate data science.

What is anonymization?

To anonymize is “to remove identifying information from (something, such as computer data) so that the original source cannot be known” [1]. When we want to anonymize personal data, we want to make sure the data cannot be traced back to a particular individual. But how exactly might we ensure that?

In the past, there have been several approaches, most of which have been debunked and replaced with newer understandings. Some of the initial approaches (categorized under Statistical Disclosure Control) used a variety of older and no longer recommended methods like suppression, aggregation and transformations to anonymize the data. These methods were like early cryptographic ciphers. They obfuscated the source, but a dedicated attacker would be able to infer or extrapolate more information.

Similar to the developments of cryptanalysis in cryptography, new privacy attacks were developed over decades that highlighted the weaknesses in these basic approaches. You might have come across “anonymized” data that had undergone techniques like:

- Removal of particular columns or attributes (i.e. drop the names, addresses, birthdates, etc.)

- Aggregation of certain row attributes (i.e. “5 people in this dataset live in this postal code”)

- Flipping or other transformation methods to obfuscate individual characteristics (i.e. mixing the row attributes from multiple rows so that no one row has the original values, but the overall distribution of values is unchanged)

Each of these are interesting obfuscation techniques, from a security standpoint, but unfortunately they cannot and do not guarantee privacy.

Cynthia Dwork and several of her peers helped debunk this type of data anonymization by proving that release of any data has a potential for a re-identification attack or information gain.[2] This is illustrated in the seminal paper via an example. Let’s say that you know that a person is 2 cm taller than the average height for people in a particular country. Then, a government agency releases the average height of a large group of people in that country. Now, you know this person’s exact height. They don’t even need to have been in the dataset! The essential lesson here is that we cannot know what an attacker already knows about an individual or a group of individuals (often called auxiliary information). Therefore, we cannot guarantee that a release of information, no matter how obfuscated, will keep a person safe from identification or won’t reveal some information about them.

You might be asking, if so, then what is even the point of anonymization? If there is no guarantee that an attacker cannot learn something, then why do we even try? Even though we cannot guarantee an attacker will learn nothing, we can start to think about how to quantify the amount of information that can be learned. This is what Dwork and several other researchers began to develop after proving statistical disclosure control was a myth.[3]

The problem at hand shifted from “How do we anonymize data?” to “How do we measure the loss of privacy via information leakage”? Their goal was to find ways we could measure things like privacy loss in a data release, rather than guarantee it would never happen. This meant shifting from a binary understanding of privacy—as “on” or “off"--to a gradient of privacy. This is an important mental shift for you as well. The only guarantee of complete privacy is to delete all data or never collect it in the first place. While I’m a big fan of minimization, I am also a pragmatic data scientist. We need data to solve problems. Thinking about privacy as a range can helps us clarify our needs. How much privacy loss am I willing to create in order to answer this question? How much privacy can I offer and still get reasonable answers to my questions?

Figure 2-1. Information and Privacy Continuum

To help you reason about these questions and begin thinking in terms of privacy gradients, I’ve made a small diagram (Figure 2-1) showing the relation between privacy and information. On the far left, we see absolute information, which means zero privacy (if the data is private to begin with). On the far right, we have absolute privacy which also means zero information—all data is deleted. Along this continuum, we can choose our use of privacy technologies to provide reasonable privacy guarantees and determine an adequate balance for our analysis.

Differential privacy can help us measure information content and try to find a reasonable balance between the problem we are trying to solve while still providing individual privacy guarantees. Let’s dive into exactly what differential privacy is and is not, so you can begin to imagine your choices along this continuum.

Defining Differential Privacy

Differential privacy is a condition we need to meet in order to release information but still calculate and guarantee, in a worst-case scenario, the amount of privacy loss for any individual within the dataset. To do so, we build algorithms (called mechanisms) that help us release enough information to learn something from our data without revealing too much about any one individual.

The original definition of differential privacy describes two databases
D 1 and
D 2 with one row difference between them. Either a row representing one person has been added or a row has been removed from the first database to produce the second one. The goal is that a motivated and informed attacker cannot obtain too much information about the person whose data has been added or removed. We use an algorithm (
A to release the query response, which we want to ensure satisfies the following definition:

P r
[
A
(
D 1
)
∈
S
]
≤ e x p
( ε
)
×
P r
[
A
(
D 2
)
∈
S
]

To restate a different way, e ε represents the amount of information gained by a dedicated adversary investigating the query response in a worst case scenario. This aligns with our understanding of the information vs privacy continuum above; we want to measure how certain an attacker can become about the individual represented in that row. The attacker will be attempting to learn information or update assumptions based on the response, and we want to limit their ability to do so.

The algorithm (
A
) must hold these bounds no matter what row has been added or removed. When we build differential privacy algorithms, we need to determine what queries we can support. This showcases the difficulty of the problem. What if we add an outlier? What if we remove the minimum value? What if the query has joins with several different databases involved? For this reason, tailored algorithms for specific use cases work well for complex problems; and formulating your query clearly is an essential step!

Let’s dive into more details on privacy loss and ε
!

Understanding Epsilon: What is privacy loss?

When we think about the above equation, the epsilon is doing a lot of work! For each individual in our dataset, their information leakage is being bound by e ε
. What does this mean in a practical sense?

Let’s take a real world example to make it more concrete. Let’s say we work together at a company and I want to learn your salary. You are a new joiner and I thoughtfully took a screenshot of a company-wide dashboard for salary transparency that showed me the average across a variety of roles. After you joined, I compared my screenshot to the updated dashboard and tried to infer what role you have and what salary you have. If the average salary went up by 1000 monetary units for a particular role and I had an idea of the number of employees for that role, I could use that information to reverse engineer your estimated salary.

My ability to successfully perform this privacy attack is based on my belief that the dashboard is updated and the accuracy of my knowledge on the number of employees for that role as well as my assumption that you are the only one who has joined. If I am uncertain about the number of employees in that role or of the accuracy of the dashboard result, I insert some estimation of potential error. Instead of being 100% certain that your salary is a certain number, I can now update my suspicion to within a probability range or even begin to think of your salary as a probability distribution (i.e. strongly likely to be between X and Y, but with a small chance of being lower or higher than those bounds).

This example gives us the tools we need to interpret the bounds we are setting with differential privacy. We would like to quantify how much information an attacker can gain, and keep this within some measurable bounds. What is “too much” information gain? Would the ability to update my prior suspicion that a person is in the dataset from 25% to 75% be too much?

Using Bayes Rule, we can reformulate the posterior of the attacker—how much they believe that
D
=
D 1 after looking at the output based on their prior and the other variables. The prior suspicion is represented by
D
=
D 1
, the information from the query result is
A
(
D
)
=
O
,
P
[
A
(
D 1
)
=
O represents the probability that the result would come from the changed database and the posterior is represented by \(P[D=D_1|A(D)=O\)]. Using Bayes’ rule, we can now reformulate the differential privacy formula to give bounds on the posterior depending on the prior.[4].

P
[
D
=
D 1
|
A
(
D
)
=
O
]
=
P
[
D
=
D 1
]
×
P[A(D 1 )=O] P[A(D)=O]

where O is the output that the attacker sees when querying the database. The attacker is trying to detemine if the output is from
D 1 or
D 2
In our case, we know that \(P[D=D_1 = 1 - P[D=D_2]\)], because the attacker is attempting to deduce the shift in the database. Doing some math allows us to reformulate the equation as such[5]:

P[D=D 1 ] e ε +(1-e ε )×P[D=D 1 ]
≤
P
[
D
=
D 1
|
A
(
D
)
=
O
]
≤ e ε ×P[D=D 1 ] 1+(e ε -1)×P[D=D 1 ]

This now shows the potential information gain in forms of epsilon and the initial suspicion. A Bayesian attacker could then use the query response to update the suspicion that the database has changed or not, and we can clearly see that this update is bounded by our choice of epsilon.

Now, let’s look at this span in graphical form in Figure 2-2, which is based on Desfontaines’s introductory post on differential privacy.

Figure 2-2. Estimating Knowledge Gain Based on Epsilon

You can see here that certain values of epsilon are quite revealing, potentially allowing an attacker to immediately update a suspicion quickly and with higher certainty. This seems obvious when we think about our dashboard example.

You might be familiar with choosing a small epsilon, where epsilon is close to 1 or even below 1. As illustrated in the figure, this keeps the information leakage quite small. An epsilon of 1 means that the most a Bayesian attacker can update an initial suspicion of 0.5 by + or - ~0.2. So, they can be more certain that you were in the dataset or were removed from the dataset depending on the query response. You can test out different prior suspicions using the equation derived in the prior paragraph.

How do we actually implement algorithms that satisfy these bounds? We add specific and calibrated noise to the query result that guarantees the information leakage is within those bounds. This noise is not uniformly random, but instead follows a particular distribution; this also allows us to debias the outputs and reason about the inserted error of that noise at scale. This noise allows us to set the privacy loss guarantees and calculate best case, worst case and average case privacy loss scenarios based on the noise distribution. This will be further explained later in this chapter based on the noise distribution we choose.

What Differential Privacy Guarantees, and What it Doesn’t

I hope you are developing a bit of intuition right now around differential privacy, so let’s build on that by covering what it guarantees and what it doesn’t in Table 2-1. The concept has become very popular, generating a lot of hype about its capabilities but also somewhat distorting what it does and does not do.

Differential Privacy is… | Differential Privacy is not…

Mathematical way to bound probabilistic information leakage about an individual

Zero privacy loss, everyone is always “anonymized”

A way to quantify and track privacy loss of an individual in a released (or available) dataset

Applicable for every data science use case and its potential privacy risk (See Chapter 9)

Best way we know (so far) to offer participants plausible deniability

Inference of private attributes based on group membership (i.e. Smokers have a higher chance of lung cancer)

Differential privacy is the best way we know - so far - to offer research participants plausible deniability while still conducting conclusive research. Even still, the results of our research can reveal private information about participants based on their group membership. So if I know you participated in a study and the study showed that 80% of women participants in that study were paid less than their peers, then I can infer based on your gender whether you are paid fairly or not. What differential privacy allows you to do is to retain your ability to say “That may be a lot of [people in my gender category], but not me in particular”.

Another advantage of differential privacy is we can stop talking about auxiliary information, but instead reason about probabilities and information gain based on the differential privacy mechanism, which means we are not trying to figure out what could someone know (impossible!), but instead using our mathematical and probabilistic understanding to choose reasonable privacy guarantees. Since we can always calculate the maximum change that a release of data under differential privacy would bring, we don’t need to explicitly consider what every piece of auxiliary information could be. Since determining all possible pieces of auxiliary information would be a Sisyphean task, this is a truly liberating revelation!

Warning

In any environment where you are discussing anonymization or sharing of person-related data, you’ll want to check with legal professionals to determine the liability of such a release as well as get advice on the organization’s definition of anonymization. In Chapter 9, we’ll review some of the common regulations, but these can change. Good news: Differential privacy is seen by many legal professionals as the “gold standard” because it is seen as state-of-the-art in the technical field. However, this does not mean you get a free pass from the consultation—you must still develop an approach that makes our legal and technical stakeholders happy!

Understanding Differential Privacy

To help you better understand how differential privacy actually works with your data, let’s walk through a few implementation examples.

We’ll start by analyzing a real-world use case, and then build our own mechanism and consider its privacy guarantees.

Differential Privacy in Practice: Anonymizing the US Census

The Constitution of the United States of America calls for a full census of all persons every ten years. This count is used for numerous significant decisions, including representation in Congress, federal funding and monetary support for state initiatives. Getting it right—ensuring that everyone is counted but only once-requires a huge effort. The privacy implications are equally significant.

In the past, the US Census Bureau has used a variety of obfuscation methods to ensure “anonymization” of the results. These included combinations of aggregation and a method called shuffling (or sometimes scrambling), which took census block data and shuffled the households so that the census blocks were mixed with one another. Because the methods used retained lots of information about the individuals, it allowed for private information to leak in ways the original Census workers did not anticipate.

To determine the potential for outsiders to re-identify households in the released data, the Census Bureau ran several attacks on the 2010 Census results. Reconstructing age, gender, race/ethnicity combinations revealed correctly reidentified data with 38% accuracy by combining the data with an external source that was readily available. These external sources often had complete identity information (names, contact information and other details). This could have been a consumer database, a voting or driving record database (for adults) or even an insurance database. For smaller census blocks, they were able to perform this re-identification with much higher success. With ubiquitous data available for free or low prices or when performed by a company with large access to consumer and household data like a large e-commerce provider, this type of attack is not only feasible, it is actively used for direct marketing campaigns and targeted advertising.

How exactly did these reconstruction attacks work? They literally built a system of equations and used a solver to determine potential candidates (more details in this article). From those candidates, they were able to deduce the most probable by linking this information with a few consumer databases or another dataset acquired via a data breach. Although there are plenty of false positives, in less populated regions this proved even more effective (up to 72%).[6].

As a consequence, the US Census Bureau decided that they would use differential privacy for the 2020 Census. The task at hand — could they create a data workflow that allowed for differentially private census results for 308 million persons that was still usable for the critical tasks requiring accurate results?

They refined their privacy parameters using example data and prior census responses, determining exactly what noise measurements and distributions fit their needs. They worked diligently to determine the balance of privacy to utility, ensuring the most accuracy while still guaranteeing basic privacy protections. They built up infrastructure in Apache Spark to run through, aggregate and finalize all results, which are available on the US Census homepage.

A high level overview of how they implemented differential privacy is outlined in Figure 2-3. They started at the highest population levels (which they call a top-down approach), created histograms or sketches of the data and then added the noise barrier (differential privacy mechanisms on histograms) to create noisy histograms. Then, they did this for each smaller regional set and went back to ensure each level ended up with the same aggregate statistics. One exciting property of differentially private data is that you can do as much post-processing as you want without affecting the privacy guarantees.

For more details, I recommend watching the US Census video from Spark Conference or reading through their blog posts on the NIST homepage for differential privacy as well as checking out this comprehensive slide deck on the initial approach .

Figure 2-3. US Census with Differential Privacy

One critique of using differential privacy for public data is the potential of undercounting under represented groups. If there is only one person of a given ethnicity in a particular census block, it’s unlikely that ethnicity will be represented in the neighborhood histogram. How does this affect federal funding, redistricting and so forth? Is it better to ensure you are counted and to reveal your private details?

These questions can’t be answered by any one person, but should be something we consider in terms of how we deploy differentially private systems. Understanding the implications for under represented groups in data will always be a serious point to consider when deploying any privacy protection mechanism. Indeed the individuals in under represented groups are also at a greater danger with regard to their privacy risk than others. You must fully explore the trade off between giving under represented groups more privacy as a defense against further targeting and exploitation and ensuring that those groups don’t disappear or lose benefit from the outcomes (here: democratic processes).

To better understand how we can approach the technical aspects of the problem and assure we are making thoughtful choices, let’s dive into how differential privacy actually works!

Differential Privacy with the Laplace Mechanism

Differential privacy theory tells us that adding noise to data and/or query results can be effective in guaranteeing individual privacy. What we want to do as data scientists is add as little noise as possible, finding the appropriate point on the information vs. privacy continuum. In order to do so, we have to fulfill the requirements of differential privacy while determining how to create an accurate result.

One approach to maximize information that was proposed early on and is still commonly used today is to sample noise from the Laplace distribution. This allows us to meet epsilon guarantees for privacy while adding minimal noise. Let’s take a look at its properties to analyze why it fits so well.

Figure 2-4. Laplace Distribution

As we can see in Figure 2-4, the Laplace distribution has qualities similar to an exponential distribution. It is sometimes called a double exponential distribution. The probability density function is given as:

1 2b e x p
(
-|x-μ| b
)

Where μ is the location where the exponential functions meet (and essentially the peak of the distribution) and b represents the scale of the distribution. You can see that changing the location or the scale shifts the distribution significantly in Figure 2-5.

Figure 2-5. Laplace Distribution: Comparing Scale and Location

When we look at this distribution as a source of noise, we notice that it has a high probability of ending up with a quite low amount of noise. This is great for retaining information. But there is also a chance of gathering noise from the tails of the distribution (i.e. further from the original value), ensuring plausible deniability. As a consumer of the data, we shouldn’t be able to infer whether the noise came from one part of the distribution or another.

Let’s take a look at how this might work in a differentially private mechanism. We want to ensure that the noise is added in a way that protects each individual. One such way would be to add Laplace noise to our query result. We’re going to do this in a fairly naive way to demonstrate why it doesn’t work.

Warning

The code in this chapter are naive implementations of differential privacy mechanisms and are insecure! Please do not use it for any real data or use case. I want you to develop good instincts about how the mechanism works but in real systems, subtle bugs can occur and negate the privacy guarantees (see Chapter 4 for more). In general, building your own differential privacy is like building your own cryptography—you should only do it if you are an expert, and even then, you should have other experts audit your work!

Differential Privacy with Laplace: A Naive Attempt

In Python, a naive implementation might look like:

epsilon = 1.1

def naive_laplace_dp_mechanism(value, epsilon):
# Please do not use this function, ever :)
orig_value = value
value = np.random.laplace(value, 1/epsilon) # more on this choice of 1/epsilon later!
print("Noise: {}".format(value - orig_value))
return value

Let’s create an artificial dataset containing salaries and ages. To run any of this code yourself, please visit the book’s code repository on GitHub.

Here is a histogram of the age distribution (Figure 2-6).

Figure 2-6. Histogram of Ages

I can see here that the count I get back is fairly accurate given our dataset size. Before I start counting age groups, however, I might be curious about the average age in the dataset. I could then write the following.

laplace_dp_mechanism(np.mean(ages), epsilon)

When I ran this, I got the following result. Your result will be different!

Noise: 1.277692233946226

46.55769223394623

So, we can see that the “average age” is around 46.6. But this doesn’t tell me much about the underlying distribution. Perhaps segmenting the dataset by age might help (or building something like a histogram!). If I want to see how many people are over 40 years old in the dataset, I could do so by running the following:

laplace_dp_mechanism(len([a for a in ages if a > 40]), epsilon)

When I ran this, I got the following result. Yours will be different!

Noise: -0.2999259415413036

67.7000740584587

In this toy example, I explicitly added a print statement to show how much noise was being added. Normally we won’t see or infer this information directly from any mechanism because it would violate the privacy guarantees. In order to preserve privacy, we need to also conceal how we add noise to our users and even ourselves; otherwise, we cannot meet differential privacy’s strong guarantees. Since this is a toy example that we will not use in a real system, we can have a look at the internal workings to support our understanding.

You might have noticed, in our Python code above, that we are actually sampling the Laplace distribution with a particular epsilon. Let’s dive into what this calculation means.

np.random.laplace(value, 1/epsilon)

value here is the μ value or location of the Laplace distribution, telling us where it is centered. In this example we are setting the location to the value of the input, which means the center of the probability mass is close to actual input value.

epsilon is our epsilon value for our differential privacy guarantees. In this case we’ve defined epsilon=1.1.

From the numpy documentation, the np.random.laplace method takes the location ( μ
) and the scale (b) as required arguments. Given our above definition, the 1/epsilon value directly calculates the scale of the distribution, affecting the entropy and variance of the distribution.

But can this be correct? The scale 1/epsilon seems to be quite small for accounting for something like an average age over a large population. When we take a look at Figure 2-7, we can see that the vast majority of the noise values will lie between -2 and 2. But what if my age is significantly low or high compared to the average? This could easily shift the average enough that details about my age are revealed in this result.

Figure 2-7. Understanding Sensitivity=1

It seems like we’ve built something dangerous here, so let’s evaluate sensitivity at a deeper level to see what might have gone wrong.

Sensitivity and Error

When I chose 1/epsilon to scale the Laplace distribution, I also set another parameter for differential privacy; namely, the sensitivity of the mechanism. The sensitivity measures the maximum change in the query result based on a change in the underlying dataset. For differential privacy’s definition to hold, you need to account for a dataset both before and after a person was added and removed, therefore limiting the change in query results due to their addition or removal.

Sensitivity equal to one implies that we expect the query result to change—at most—by one when we add or remove one person. This is easier to see this sensitivity when we think about something like a count versus a sum (good news for us, we need both to compute an average!).

Let’s say that I have a salary of 55,000 and an age of 41. When I am added to the dataset and we rerun the count of employees over 40, I change the result by 1. But when we calculate the average age, I can presumably change the sum part of that mean result (count/sum) by 41! This means, that to properly compute an average age, we need to figure out a sensitivity for the ages in the dataset as part of a sum operation.

Some queries (like a sum) will have mathematically unbounded sensitivity. A sum of all salaries to calculate total salary spend could leak extensive information if any salary is added. Without a significant amount of noise, I can easily guess the salary of the new employee or the salary of the employee who is leaving! I just take the first sum and see how it changed after the person joined or left.

Another good example of unbounded sensitivity is to look at minimum or maximum value. Although the query response is unlikely to change when you add or remove a random user, when it does you will again have a very close estimate of the underlying value regardless of the amount of noise added and this would be a catastrophic privacy loss.

When a query has unbounded sensitivity, we need to approach it differently than queries that have sensitivity we can measure. One solution is to not answer or allow queries that have unbounded sensitivity. Another way is to introduce bounds and artificially limit possible answers to within a chosen bound. This presumes that no one will show up outside of those bounds, or if so, their existence in the query response will be left out or removed by changing their value to within bounds.

Some queries will have fairly clear and bounded sensitivity. When I want to count the number of users we have, each user should only be counted once. They can affect the output by only 1. If I am building a rating system and each user can only rate on a scale of 0-5, the sensitivity is 5. Similar to artificially bounding the maximum contribution of any user, there are certain use cases that lend themselves well to measuring sensitivity accurately. There are some examples in the book code repository to show how that might look like.

When looking at sensitivity, however, you also have to think about error. You need to measure privacy loss for each individual and keep that within a small range, but you also must keep in mind how much error you are inserting in your results. When we use a particular noise distribution, we have the ability to estimate our error insertion and to look at how that affects our relative error rates. We could even create an estimate of our error and model plausible outcomes by experimenting with similar non-private data sources to ensure we understand the impact of this error on our later analysis. There are some experiments in the book’s code repository that show this approach.

With this in mind, how should we adapt our query from earlier? What is the sensitivity of the average operation? It depends on the number of people in the dataset, which itself might be sensitive. How do we solve this problem? We’ll first need to ensure we can run one count query and one sum query and calculate the results. This means tracking privacy loss across multiple queries. Let’s look at how to do that with privacy budgets!

Privacy Budgets & Composition

Differential privacy was built to answer questions about data and guarantee privacy, but so far we’ve only looked at how to answer one question and collect that in a differentially private manner. What happens when we want to answer multiple questions about the data? What happens when we want to open our database for external queries using differential privacy? Differential privacy promises that we can track and measure privacy loss—how do we do this?

One amazing property of differential privacy is that the epsilon value (ε) is our privacy loss for one particular response and it has the property of being individual to each query and yet composable — meaning if I answer two queries, I can add their epsilons to calculate the privacy loss!

What does this mean, in practice?

- There is a way to track this privacy loss; it’s called a privacy budget! I can budget my total epsilon appropriately over the course of a certain amount of queries. This allows me to understand both information gain and any individual’s privacy loss when there are multiple queries across the data.

- I may decide to spend more of whatever epsilon I am using on one particular query than on the others. So if I know the queries in advance—or if I am releasing data all at once—I may want to allocate my budget on a query by query basis!

Let’s take a look at how this might work by calculating our average for the average age.

To calculate an average, I need a differentially private sum and a differentially private count on the ages. I know that the most a count can change by adding or removing a row is 1, so I can set sensitivity to 1. I could naively split my epsilon between the two queries, meaning this epsilon would be 0.5.

Next I need to run the sum. What is the sensitivity of an age, though? In order to calculate this reliably, we can use a technique called clamping, that allows us to create artificial floors and ceilings for values. This bounds the sensitivity in a deterministic way.

Since I know this is a dataset of employees, I can choose a floor and ceiling based on the assumptions I have about employees. I might guess that the lowest age in the dataset is 20 years old, and the highest age is 70. This means that I will have to filter my dataset to match these bounds and that my sum sensitivity is 50, because that is the most that the value can change when someone is added to the dataset.

def filter_bounds(value, lower_bound, upper_bound):
if value < lower_bound:
return lower_bound
elif value > upper_bound:
return upper_bound
return value

bounded_ages = [filter_bounds(age, 20, 70) for age in ages]

Now I am ready to calculate my differentially private mean. I will use the other 0.5 of my epsilon to calculate the sum.

epsilon_for_sum = 0.5
epsilon_for_count = 0.5
summed_ages = laplace_dp_mechanism(np.sum(bounded_ages), epsilon_for_sum, sensitivity=50)
count_ages = laplace_dp_mechanism(len(bounded_ages), epsilon_for_count, sensitivity=1)
summed_ages / count_ages

When I run that code, I get:

Noise: -53.24175804583501
Noise: 1.928454653399541

43.88135047434459

The actual mean of the original unbounded age dataset is 45.28, so this result has a relative error of ~ -0.03. Depending on what I was using this for or related queries that this might affect, this error could be reasonable or too much!

Note

For more examples of using toy examples of differential privacy and calculating different sensitivities, please see the book repository! There are several additional calculations and explanations related to the work in this chapter; and a few notebooks that operate as appendices for other related concepts (like local differential privacy and debiasing differential privacy results).

In the queries here, I split my budget across each query, but instead I could think about my budget as I do any limited resource and plan accordingly. This might mean spending more for queries which are more important for my final result or what I am trying to calculate.

Hopefully, you are building up an idea of how these budgets work with example data. I challenge you, at this point in time, to take some of these toy examples and use them with an actual dataset to which you have access. How does your data operate under these mechanisms? What works well and what would you need to reconsider?

Tip

You might be now wondering how you’re going to actually track budgets when implementing differential privacy. The good news is, you can use a library that automatically does this for you, equipped with good tools to minimize mistakes. These could include miscounting the number of contributions a user can make, inadvertently running computations more than once or even incorrectly calculating sensitivity. Most important—now you know how to allocate your limited budget and make the most of it! In the next chapter, we’ll use a library that does just that!

Exploring Other Mechanisms: Gaussian Noise for Differential Privacy

As you are thinking about budgets, tracking and deploying differentially private systems, you might also be wondering if there are other distributions besides Laplace that provide differential privacy guarantees. As a data scientist, you may choose to work with something more familiar than Laplace’s distribution—like a normal distribution.

Great news! You can use a Gaussian (normal) distribution to produce differential privacy “noise” with differential privacy guarantees. You then get all the benefits of working with a Gaussian distribution and can better reason about the results and de-bias the answers using familiar tools and mechanisms.

To use Gaussian noise, we must introduce a more relaxed definition of differential privacy. This definition adds a small delta to the original definition, allowing a relaxation of the probability bounds. This delta accounts for an improbable outcome which could mean “catastrophic failure” or a so-called distinguishing event.

P r
[
A
(
D 1
)
∈
S
]
≤ e x p
( ε
)
×
P r
[
A
(
D 2
)
∈
S
]
+ δ

How should you properly choose a delta (margin of error)? We could imagine a delta which allows us to, essentially, release ALL information about a small number of rows with no noise added, if that fits within the delta! Take a database of 1 million rows, and release 1 row “in the clear” (i.e. without any changes or modifications); if your delta is
1 10 -9 or greater, then you can call it differential privacy (just kidding, please don’t!). For this reason, it’s recommended that delta be cryptographically small; or minimally smaller than 1/total number of rows, to disallow this undesired behavior.

Let’s take a look at how this relaxation and use of Gaussian noise works in practice. First, we need to define our differential privacy guarantees when working with a Gaussian mechanism.

Warning

The following definition for a Gaussian mechanism is only for epsilon < 1 and adds more noise than you actually need or want to do, as it is based on the original paper which has been improved. The recommended way to add Gaussian noise at the writing of this book is analytic Gaussian noise, described in Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising, Balle and Wang, 2018. There is also a blog post you can read showing some of the properties! Because it involves a more advanced formula, we’ll use the simpler and less useful one here to build our initial understanding and then use a library in Chapter 3 to assist in adding better Gaussian noise.

A Gaussian mechanism is
( ε
, δ
)
differentially private if it follows the criteria (so long as the epsilon is < 1):

F
( x
)
= f
( x
)
+
N
( σ 2
)

where
σ 2
=
2s 2 log(1.25 δ) ε 2
where
N
( σ 2
)
samples a Gaussian distribution centered at 0 with variance σ 2
. Here s represents the sensitivity.

You’ll notice that now we must account for delta, which we didn’t do with Laplace. This is because Laplace satisfies
( ε
,
0
)
differential privacy, but Gaussian does not.

You know now that delta should theoretically be “cryptographically small” (which is defined as
≤
2 -30
. But, in practical differential privacy, the delta should be much smaller than
1 numberofrows in the dataset. Since you are going to use a differential privacy mechanism and not some tricky release of a row, we can relax some of this advice. A good choice for delta is somewhere between “much smaller” than
1 numberofrows and
2 -30
. Let’s choose a number to begin with for our age and salary dataset. Since we have 100 rows, this means it must be significantly smaller than 0.01 and it could be larger than
2 -30
(which is approximately 9^-10). Let’s choose δ
=
10 -
5
, as an appropriate number between those recommended bounds.[7]

Comparing Laplace and Gaussian noise

To evaluate and compare Laplace and Gaussian noise, we need to go back to the fundamentals of sensitivity. These two mechanisms have different notions of sensitivity. To reason about this, let’s think in multi-dimensional vector space. We have the initial vector representation of the true result of the queries without noise. In order to add a fitting differential privacy mechanism, we must apply the appropriate sensitivity. How do we calculate the distance appropriately to measure the sensitivity and to calculate and track our budget?

For Laplace mechanisms, we use Manhattan distance (
L 1 norm) to optimize and minimize error. With differential privacy we are inserting the error, but need to know how much to insert, therefore we measure the sensitivity the same way we would measure and reduce error.

For Gaussian-based mechanisms, we can use Euclidean distance, because the noise we are adding would be the same noise we would expect from normal error. Gaussian noise will distribute the error normally and it is best accounted for using Euclidean distance or the
L 2 norm.

To find the Manhattan distance or
L 1 norm in vector space, you calculate:

||x|| 1
=
∑ i=1 N
| x i
|

To find the Euclidean distance or
L 2 norm in vector space, you calculate:

||x|| 2
=
∑ i=1 N
|x i | 2
= x T x

To visually inspect what these differences in error and noise mean in 2-dimensional space, let’s look at a comparison of
L 1 versus
L 2 distance of two points (Figure 2-8).

Figure 2-8.
L 1 vs
L 2
Distance

Figure 2-8 makes it clear how the methods get their names! For Manhattan distance, we treat distance like blocks of Manhattan in New York City, which is a grid-based city design. For Euclidean distance, we are using geometry to find the hypotenuse of two points. Of course, this must be expanded to account for multi-dimensional space.

The noise and, therefore, error will grow as the amount of statistics a single individual can influence increases. You’ll need to make note of that growth, depending on the overall series of computations and queries you’d like to run. For example, if I want to calculate the correlation between several features, a single person can affect many points, assuming that the features are person-related.

Let’s take a concrete example to make it really clear. You are working with Human Resources to analyze whether particular teams or departments have issues that leadership needs to address. You want to protect the individuals who are contributing these issues. Leadership wants to track problematic areas and properly prioritize, so you need to calculate the average number of complaints per employee, team and/or department.

First, you will need to choose clamping bounds, since the number of complaints per month could be 0 to unlimited. You decide on a range of 0-10. You want to scale these with mean-time to resolution, so each complaint is given a range from 0-5 (resolved by the employee who filed it that day - not resolved within a rolling month). This will help highlight chronic issues as opposed to potential false positives, where someone might have miscategorized the ticket.

If you use Laplace noise, and a co-worker sends 10 or more complaints which remain unresolved, you need to scale their noise along with Manhattan distance. This means scaling the noise by 5*10. On the other hand, if you use Gaussian noise, the noise will scale by 5* s q r t
10
. Now you can see the advantage of understanding how one user contributes multiple statistics!

A good rule of thumb is to understand that Laplace noise scales linearly with the amount of statistics a person can influence (k), but Gaussian noise scales by square root of k. So if the sensitivity is high or one person can affect quite a bit of query results, Gaussian noise is superior.

What does this mean in practice? Let’s add some noise to our salary and age data to have a look! Again, this is a toy example and should not be used in any real-world situation — please always use a public and properly audited library!

epsilon = 0.99
sensitivity = 1
loc = 0
delta=10**-5

def gaussian_dp_mechanism(value, epsilon, sensitivity=sensitivity):
# Please do not use this function in real life - it is susceptible to well known attacks
# instead, use a well-known and audited open-source DP library
orig_value = value
gauss_scale = sqrt((2*sensitivity**2*log(1.25/delta))/epsilon**2)
value = np.random.normal(value, gauss_scale)
print("Noise: {}".format(value - orig_value))
return value

gaussian_dp_mechanism(len(ages), epsilon)

When I run this code to see how much noise is added to the age count, I get the following result.

Noise: 2.7443351762994865

102.74433517629949

Great! Now you’ve had a chance to compare a few noise mechanisms and begin building the mental model you need for making decisions. There’s not enough room in this chapter to cover all the methods you could use, but you should now have the basis for evaluating how sensitivity, noise and error relate. Let’s expand on this and examine how we actually will use noisy results in our analysis.

Real-World Differential Privacy: Debiasing Noisy Results

When we are using differentially private data, we are essentially using data with higher error. We want to start thinking intelligently about how to use the results we get. What can we do to reduce errors inserted by differential privacy? Let’s again look to Bayesian reasoning to determine if we can find a way.

One way we can approach denoising the data is to gather information about the mechanism. Sometimes we know the mechanism used or even the exact parameters for the mechanism (i.e. epsilon, delta and sensitivity). Other times we might only know that differential privacy is used.

To begin our Bayesian reasoning, we’d first take what knowledge we have and begin to develop our priors. Was Laplace noise, Gaussian noise or a different distribution used? What was the scale of the distribution? What are plausible values for epsilon?

For even more certainty and to model our assumptions, we could use a Bayesian library like PyMC3 to run a series of experiments. By properly identifying, documenting and modeling our priors, we can see if our experiments match the behavior of the differentially private data source. If our model looks incorrect, it probably means we need to update our priors!

One neat reason to use Gaussian noise is that we often assume errors are distributed normally in our data in the first place. Whatever error normalization or correction you have built into your initial design and reasoning likely works well when using Gaussian noise.

Note

In this chapter, we use the term noise and error at times interchangeably. This is on purpose, to highlight that our data already has an element of error and noise when we collect it. There is no such thing as “ground truth” or “pure data”, which means when we insert artificial noise and error via differential privacy, we are modifying one already error prone version of reality to another.

For more on this iterative approach, please take a look at the book’s code repository which has a notebook to walk through Bayesian debiasing!

Tip

Another good reason to look at Gaussian noise for data science, is the properties of a normal distribution. If your dataset or minimally certain variables in the dataset are also normally distributed, you still end up with a normal distribution. Normal distributions also have nice properties for linear algebra and machine learning, when we need to apply linear and affine transformations to Gaussian matrices or vectors.

Based on this discovery, and supporting evidence outlined in the final section of Desfontain’s blog:

- If you need delta=0, you stick with Laplace.

- If a single person can only have very few statistics (typically, fewer than 8, but it can vary with other parameters), then Laplace is better; while if you have lots of statistics, Gaussian will be best.

Of course, approaches to differential privacy are still actively under development and research, so I recommend not memorizing a particular number to design your approach; but instead to look at the bigger picture as you develop your understanding of this technology and determine what makes sense for your particular use case. Let’s review a more abstract or theoretical view of differential privacy to support these decisions.

Differential Privacy: A more nuanced definition

Up to this point, we always defined the change in the database as having one person added and one person removed. But this may, at times, be a poor fit for the privacy problem we are trying to solve.

What if a device is used by multiple users (such as a car or a home assistant or a wireless router). We might then want to ensure the privacy of that device for all users is protected, making the difference between (\D_1\) and (\D_2\) device-based.

Or what if we want to investigate user behavior over a long period of time (such as browsing history, location history, purchasing history). Running multiple queries on this data over a long period of time would likely use up our privacy budget without fruitful analysis. We might at that point in time decide that we want the differential privacy budget to be time-bound (i.e. to reset it at a particular interval) or for the guarantee to be for a smaller set of actions or even on an individual action basis.[8] Of course, these guarantees then become weaker when releasing on a smaller privacy unit — such as only guaranteeing privacy on a per-day or a per-week basis or for an action rather than a person-level. If an attacker has auxiliary information, they can use the differentially private queries or data release to increase their suspicion with more information over time.

For those of us performing anonymization in order to comply with data protection legislation, like differential privacy to anonymize data under GDPR or similar laws, I think a discussion with your team on these choices are particularly relevant for your application or use case. You might also want to look at research literature around a particular algorithm or use case to determine if advanced approaches meet your privacy and utility requirements.[9]

We’ll dive into one of the few useful open-source differential privacy libraries in the next chapter which should make practical approaches more useful, but you might still be wondering if there are other definitions of anonymization. Maybe you’ve heard of k-anonymity? Let’s explore what it is and why it’s no longer recommended for use.

What about k-anonymity?

K-Anonymity is an older approach to anonymization created by computer scientist Latanya Sweeney. It was created after she proved that she could de-anonymize medical records released by the State of Massachusetts. The governor William Weld had proposed a release of all state medical records for public review, claiming they had been properly anonymized. She proved him wrong after joining the public medical records with easily purchased voter registration records, joining using age, zip code and gender. She was able to single him out based on these identifiers and mailed his records to him, and he revoked the public release.

K-anonymity was first created as an attempt to mitigate this type of linkage attack. When using k-anonymity, we group persons with similar sensitive attributes (like age, zip code and gender), and we don’t release groups that have less than k persons in them. For example, if we had a table that looked like this:

Age

Zip Code

Diagnosis

54

11189

Covid-19

44

11188

Normal Flu

22

11187

Covid-19

25

11189

Covid-19

44

34555

Covid-19

72

34555

Covid-19

66

34556

Normal Flu

77

34555

Normal Flu

78

11189

Normal Flu

75

33444

Covid-19

We might update it to look like this, setting k=2:

Age

Zip Code

Diagnosis

40-60

1118*

Covid-19

40-60

1118*

Normal Flu

20-40

1118*

Covid-19

20-40

1118*

Covid-19

60+

3455*

Covid-19

60+

3455*

Normal Flu

60+

3455*

Normal Flu

As you can see, we’ve generalized some of the groupings to preserve more privacy and to ensure that no one person can be singled out. We’ve also dropped some rows in order to let k=2 and any singletons are therefore removed. Is privacy protected?

What about if I’m fairly certain you were in the dataset and you are in your mid-20s and you live in a zip code that begins with 1118. I can immediately deduce that you likely had Covid-19 because 100% of the 20-40 year olds in this dataset and that zip code had a positive diagnosis. This isn’t what I wanted or intended. When noticing problems like this in real datasets, researchers proposed a new method that should be used in conjunction with k-anonymity called l-diversity. L-diversity said that there must also be variance in related sensitive attributes, so the k-anonymity process doesn’t create homogenous groupings where sensitive attributes are unintentionally exposed.

However, that wasn’t the only restriction that became necessary. There also was a chance that the data would become skewed in the process of creating the groups, rendering it useless for processes like machine learning or statistical inference as well as creating imbalanced groups that were non-representative of the actual data. For example, what if we proposed a value of k=2 and a value of l=2. We would end up only with these rows.

Age

Zip Code

Diagnosis

40-60

1118*

Covid-19

40-60

1118*

Normal Flu

60+

3455*

Covid-19

60+

3455*

Normal Flu

60+

3455*

Normal Flu

Not only have we significantly reduced the information we have and the data we can use, but we also now have a skewed dataset that shows a higher incidence of the normal flu than Covid-19, which was not the case in our full dataset.

To try and manage this problem, t-closeness was proposed, which stated that the l-diverse groups needed to have t-closeness to the global distribution of these attributes. Therefore, we now need to create groupings that more closely reflect the global distribution of these related attributes. Let’s try that here…

Age

Zip Code

Diagnosis

None

None

None

Yes, that’s correct. We cannot release any data and match these requirements…

I think you’re starting to see the limitations by now. K-anonymity is like taking a hatchet to your dataset, when a chisel might work best for what you are actually trying to achieve. K-anonymity provides no way to quantify privacy guarantees, must be customized to each dataset, can take a lot of compute power to properly implement and adds large amounts of bias to the dataset. Unlike differential privacy, there is never a way for us to interpret how much information an attacker can learn or to reason about privacy loss of individuals in the dataset. Therefore, it is not generally used or recommended today to anonymize data.

Summary

In this chapter, you’ve learned how to define privacy and assess definitions and requirements of anonymization for your organization. You’ve learned the basics of differential privacy as the leading solution for anonymization. You’ve learned why older approaches are no longer relevant, compared different types of mechanisms and budgets and developed an initial understanding of how differential privacy works in a practical environment. You’ve started to appreciate the difficulty in comprehensively building mechanisms to address things like sensitivity and worked with toy implementations to get an idea of how differential privacy would work for your data problems.

In doing so, you’ve begun to appreciate the complexity of the privacy problem at hand. In the next chapter, we’re diving into how to take your understanding of governance, data protection and differential privacy and use it in real systems today leveraging available libraries.

[1] From Merriam-Webster Dictionary, accessed January 2022

[2] See: Dwork et al., Calibrating Noise to Sensitivity in Private Data Analysis, 2006.

[3] Aside from the aforementioned paper, you might want to read Dwork, Differential Privacy, 2006 and Dwork and Roth’s publication The Algorithmic Foundations of Differential Privacy (2014) which is freely available online.

[4] You don’t need to have studied Bayesian theory to understand the following, but it does help to think about priors and posteriors when determining information gain and the resulting privacy loss. If this is new to you, you can familiarize yourself with this theory via many O’Reilly publications and freely available blog, textbooks and video courses online — choose one that speaks to you!

[5] A proof can be found in this blog post

[6] Even before the Census released this method, I knew of several data teams using statistical reasoning to perform direct marketing based on census data in combination with consumer databases. Think about what we can infer from customer databases, particularly with filtering mechanisms that highlight purchase or browsing history to estimate income or gender, and then combining that with publicly available data—very revealing and quite scary! More on this in Chapter 4.

[7] You can read about ways to optimize scaling of the noise in relation to delta in Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising, Balle and Wang, 2018; or take a look at their code repository.

[8] For an example of this method, you can take a look at Facebook’s public release of actions, which was created via an action-level differential privacy mechanism.

[9] A good source of learning about potentially interesting differential as well as new developments in the field is to take a look at privacy-oriented technical conferences, like PETs Symposium, the USENIX Privacy Engineering conference or PPML conferences like the ACL one or IACR’s one.

Chapter 3. Building Privacy into Pipelines

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at katharine@kjamistan.com.

Now that you’ve evaluated different approaches to pseudonymization and anonymization, let’s dive into how we can build these approaches directly into normal data workflows. If you work with pipelines and other large scale data infrastructure, you can leverage those to implement privacy. This can be a sustainable and extensible approach to designing privacy into your data architecture. You can then determine what other workflows (like continuous delivery or continuous integration workflows) might benefit from using these methods. When we build out privacy methods that are scalable, defined by a multi-disciplinary team of experts and implemented by a group who understands not only the privacy technologies but also the infrastructure of the company, we move from throw-away one-off operation to actually scalable and maintainable privacy by design.

In this chapter, you’ll learn about how to incorporate privacy technologies into your data engineering infrastructure and software you work with daily[1]. You’ll also learn some tips for working with data engineering teams (in case you aren’t in one already!). Finally, you’ll learn how to engineer privacy into your data collection methods and how to increase your self-documenting data collection.

How to Build Privacy into Pipelines

In Chapter 1, we looked at data governance basics and how to modify our data to apply basic privacy protections. In Chapter 2, we learned about anonymization and differential privacy methods. Now that we understand these basic building blocks of privacy, we need to figure out how to automate and scale them into real data infrastructure. In this section, you’ll learn how to think about scaling those techniques in a sustainable manner, how to test they are working and how to ensure the workflows and pipelines you are building help add privacy as early as possible.

Before we begin building privacy into these workflows, we need to have properly outlined the risks, documented the data (as best we can), understood the use case and sensitivity of the data and outlined our processing for others (on our team and other teams). Based on the governance and sensitivity you determined after going through the process in Chapter 1, you now know the methods you need for the data you are collecting, using or transforming. Whether you are using masking, pseudonymization or differential privacy, you can follow these steps to plan and engineer your privacy efforts into pipelines.

Design Appropriate Privacy Measures

You might have hundreds or even thousands of data engineering jobs running daily or continuously to process data. Regardless of the scope of an extensive data processing setup, outline what privacy-preserving measures you will take for the variety of data types you are consuming so that it is clear to everyone involved. If you are not sure what an appropriate privacy technology measure is, start small (with something like masking or redacting sensitive fields) and experiment. You are looking for the sweet spot between information and privacy, allowing enough utility so the data can be used for the purpose it was collected. At the same time, collecting personal or sensitive data that just sits around and no one uses it is always a bad idea!

Let’s walk through a few guiding questions and recommendations you can use to help assess if your measures are appropriate.

Define your purpose and use case: What is the purpose of collecting this data? Who will use it and for what? Define your use cases clearly with the internal users or end users. If you aren’t sure why the data is being collected or used, you need to talk with others about if it is worth the risk of collecting data you might never use! Follow and learn more about data minimization, which is the best privacy-preserving mechanism available!

Maximize privacy: Based on your initial use case, what is the maximum privacy we can offer while still getting our job done? Sometimes this means experimenting with different privacy technologies and seeing the results. To do so without losing data, create a temporary, secure, internal storage or sample that has no privacy protections. Test different methods and have the team or data consumer take a look and see if they can answer their questions. You might want to therefore start with higher privacy (i.e. removal of fields, masking and/or differential privacy). You can then relax these if the analysts, scientists or engineers need more utility. After a few rounds, you’ll be building experience and knowledge for yourself and the team on what measures are appropriate for what types of tasks. You’ll also likely be learning new libraries and approaches as they become available!

Expand use cases: When the appropriate privacy-utility equilibrium is reached for the initial project, what other tasks, use cases or data consumers would fit the same or similar requirements? Try an agile approach where you are rolling out these small changes, learning and adapting one more use case or task at a time. Your data consumers or end users are also learning as they go, and they might have different needs that will require fine tuning. When an approach works across multiple teams and similar use cases, it can become a standard method or approach. This can be documented and taught to new data engineering or data privacy joiners. If for some reason, the approach fails after a certain time, ensure there are ways for the data consumers to communicate what has changed!

Experiment, learn and adjust: When new privacy technologies become available, or when you grow the privacy skills on the team, where can you optimize and adjust? As the space is continuously growing and changing, you want to always have a few team members looking at what is next! Is there a new version of a cool library you are using? If so, what new features does it have? Is there a new team member who brings a particular expertise or a new privacy engineer hired? How can you integrate their advice into our workflows? Is there a set of measures you want them to review? Being forward looking and thinking will ensure your privacy measures are fitting and appropriate now and in the future.

Once you’ve found the measures that work for your use cases, it’s time to solidify them into each workflow. But before you do that, let’s discuss building data flows that work for everyone involved.

Meet the User Where They Are

Who is receiving the data at the end of your pipeline? What requirements do they have from a privacy and a quality perspective? Figuring out who is your “data customer” is an important part of building better and more useful pipelines.

In getting to know their needs and use cases, you’ll also better understand exactly how to approach the privacy part of the problem as well as institute better checks to ensure everything is operating as planned.

As you learned in Chapter 2, building differential privacy in requires a good understanding of the data, the use case and the sensitivity involved. Each of these factors will change over time. Performing regular check-ins with the data consumer is a good idea to ensure that the differentially private data is still meeting expectations and fulfilling the needs of the team. If that changes or if the underlying distribution fundamentally changes (e.g. from a change in the software or application that produces the data or a shift in the population), then the differential privacy mechanism should be rearchitected to fit. This might be as simple as changing the clamping bounds or sensitivity, or as complex as developing a new algorithm.

Just as important as ensuring the data is as expected is communicating the shifts that the data consumers will see due to the privacy protections in place. This should also be an ongoing conversation where the team implementing privacy controls is helping adjust them to match the needs of the business as well as compliance, legal and other stakeholders. Teaching people how to use data that has undergone privacy measures will now be a new skill for the privacy and data teams. Ensuring this conversation goes smoothly and that folks feel empowered to ask questions and experiment will be a key factor in the success or failure of your privacy initiatives.

I’m personally a big fan of meeting people where they are at in terms of user experience. If you run into an analyst or engineer who has spent 20 years doing their work in a particular way and you’re about to change it, figure out how to do this in the least disruptive way as possible. They might have been using the same methods for many years, and ensuring that you don’t disrupt their work and help teach them to embrace privacy technologies is important. Make small changes almost imperceptible, walk them through changes step-by-step, listen to their concerns and incorporate them into your architecture and planning, keep interviewing them as the process is going to determine if it is working for them. A little bit of psychological safety can go a long way!

In doing the work here, you’re also figuring out ways to take small slices of work and incorporate them into your pipelines.

Engineer Privacy In

As you build out the privacy engineering you’ll need to automate privacy, you want to try to engineer privacy into these systems rather than create a bunch of one-off user-defined functions. That’s how they might start, but take a step back to determine if there are holistic ways to engineer privacy into the system once your small tests have shown that they work.

One way to do so is to create small software packages in whatever language or platform you are using that perform these steps. If you are already using software that supports some of the functionality (like Apache Beam with built-in support for differential privacy), then leverage those tools whenever possible! If you are using Spark, you can also use the Tumult Analytics library (coming later in this chapter!) or the PipelineDP project. Only do the heavy lifting if you have a specific use case that you need (for example, a very particular type of format-preserving encryption you’d like to fulfill a particular team’s requirements).

In addition to leveraging privacy technologies directly in the pipeline, you’ll also want to leverage testing to validate that things are working as you expect.

Test and Verify

You are testing your pipelines, yes? This is a best practice for many years now, allowing you to better understand and validate that things are working as planned. Outside of unit testing or running integration tests, testing the actual data that is being transported is essential to determining the health of your system.

I’ve been a long-time fan and user of Great Expectations for this work, which has a fairly intuitive interface that allows you to use both built-in expectations as well as build new ones. You can think of this as static analysis for your data. Does the data pass my smell test[2]? If not, you can have the data flagged or halt the entire process. No sense in using a cluster for a half-day if a required field is actually missing because the related database didn’t load properly!

When we think about using a tool like Great Expectations for privacy, what would we want to test? Well, we might test that particular fields are present or missing if we have added fields post privacy processing or if we expect that particular sensitive fields to be removed or not included. We can also test that our differentially private clamping bounds are working as expected. We can even come up with fairly clever tests that particular fields are hashed or encrypted (by testing string entropy), or that certain masking tokens are present in fields where we expect them to be.

Now that you’ve learned the guiding principles for building out the privacy technologies and enforcing standards across the organization, we want to actually engineer these methods. In the next section, you’ll learn how to build these processes in a pipeline context where we want to collect governance data and apply privacy transformations!

Engineering Privacy and Data Governance into Pipelines

After you’ve spoken with the data consumer, clarified the privacy requirements and run some experiments on the data to confirm it will fit expectations, you’ll be ready to actually engineer the initial use case or several use cases. To get these privacy transformations into a production environment, you’ll want to ensure they match your current data workflows.

Let’s see how this can work with a concrete example of sharing data between different departments at a company. After, we’ll explore how to collect provenance and consent data in a workflow.

An Example Data Sharing Workflow

A common internal data sharing pattern is the need to share data across an organization or even between organizations. How can you do this in a secure and privacy-aware manner? You need to consider two goals: the privacy of the users or employees as well as the needs of the data consumer. You have consulted with internal experts to analyze the privacy and security risk and have determined the appropriate level of protection required.

Let’s say you are working at a sustainable chocolate company. Your data is purchase data based on items the users have purchased on the website and you need to share it with the marketing department, who wants to measure the effectiveness of their campaigns on a user-by-user basis.[3]

Based on your initial analysis, you have a plan for how the data should be transformed to apply the fitting privacy and utility for the use case. Your plan is:

- Remove personal identifiers with the exception of the user ID, which is hashed so that if marketing team comes back with specific questions, you can answer them by finding the matching user ID

- If the user session came in with a tagged campaign, retain campaign information; otherwise leave blank

- Retain billing city and state, as this was specifically asked for by the marketing department to effectively answer questions for their campaigns

- Join order data and aggregate order quantity (number of orders) and order value (sum of quantities) per user

- Bound outliers that have exceptionally large or small amounts of customer value (if needed, you can create a written summary of these for marketing review)

Let’s outline how this might look in a workflow. In pseudo Python it might look something like this:

order_dataframe.drop(['street_address', 'first_name', 'last_name',...])
browser_dataframe.drop(['ip_address', 'browser_user_agent',...])

order_campaign_df = order_dataframe.merge(browser_dataframe, how='inner', on=['order_id'])

# here you would use a key that was securely generated and stored for this particular job
# the key is maintained only for the duration of the data investigation and then destroyed
order_campaign_df.user_id.map(lambda x: encrypt(x, key))

order_campaign_df = order_campaign_df.groupby('order_id').agg(
{'campaign_uri': 'first',
'user_id': 'first',
'city': 'first',
'state': 'first'
'total': 'sum',
'num_items': 'count',
})

order_campaign_df = order_campaign_df.total.map(remove_outliers)

# then export and share with details on processing!

An example of this process is also in a Jupyter notebook in the book’s repository. Of course, please adapt these examples to fit the framework and languages you use at your organization!

How do you know if the pipeline is properly working? You test it, of course!

To do so, let’s use a few expectations from the Great Expectations library. First, you can import Great Expectations directly into your notebook and set it up so that it can sample from the dataframe.

import great_expectations as ge
context = ge.get_context()

Next, you can start to explore what expectations are available by creating an augmented dataframe that has the expectations built in. To do so, you can use the tab auto-complete function in Jupyter and type expect and then Tab. You’ll see a long list that you can navigate through!

ge_df = ge.from_pandas(summary_by_order)
ge_df.expect # Now press tab!

Here’s one example, to test that the total price has removed the outliers based on the prior analysis. You could also define it to instead look at percentiles, max and min as well as standard deviations!

ge_df.expect_column_values_to_be_between('total_price', 1500, 27000)

You’d normally then continue to add expectations by hand, or there are ways to run a Great Expectations profiler and build a set of expectations. Then, you need to save the expectations and set up the pipeline to run them as a step in the processing. There are more details that you’ll need to decide and get working in your setup, with many guides available via the Great Expectations documentation.

ge_df.get_expectation_suite(discard_failed_expectations=False)

# Here you can check that the expectations meet your needs!
# If so, write them to a JSON file and set up your system to use this file plus GE to test data as it processes!

import json
with open("order_summary_for_sharing_expecation_file.json", "w") as my_file:
my_file.write(
json.dumps(ge_df.get_expectation_suite().to_json_dict())
)

Now you can be certain that should anything malfunction, you will be alerted. The first few runs, you might also want to check with your data consumers and ask if their analysis is going well. I recommend daily check-ins the first week, then once a week check-ins for a month and then you can have someone on the team take a look at the overall statistics for Great Expectations as part of a weekly health report for all pipelines and note any massive deviations which would require further investigation.

Let’s investigate another type of use case to get a feel for adding governance to pipelines. To begin, you’ll want to ask: How do we get the order and session information in the first place?

Adding Provenance and Consent Information to Collection

Hopefully after reading Chapter 1, you are excited about determining how to add provenance and consent information to your data collection. To do so, you’ll need to start either collecting and attaching consent as you collect data, or collect consent via other mechanisms and combine it with the data you have.

Ideally, consent would be given not only for data collection but also data usage. This is not how we design systems and interfaces today. Generally, we push the user a list of data we collect and then a long Terms of Service or Privacy Policy that covers all of the possible uses, usually not specifically outlined per use case but instead all encompassing.

If you’re at a company that wants to approach privacy differently, you could try to do your policy and data collection differently. It might look something like Figure 3-1. When a user first opens the application the initial setting are loaded and the interface is shown to them. The defaults should be privacy-aware, ensuring that only the required data collection is on and all other settings are turned off. When new features or processes are added, the user is prompted to adjust their settings. In addition, there are options for local-only data storage where data is stored only on device and not shared with a central server or application unless absolutely required for functionality (more on this in Chapter 6).

Figure 3-1. Privacy-Aware Consent and Data Collection

If you wanted to change the use cases or you had a new way you were going to use the data, like for a new machine learnign model, you would push that to users and explicitly ask for consent for that new use case. One step further to make it easy for the users is to give them an interface to review all of their consent options with each category of use case and toggle them as needed. That could look something like in Figure 3-2. You can see that in this interface it is clearly stated what data is needed for each different data use case and the vocabulary used is easy-to-understand. The user is given a variety of choices, including the ability to anonymize data or to keep data on the device.

Figure 3-2. Fine-Grained Consent Management Interface

There’s been a lot of user design and user experience research in better defining privacy interfaces—like ind.ie’s Ethical Design Framework and Privacy UX, as well as what not to do, like the University of Ulm’s work on bad privacy design patterns. If you have customer experience designers at your organization, it’s worth starting a conversation with them to determine how to align workflows and enable better consent collection and clearer messaging. As you probably know from being a user of user consent interfaces, it’s a lot easier to say yes when it’s clear what your data is being used for and you know there is a place you can go to and revoke if you would like to do so.[4]

Even if you cannot redesign consent collection in a meaningful way, you should note what consent data you have at time of collection in case it changes. As you know, the regulatory landscape continues to change and knowing what the version of the privacy policy and terms were at collection time can help you make better decisions later with regard to data usage, retention and minimization.

How might privacy-aware consent and data collection work in practice? Let’s outline our session collection requirements and design a schema that is useful for later usage! Here’s what we would need to collect from the user on initial collection:

- Privacy Policy Version

- Privacy Policy Retention Period

- Date the User accepted the policy

- Privacy Policy Legitimate Usage (i.e. what can we do with the data for legitimate use cases?)

- Terms and Conditions Version

- Terms and Conditions Data Localities

- Usage Preferences (i.e. Please use my data for this service but not related partner services, or please use my data for machine learning but do not share it with other companies)

- Provenance Details

And finally, it’s not really developed until it’s been tested and validated. How can we ensure that our consent data is properly tracked?

Here’s an example schema validation we could use to ensure that things are properly collected and stored. Here I’ll use Apache Avro syntax to show another example of testing data schema and validating data structures in pipelines.

{"namespace": "example.avro",
"type": "record",
"name": "User Consent Data",
"fields": [
{"name": "username", "type": "string"},
{"name": "policy_version", "type": "float"},
{"name": "policy_retention_months", "type": "int"},
{"name": "agreement_date", "type": "datetime"},
{"name": "legitimate_usage_categories", "type": "array", "items" : "string", "default": []},
{"name": "terms_version", "type": "float"},
{"name": "terms_localities", "type": "array", "items" : "string", "default": ["us-aws-east"]},
{"name": "usage_detail__location_on", "type": "bool"},
{"name": "usage_detail__location_ml", "type": "bool"},
{"name": "usage_detail__location_analytics", "type": "bool"},
{"name": "usage_detail__location_sharing", "type": "bool"},
{"name": "usage_detail__session_on", "type": "bool"},
{"name": "usage_detail__session_ml", "type": "bool"},
{"name": "usage_detail__session_analytics", "type": "bool"},
{"name": "usage_detail__session_sharing", "type": "bool"},
{"name": "provenance_location", "type": "string"}, /* application or website, etc */
]
}

Here, I have chosen to flatten the detail fields so that it is very easy for us to filter data based on the type of fine-grained consent controls. This means that when I build a continuous integration pipeline for machine learning, I can actually select all the users who have given consent to have their data used for machine learning automatically!

I also now have data on when the policy says to retain data and when the agreement date was made, which makes it quite easy to batch anonymize or delete records that are expiring soon because they have reached the full retention period. Building pipelines for these tasks is a smart thing to do before those expiration dates get too close, which means thinking through how to make differential privacy a normal part of your data workflows. Let’s take a closer look!

Using Differential Privacy Libraries in Pipelines

When adding differential privacy into current or new pipelines, it’s best to use a well-supported library that integrates into the technology that you currently use. If you are already using Apache Beam on Google Cloud Platform, I recommend using their Codelab notebooks to walk through possible ways to implement differential privacy.

If you are using Spark, I recommend the Tumult Labs library, which automatically has privacy accounting and a variety of other features built in.

Warning

Since the Tumult Labs library is fairly young, you’ll want to check out the documentation in case anything has changed. The notebooks from the book repository will also regularly be updated, so please refer to those to try out the library!

Let’s take the Tumult Labs differential privacy library for a spin to see how it would work as part of your normal workflow. For this, you will use a new dataset that is referenced in the Tumult documentation.

To begin, you need to first start a Spark Session and initiate a differential privacy budget. Here, you will chose epsilon=1.1.

session = Session.from_dataframe(
privacy_budget=PureDPBudget(epsilon=1.1),
source_id="members",
dataframe=members_df,
)

In this dataset, you have a large list of library members. Let’s first check the columns:

members_df.columns

This outputs:

['id',
'name',
'age',
'gender',
'education_level',
'zip_code',
'books_borrowed',
'favorite_genres',
'date_joined']

You are curious as to whether there is a correlation between the amount of books borrowed and education level. To take a look at categorical columns using Tumult, you need to create a KeySet, which allows you to define the categorical variable values you expect to see.

edu_levels = KeySet.from_dict({
"education_level": [
"up-to-high-school",
"high-school-diploma",
"bachelors-associate",
"masters-degree",
"doctorate-professional",
]
})

edu_average_books_query = (
QueryBuilder("members")
.groupby(edu_levels)
.average("books_borrowed", low=0, high=100)
)
edu_average_books = session.evaluate(
edu_average_books_query,
privacy_budget=PureDPBudget(0.6),
)
edu_average_books.sort("books_borrowed_average").show(truncate=False)

Here, the Tumult KeySet and QueryBuilder classes help create differentially private query. The example code clamps the number of books borrowed to between 0 and 100 as you learned about in Chapter 2. You can then evaluate the query and tell the library how much of the privacy budget you want to spend. When I ran this, I got the following output (yours will be different due to the use of differential privacy!).

+----------------------+----------------------+
|education_level |books_borrowed_average|
+----------------------+----------------------+
|doctorate-professional|18.929587482219063 |
|masters-degree |19.1402224030377 |
|bachelors-associate |19.173858890761228 |
|up-to-high-school |19.361286812215194 |
|high-school-diploma |19.57674149725407 |
+----------------------+----------------------+

You can see that there is no discernible difference in this particular sample between education level and number of books borrowed. The library members don’t change their borrowing behavior (at least in terms of magnitude) based on this feature. The clamping range might also be high, so you could take a look at updating that in future queries.

Let’s see if folks borrow differently based on an intersection of education and age. First, you will need to create age bins, so you can group by them:

age_binspec = BinningSpec([10*i for i in range(0, 11)])

age_bin_keys = KeySet.from_dict({
"age_binned": age_binspec.bins()
})

binned_age_with_filter_query = QueryBuilder("members")\
.filter("education_level='masters-degree' or education_level='doctorate-professional'")\
.bin_column("age", age_binspec)\
.groupby(age_bin_keys)\
.average("books_borrowed", low=0, high=22)

session.evaluate(binned_age_with_filter_query,
privacy_budget=PureDPBudget(0.4)).show(truncate=False)

When I run this code, it outputs:

+----------+----------------------+
|binned_age|books_borrowed_average|
+----------+----------------------+
|100-109 |-2.0 |
|80-89 |11.476923076923077 |
|40-49 |11.034418604651163 |
|30-39 |11.501822600243013 |
|70-79 |11.256830601092895 |
|20-29 |11.08816705336427 |
|50-59 |11.599250936329588 |
|10-19 |14.0 |
|90-99 |-24.0 |
|60-69 |10.970472440944881 |
|0-9 |19.0 |
+----------+----------------------+

Oye! You can see that there is a lot of differntial privacy noise added to some of these columns. What did we do wrong? In this case, the code filtered on age and did not take into account that some of the age groups represented would likely be underrepresented in the filter. The likelihood that a 8-year-old has a masters degree is quite small.

This is a great reminder: Thinking through your data and clarifying your queries, hypotheses and planning out your analysis before you run any code is required when working with differential privacy. In this case, you could have run a query early to get an idea of the age ranges, or run the map and shown what age ranges are present (without looking at counts), so that you have a base understanding in order to write the queries beforehand and focus on what you are trying to learn.

You can see the rest of the example queries with this dataset and library in the book’s repository, and experiment with the library on your own dataset. For times where you are just experimenting, you can set your privacy budget to infinity! But in reality, you will need to set an epsilon and determine how to split your budget. If you see an error similar to this one:

RuntimeError: Cannot answer query without exceeding privacy budget:
it needs approximately 0.100, but the remaining budget
is approximately 0.000 (difference: 1.000e-01)

you know that you’ve allocated too little budget to answer the query.

I highly recommend using an example dataset and getting to know the capabilities of the Tumult library, or any well-reviewed open-sourced differential privacy library! Tumult isn’t the only library that works with Spark, you can also use PipelineDP, which is maintained by some Google team members alongside the OpenMined community. They also have several examples in their GitHub repository.

Warning

Not every library has what are called end-to-end differential privacy guarantees, meaning that budget tracking and allocation becomes a difficult and error-prone task. If you are getting started with differential privacy, I recommend sticking with libraries that will manage the budget for you!

As you experienced in the prior chapter, building your own differential privacy mechanism and tracking the privacy budget can be a lot of work and fraught with edge cases and errors. It’s wonderful that we are now entering an era when open-source libraries can support this work. Now that you’ve seen the internals of building your own, you can also more easily understand how to spend your budget accordingly and ensure that the privacy guarantees are well understood for the data participants. You can also inform downstream recipients of the data about the processing that it has gone through in order for them to better understand and address the relative error.

If you are thinking of inserting differential privacy at the collection point, there are actually quite a few different ways to approach this. Let’s dive into anonymization in data collection!

Collecting Data Anonymously

Now that we are able to use differential privacy libraries (that are open source and audited!) in our pipelines, we can also then think about collecting data in an anonymized way. This brings several new considerations, such as, how do I know sensitivity of data which I do not have access to? As well as, what privacy guarantees can we offer when we move anonymization to the collection?

To investigate what is possible, let’s see how Apple implemented their initial privacy-aware sampling of emojis and see how local differential privacy works. Then, we’ll take a look at a method researched and used at Google for aggregating data in a differentially private manner. After, we’ll look at RAPPOR, which was an initial differentially private data collection method for Chrome browsers.

Apple’s Differentially Private Data Collection

Apple was one of the first companies to deploy differential privacy at internet scale. They have a team of scientists and engineers who work on privacy systems and have decided, as a brand and company, that privacy is a major product differentiator.

Apple wanted to improve their emoji suggestions, but didn’t want to send private text data to their servers in order to learn what emojis occurred near what text. They determined that differential privacy applied to emoji frequency and neighboring tokens would solve the problem and went about implementing it for iOS devices.

In the initial paper released about their differential privacy mechanisms they included several useful graphics that help us reason about differential privacy on devices and at the beginning of data collection. Unlike the privacy-preserving data collection mechanisms shown in the beginning of this chapter, the mechanism Apple implemented is actually done so on the device itself, meaning the data is sent to Apple servers after the differential privacy mechanism is applied.

Figure 3-3. Apple’s Emoji Use Data Collection with Local Differential Privacy

As shown in Figure 3-3, the specified noise is added to the emoji on the actual device. In fact, if you have an iOS device, you can look at the epsilon values on your phone by going to Settings > Privacy > Analytics and looking at Analytics Data in entries that begin with “DifferentialPrivacy.” After noise is added, the differentially-private emoji is sent to Apple servers where the IP and origin information is dropped and the results are aggregated to produce a statistical distribution, which can then be de-noised due to it’s scale and the differential privacy parameters used to get results. In the paper, they show the entire pipeline and results using the graphics. We can see sections of it in Figure 3-3 and 3-4.

In Figure 3-4, we see that the initial differential privacy mechanism is applied on the device. When the anonymized data is then sent to the server, the IP address is dropped so that no one can trace the data directly to a user (which would immediately counter all privacy guarantees we gained from differential privacy!). Internally, Apple uses an ingestor and an aggregator, which transforms the differentially private data and aggregates it for analytics use. Finally, they have a internal dashboard that is then used by the data team and application developers to gather insights from the data. Each of the dotted lines represents a trust boundary, where if the processing is violated or if an internal employee can trace the data, it would nullify the privacy guarantees.

Figure 3-4. Apple’s Emoji Data Collection Pipeline

Note

What is a trust boundary? You can think of this as a border where security changes in a fundamental way. Data or computer instructions that cross trust boundaries are fundamentally altering the security guarantees. For example, when a user enters data into a web application form and sends it to the application server, it crosses a trust boundary before a response can be given. Usually this involves insuring that the request is properly formulated, validated and contains no malicious commands (such as a SQL injection attack). Only when the data encapsulated in the form and the request itself is “trusted”, will it then be worked on by the application, which might need to then talk to a database and formulate a response. Trust boundaries are a fundamental concept in security and software architecture and you’ll be learning more about them in Chapter 9!

Once the pipeline is done, the resulting analytics are interpretable. This means each user has culpable deniability of which emojis they use but the keyboard team now has information on how to better deliver emojis for different keyboard languages. The analytics they showed in the paper are in Figure 3-5, which show that there is a difference in the third most popular emoji (and the following most popular emojis) for French and English keyboard users.

Figure 3-5. Differentially Private Keyboard Emoji Use Analytics

This approach might be useful if you have some simple sketches or distributions that you want to learn and you have device-based data collection. One thing to note, which we will cover in the RAPPOR section, is that this is really only feasible at large scale, due to the amount of noise inserted on the individual level.

To explore more of Apple’s differential privacy research, take a look at their latest research.

What about other ways to collect data with respect to privacy? Google developed an approach that has slightly less noise insertion, which we’ll explore as another potential use case we can learn from.

Google’s Differential Privacy via Separation of Responsibility: Encode, Shuffle, Analyze

In another application of differential privacy, Google released a paper in 2017 covering how differential privacy was achieved through a very small addition of noise during the process and a clear separation of responsibilities. This was possible due to the way the data was collected and analyzed, as well as how the compute infrastructure at Google is set up. Although most of us don’t manage Google-scale infrastructure and data, I think it’s a useful example to think about differential privacy at a higher level and to see how you might explore new ways of reasoning about privacy guarantees.

Let’s take a look at how the Prochlo method works to understand how it can offer certain guarantees, as well as some weaknesses in the approach. Figure 3-6 shows the different steps in the process in a graphical form (encode on the left, shuffle in the middle and analyze on the right).

Figure 3-6. Prochlo: Encode-Shuffle-Analyze

Encoders: Encoders work on the user’s local device and apply one or more of several methods to add obfuscation of the data, including a breaking sequences into separate sets of actions, removing specific user details and instead ascribing them to a larger group of users as well as other methods like coarsing, scrubbing and adding a very small amount of noise in comparison to what normal local differential privacy uses (see: RAPPOR method which is up next in this chapter). The data is encrypted twice and then sent from the device to a server, crossing it’s first trust boundary.

Shufflers: Shufflers work as the receivers of data from the endpoints. A primary role of the shufflers is to anonymize “who and where” the information came from. This provides the differential privacy guarantees without the randomized response mechanism, so long as the shufflers do not share or store that information in any way that it could be used later. This information would be the network metadata (origin, IP address, user client information, etc) as well as any particular user information in the data itself. The shuffler also must insert timing randomness, as this can leak private information. The shuffler can also act as a filter, only letting values through where a particular threshhold has been reached. The first layer of encryption is removed and the data is grouped. When the threshhold is passed, it is then sent along to the analyzers, crossing another trust boundary.

Analyzers: Analyzers take the now scrubbed and stripped data, decrypt the final layer of encryption and prepare it for analysis. This can mean simple aggregation. Depending on the processing beforehand, this might also use a differential privacy mechanism, but one that no longer calculated budget on an individual basis (i.e. central differential privacy mechanisms).

Great! So, we can get differential private data collection with better accuracy than local differential privacy. Sounds awesome! Is there a catch?

Let’s analyze the above architecture with an attacker mindset. What could go wrong and where?

When the encoder and analyzer cross the trust boundaries, these are opportunities to collude or retain more information than is promised. For example, if the analyzer and shuffler are run by the same organization, what guarantees that there are not artifacts that allow a curious employee to take a look at the activity inside the shuffler and compare it with what is seen in the analyzer? Or even an attacker who has access to the network information over the trust boundaries and either access to the analyzer or the shuffler or anything else? What if the shuffler and the analyzer collude?

This points out the important intersections of architecture choices as well as computer security and the implications it has on any privacy mechanisms we implement. If we cannot trust the network, or the hardware, or properly define our trust boundaries, then the potential to leak information increases with every risk. Especially in scenarios where we expect multiple services to never exchange extra information or pass over data that hasn’t been properly prepared with the guarantees that are expected, we must be cautious and ensure that we haven’t overlooked a potential vector for attack, and therefore privacy loss.

You might also wonder if some older approaches you’ve heard about reach the same guarantees, as the Google researchers found. Let’s review older methods to clarify the guarantees.

Why Chrome’s Original Differential Privacy Method Died

RAPPOR was one of the first successful deployments of differential privacy at scale. It was released by Google in 2014 and you can still see the implementation details and code via the open-source repository and documentation.

The goals of RAPPOR was to allow client reporting from Chrome browsers in a privacy-preserving way. To do so, the teams implemented randomized response in a library that was used to collect basic statistics in the form of bitstrings. On the machine, there might be a series of events or metrics that the Chrome developers were interested in, such as whether a particular add-on was used, if a particular error had occurred or some information on the user’s Chrome history.

RAPPOR used the following workflow to report metrics from a local machine over a longer period of time while still offering differential privacy guarantees. The guarantees are similar to the randomized response methodology, which was covered in Chapter 2.

Encoding Step: For a particular Chrome feature or value, a bit will be chosen if it is on or off. So, if we want to report 20 features, a 20-bit-long sequence will be sent and each bit will be chosen via the RAPPOR mechanism. If we want to collect categorical values, a Bloom filter is used where values are matched against a set of candidate strings. The Bloom filter will then itself go through a differential privacy mechanism before being reported.

RAPPOR Algorithm: Permanent Randomized Response: First, the client value is prepared if we are using the Bloom filter, by hashing it onto the Bloom filter. Then, the value goes through a randomized response process, which has probability 1/2f of randomly returning a 0 regardless of input, the same probability of returning a random 1 and 1-f probability of returning the actual value. The response is then saved for future use, which is why this is called the “permanent” response. To note: f is a “tunable” parameter.

RAPPOR Algorithm: Instantaneous Randomized Response: Based on the results of the permanent randomized response, a bit array will be prepared with some additional noise insertion. This is to prevent the possibility of someone learning the permanent response over time.

At this point the report is sent to the server.

The graphic from the original paper is quite instructive. Here we can see that the permanent RAPPOR response ((/B´/)) is stored, and that the instantaneous randomized response has additional noise to provide local privacy guarantees.

Figure 3-7 shows an example of the real string as well as the randomized response Bloom filter B´. Each level shows increasing noise.

Figure 3-7. RAPPOR Report Visualized

Google open sourced the project in 2014, and then stopped updating it in 2016. Why?

The relative error of randomized response in comparison to other methods in centralized differential privacy (like in Chapter 2) is quite high. So, in order to get semi-accurate results for randomized response, we must not only collect large quantities of data, but also account for large error in the data. We can compare the amount of relative error introduced in building randomized response data collection versus using a centralized model for differential privacy, where we can ensure the clamping and type of noise fits our understanding of the data as well as what we are trying to infer from our analysis.

To get differential privacy guarantees in a centralized model, we scale the noise to sensitivity/ε to the total of data points (k). To get the same guarantees for a local model you need to add error of sensitivity/ε for each user. This means that the relative error for local differential privacy scales with the users at approximately f r a c
1 s q r t k
*
Î µ
, much steeper than the relative error of a centralized model, which would be approximately f r a c
1 k
Î µ
.

This means that if we want to provide local differential privacy, we need to reason about a truly large amount of error over time. It’s likely that this was the experience of RAPPOR at Google, which is why it eventually stopped being maintained and is no longer currently used or supported.

Although this might make sense if there is “no other way”, it definitely does not make sense if there are better methods. As this history points out, working with stakeholders to properly explain the error introduced, the questions that can or cannot be answered and figuring out what fits for the long-term strategy are all parts of designing an appropriate privacy engineering system. You’ll also need to coordinate with data engineering to ensure that your privacy automations fit the expectations of the team that manages most workflow code!

Working with Data Engineering Team and Leadership

If your organization is large enough, it likely means you have separate data science and engineering teams. This is, in my opinion, a quite unfortunate development, since I see the two teams as irrevocably intertwined. If the communication between data engineers, data scientists and “data managers” breaks, then our entire ability to use the data appropriately breaks too. This doesn’t only affect data choices, but also data privacy choices. In order to ensure our data privacy methods are working for all parts of the organization, we need to ensure that leadership and the entire data engineering team understand the standards and methods that we are working with, how they affect the data and data engineering work and why they are essential.

In case your organization has hard walls between departments that need to work together on these topics, see if you can be the friendly hand over the wall. A little bit of effort here will go a long way when something goes wrong in the data work.

Share responsibility

Automating data workflows is not only the job of data engineers or data management teams. It is our job as data scientists to properly define the types of problems that are important to us and what data we need in order to answer those questions.

This means spending time as a data science team outlining business needs and goals and understanding them enough to foresee product questions that we will need data for. We don’t have to do this alone, but we do need to bring our data science expertise to anticipate potential gaps in the current data collection and bring them to the attention of other teams who manage collection and software.

Another hot topic will always be data quality, interoperability and standardization. These are part of data governance, and should be coordinated across teams regardless of organization size. To do this properly, organizations should have a governance committee or guild and regularly update the standards and guidelines based on community feedback.

In order to check that the standards and guidelines are in place and to ensure the data is actually meeting the needs of all parties, you’ll need to regularly test and review the data! The last thing you want to learn is that a field you asked added has not actually been collected for the past 6 months, or that the format of another field is indescipherable and not what you needed. This means working directly with the data engineering team to write out expectations (using a library that can help you measure these over time, like Great Expectations, dbt or others!).

Finally, to keep larger organizations aligned and communicating, document your work. Let’s cover how to do this for privacy workflows.

Create workflows with documentation and privacy

Well documented systems go a long way. This means if you are already working in a shared repository and jobs are clearly documented as a part of this repository, adding a section to this documentation that outlines privacy requirements and recommendations should be easy!

If you are just getting started with proper documentation, ensure that you add a section for privacy measures and mitigations. Outline with some clear examples what can be used and how. Ensure code is properly versioned and shared across the team so that people are not working in silos. Encourage thorough pull-request reviews and put “Privacy Measures Enabled” as a step towards your Definitions of Done.

If you have also a few good examples of workflows that have added proper privacy measures, link to them or give a short internal talk on how you did it. Often folks just need a few good examples to get started and a point-person to whom they can ask questions if they arise.

Privacy as Core Value Proposition

Your life will be a lot easier if privacy is seen by numerous team members (or by the whole company) as a core value proposition. To pitch this idea if it is new, try thinking about what parts of your product or offering touch upon privacy topics, and what the value of integrating privacy would be to your product overall.

If you deal with consumer data, then it’s likely quite obvious. Offering your customers more privacy guarantees goes a long way in establishing trust and creating a brand that people see in a good light. But if you work in business-to-business or internal data, what are you offering?

It could be that working with internal data in a privacy-respecting manner means that there is less invasion of privacy for your coworkers and yourself. It could also mean that the business clients whose data you manage appreciate the extra work put in to make privacy by design possible. And finally, it could be that the company or organization itself values these topics as a normal part of everyday business.

Building a culture of privacy, where the questions raised in this book become a natural part of conversations about data is a multi-year journey. Similar to building a champion culture for security, there will be some folks who immediately take to the topic (like yourself, perhaps?) and that bring it to the attention of others.

Tip

The security community has done a great job of thinking through how to build cultures inside organizations. Developing bridges with the security experts at your organization and learning how they build security into the culture and foster champions across the organization will be very helpful in determining how to do the same for data privacy!

For additional tips on building a culture around privacy by learning more about security techniques, I recommend reading Chapter 15 of Agile Application Security by Laura Bell, et al (O’Reilly, 2017)..

The end goal is to ensure that enough folks are having a look at these topics from enough perspectives. It doesn’t have to be everyone, but it should be enough that things don’t “slip through the cracks” and end up deployed or near deployment without someone considering the privacy questions.

Summary

In this chapter, you’ve learned how to automate privacy in pipelines by first analyzing the privacy needs of the system and finding good places to build privacy in. Congratulations! This is a major part of engineering privacy into your data workflows and products!

In doing so, you’ve learned how to design workflows with privacy in mind. You’ve also implemented a small piece of privacy and data governance technology at several different parts in the pipeline. You’ve had a chance to experiment with a few different differential privacy libraries that you can integrate with your systems. You’ve learned a bit more about local differential privacy and different collection mechanisms that enable collecting data more anonymously. You’ve also learned how to collaborate across teams and the organization to bring collaborative privacy and more privacy awareness to the teams responsible for moving and managing data workflows.

In the next chapter, we’re diving into privacy attacks — getting to know better exactly what things we are trying to protect against and solidifying our understanding of the risk!

[1] If you don’t normally work in data engineering or infrastructure, I recommend learning some of the basics by reviewing Data Pipeline Architecture books, videos or blog posts.

[2] A smell test in computer science means assessing if the code quality is high. You want to assess if the privacy standards are being properly enforced and keeping the privacy standards properly distributed across the organization and many data flows.

[3] This is, perhaps, not a great idea for privacy, but is common practice. When presented with these cases in your work, you’ll need to decide when to push back and ask a few more Whys and when it is best to just move forward and enable privacy protections as you go. In my opinion, these questions can often be answered in aggregate data (which might even benefit from adding differential privacy, depending on the audience and how widely it will be shared!).

[4] I can recommend reading through Vitaly Friedman’s Privacy UX Experience on Smashing Magazine and consulting with your UX team to determine potential changes for your organization.

About the Author

Katharine Jarmul is a privacy activist, machine learning engineer, and founder of kjamistan, a data science consulting company. She is also a passionate and internationally recognized data scientist, programmer, and lecturer. Previously, Katharine held numerous roles at large companies and startups in the US and Germany, implementing data processing and machine learning systems with a focus on reliability, testability, privacy and security. She is an O’Reilly author and a frequent keynote speaker at international software and AI conferences.

For the past five years, Katharine has focused on answering the question: How do we perform privacy-aware data science and machine learning? To answer this question, she’s worked on the legal and technical aspects of regulations like GDPR, as well as helped build an encrypted learning platform based on multi-party computation.

Table of Contents

- Preface

- Who Should Read this Book

- Privacy Engineering

- Why I Wrote This Book
- Navigating This Book
- Conventions Used in This Book
- Using Code Examples
- O’Reilly Online Learning
- How to Contact Us

- 1. Data Governance and Simple Privacy Approaches

- Data Governance: What is it?

- Identifying Sensitive Data

- Identifying PII

- Documenting Data for Use

- Finding and Documenting Unknown Data
- Basic Data Documentation
- Documenting Data Collection
- Documenting Data Quality
- Documenting Data Security
- Documenting Data Privacy
- Documenting Data Descriptions
- Documenting Data Statistics
- Tracking Data Lineage
- Data Version Control

- Basic Privacy: Pseudonymization for Privacy by Design

- Summary

- 2. Anonymization

- What is anonymization?

- Defining Differential Privacy

- Understanding Epsilon: What is privacy loss?

- What Differential Privacy Guarantees, and What it Doesn’t

- Understanding Differential Privacy

- Differential Privacy in Practice: Anonymizing the US Census

- Differential Privacy with the Laplace Mechanism

- Differential Privacy with Laplace: A Naive Attempt

- Sensitivity and Error

- Privacy Budgets & Composition

- Exploring Other Mechanisms: Gaussian Noise for Differential Privacy

- Comparing Laplace and Gaussian noise

- Real-World Differential Privacy: Debiasing Noisy Results

- Differential Privacy: A more nuanced definition

- What about k-anonymity?

- Summary

- 3. Building Privacy into Pipelines

- How to Build Privacy into Pipelines

- Design Appropriate Privacy Measures
- Meet the User Where They Are
- Engineer Privacy In
- Test and Verify

- Engineering Privacy and Data Governance into Pipelines

- An Example Data Sharing Workflow

- Adding Provenance and Consent Information to Collection

- Using Differential Privacy Libraries in Pipelines

- Collecting Data Anonymously

- Apple’s Differentially Private Data Collection

- Google’s Differential Privacy via Separation of Responsibility: Encode, Shuffle, Analyze

- Why Chrome’s Original Differential Privacy Method Died

- Working with Data Engineering Team and Leadership

- Share responsibility

- Create workflows with documentation and privacy

- Privacy as Core Value Proposition

- Summary

- About the Author