At every instant, massive volumes of data are being produced and, as John Naisbitt puts it, “we are drowning in information, but starved for knowledge.” To make sense of and leverage all this information, we must be equipped with the right tools.

Enter the world of machine learning and AI.

AI/ML as disciplines have the potential to give us a powerful toolkit that aids in the process of extracting meaning from this ocean of data. Not only that, but AI/ML models often thrive in such environments where the amount of data is way beyond what we, as humans, can intuitively grasp.

Unfortunately, it’s not all sunshine and rainbows.

Bridging the gap between information and knowledge is 

. Working with AI models is challenging and the whole process involves a cascade of decisions with consequences that are not completely evident at first.

For example, let’s say you have developed an ML model and the performance you observe is below what you want (and need) it to be. Which knob should you turn to boost your model’s performance?

? Should you use a different set of features or pre-process them differently? Or would it be better to change the loss that your model optimizes for?

Notice that there are far too many possibilities and the mere process of listing all of them is overwhelming.

More often than not, what happens in practice is that 

 when things don’t work out as expected. 

The model wasn’t able to capture all the complexity present in the data

 is just one variant of the verdict the models end up receiving. The envisioned solution, then, is changing the model.

Teams continuously jump between modeling approaches. From classical ML to deep learning. From deep learning to exotic combinations of models and losses someone came up with last week on 

When we stop and critically think about the above situation, it is straightforward to recognize that indeed the modeling approach is sometimes the culprit. But is it always the case?

Certainly not. This line of thinking is a symptom of a 

, currently pervasive in the field of AI/ML. Nowadays, the most popular approach to solving ML tasks is to hold the data fixed and iteratively improve the model until the desired results are achieved. But this is not the only possibility.

Stanford professor, Andrew Ng, recently launched the data-centric AI campaign. He argues that the process often followed by the industry must be transformed. According to him, a more effective approach to get the right results is the opposite: 

In this post, we explore three main advantages associated with following a data-centric approach when compared to a model-centric approach.

. Therefore, in industry, there is frequently a false notion that having more data is the solution to most ML models’ problems.

The motivation behind such a claim comes, partly, from the recent successes of deep learning, where enormous models with millions or even billions of parameters fed with massive amounts of data were able to beat the state-of-the-art in various benchmarks.

However, it is becoming increasingly clear that 

. Simply getting more data for the sake of it might do more harm than good if the additional data does not have a 

. Furthermore, not all models need to learn as many parameters as the famous neural network architectures that occupy the pole position of the benchmarks, which is the main reason they large datasets.

With a data-centric approach, the idea that good data is paramount becomes a reality. Practitioners can focus on constructing high-quality and representative datasets, abandoning the simplistic idea more data is the solution to all of their problems. As a result, more often than not, managing the data becomes simpler, as dataset size decreases.

Boost model performance one slice at a time

We want models that perform well for not only whole datasets but also for every potential edge case it might encounter out in the wild. The problem when we strive to achieve that goal, it is easy to be 

overwhelmed by the number of possibilities

A better and more realistic approach would be to increase the model’s performance one slice of data at a time.

The idea is that by focusing on one digestible chunk of the data at a time, we can understand what might be our model’s failure mores and how to address them, systematically improving the model’s performance gradually.

According to Andrew Ng, the lifecycle of data-centric AI follows these steps: train the model, 

 to identify the type of data the algorithm does poorly on, 

either get more of that data via data augmentation or additional data collection or give a more consistent definition for the data labels

The idea of improving models one slice at a time was one of the key motivations for the easy filtering and tagging features at Openlayer. By quickly filtering through the error classes that happened in our validation set and tagging them a certain way, teams can 

collaborate and focus on improving model performance on digestible chunks

For example, let’s say we have a model that predicts whether a user will churn or not based on a set of features and we upload it with a validation set to Openlayer. We can quickly notice that most of the errors occur when the model mispredicts a user will be retained. We can filter that data slice and tag. By doing so, all team members can know exactly which data slice is currently in the spotlight, not to mention the other functionalities available which are based on tags, such as 

A lot of the popular models can be seen as commodities. Using libraries such as 

, everyone is only a few lines of code away from model architectures that win competitions on 

This should be enough to make clear that 

model architecture alone doesn’t take teams very far

. Then, what differentiates the winning models in the real world?

. We forget that models can often be seen as statistical summaries of the data. That means that without good data, learning is doomed to fail.

Even though models can be seen as commodities, available to everyone, it is 

. Training datasets emerge from a messy process, particularly tailored for that specific application’s needs.

Therefore, having the right data for your application can be your 

The transition from model-centric to data-centric approaches is an undergoing paradigm shift in the field of ML. However, they are 

. With this post, you now have a few reasons to start thinking seriously about data-centric AI approaches, which are powerful yet often neglected in the industry. This is a topic that we deeply care about at Openlayer. 

* A previous version of this article listed the company name as Unbox, which has since been rebranded to Openlayer.

Gustavo Cid

The trend in AI

What is data-centric AI: 3 reasons to pay attention to it

Some machine learning (ML) models, after being deployed, shall directly affect the lives of billions of people in the most diverse situations, from quickly triaging medical images in a crowded hospital to influencing how you spend your hard-earned money online. However, even though they are used routinely, 

a lot of the models are still seen as black boxes

The question, then, is: if we do not truly understand what’s going on under the hood for these ML models, can we (or should we) even trust them?

The area of interpretability and explainability already offers us some 

, we introduced LIME (local interpretable model-agnostic explanations), which produces explanations for a particular model’s prediction. As discussed, 

 (like the ones provided by LIME) are extremely useful. They offer us a way to escape the trade-off between interpretability and predictive performance, which is pervasive in the field of ML.

LIME was a revolutionary technique that transformed the landscape of model interpretability, but it is not the only available method. Another extremely popular and powerful method is 

 (which is short for SHapley Additive exPlanations).

SHAP is built over the idea of Shapley values, a result from game theory obtained in the 50s. At first sight, the connections between cooperative games (which is the context in which Shapley values emerge) and ML explainability are not completely clear. However, there are strong links between the two, which allows us to progress towards understanding ML models’ predictions.

In this post, we start by exploring an example that illustrates an application of Shapley values in its original context. Then, we draw the parallels between game theory and ML explainability, this time with an example from ML. In the end, we present some of the implementation obstacles and point to connections between LIME and SHAP.

And hey -- if you are already familiar with SHAP and want to start using it to understand your models, feel free to head straight to 

Let’s start with an example that has nothing to do with ML to illustrate the kinds of problems Shapley values solve. At first, this might seem like a pointless digression but hang in there, this is a great exercise to develop intuition. We will expand on the example provided by 

Imagine you work in a big company, with many employees (let’s call the number of employees d). You all have worked very hard the whole year and, as a consequence, the company made $1,000,000 in profit. As a reward, management decides to distribute the whole profit as end-of-year bonuses to employees.

https://iancovert.com/blog/understanding-shap-sage/

Now, it’s time to start thinking about how to split all that money.

The $1,000,000 profit is a result of the work of all the employees, but would it be fair if management simply split it equally?

Probably not. Some people might deserve a greater share because they made key strategic decisions that contributed directly to the current profit and worked really hard. On the other hand, some people might be entitled to less, since they were basically “hanging around” telling jokes while standing close to the water fountain all day long.

A fair profit split should take into account the contributions made by each employee in different scenarios. To assess it, management came up with a very powerful function, called 

. It is a function that is capable of telling them exactly what the company’s profit would be if there was any subset of employees working there.

Mathematically, let D = {1, 2, ..., d}, be the set of all d employees, S be a subset of D representing the employees working at the company. So:

if all employees were working, we would have the initial situation we described, i.e.,

if employee #1 is the CEO and they were the only one not working, let’s say that the company’s profit would be half as much. In this case, S = {2, 3, ..., d} and 

if everyone was working, except for a couple of the new hires, the company’s profit wouldn’t change that much, so for S = {1, 2, ..., d-2}, 

if no one was working, obviously the company’s profit would be equal to zero, i.e., S = {} , 

The above examples are just meant to illustrate how powerful the function 

 can be. In fact, management can evaluate any of the possible scenarios (i.e., any of the combination of employees working and the profit in that case).

, management can calculate each employee’s Shapley values, which represent exactly how much of the profit each employee should receive. It turns out that employee 

\phi_i = \frac{1}{d}\sum_{S \subseteq D\setminus\{i\}} {d-1 \choose |S|}^{-1}  (v(S\cup \{i\}) - v(S)),

where |S| represents the cardinality of set S, i.e., the number of elements in the set.

The above formula might seem scary at first but worry not. The Shapley value for employee 

 is basically a weighted average of the marginal contributions it made to the profit in different scenarios. Notice that

is nothing more than the bump in profit due to employee 

 in the scenario where employees in the subset S are working. The summation is done over all of the possible 

 (so that it is possible to measure how much employee 

 added in profit in these cases) and the weights account to the different orderings (because the employee ordering doesn’t matter).

 for every employee, management has a fair way to distribute the $1,000,000 profits!

 in 1953, Lloyd Shapley showed that Shapley values computed in the way shown above are the only possible solution that respects a set of desirable properties. The discussion as to why this set of properties is desirable to achieve a fair distribution can be consulted in 

For game theory, the definition of a “game” is a bit looser than the one we use daily and a lot of things can be seen as a game.

The profit distribution problem we explored in the previous section is an instance of a game: the employees are the players and they cooperate to generate a payoff, a reward, that shall be distributed between them (the profit).

Now, you might be wondering: what does any of this have to do with ML?

Interestingly, it is possible to draw clear a connection between the problem that Shapley values solve, in cooperative game theory, and the problem of explainability in machine learning.

. These players cooperate to generate the 

This is how Shapley values apply to ML, giving rise to SHAP. The idea is that for a black-box ML model 

 that makes predictions based on a set of 

 features, it is possible to understand the influence each one of them had on the model’s predictions.

By doing so, we end up with the SHAP scores, 

. Due to one of the theoretical properties (often called local accuracy, in the ML community), if we add up all of the SHAP scores for a particular model’s prediction, we end up with exactly the model’s output. This is something trivial if we think about the profit distribution problem from the previous section, after all, summing up all the bonuses should result in the company’s profit, but less so for ML.

To make matters more concrete, let’s explore a practical example.

Let’s say we have a platform with paying users. Furthermore, we built a model that predicts whether a user will churn (i.e., stop using our service) or not based on a set of features, such as 

, among others. With such a model in hands, we can try to identify which users are likely to churn and take actions to avoid it.

The problem is that our model predicted some users were going to churn, when in fact, they didn’t. This made our organization waste precious resources trying to retain users that were not going to churn in the first place.

Let’s upload the trained model and the validation set to Openlayer and explore the model’s predictions with SHAP!

First, we can filter our validation set and look only at the data from the specific error class we are interested in, namely, samples for which the model predicted 

Done! Now the data displayed comes only from the error class we are interested in. When we click on a particular row, we auto-magically get the explanation for that prediction. Let’s click one of the rows shown to get the explanations with SHAP.

The colors and small numbers next to each feature are the SHAP values

. As a side-note, as it can be seen on the upper left corner of the image above (where it says “Show LIME values”), LIME is also available. In fact, that’s what we used in 

 was the feature that contributed the most to the model’s misprediction on this particular sample. It’s probably because the Age is high. The platform might not have many 92-year old users, that’s why our model thought this user specifically would churn.

Use the what-if analysis to see how our model would behave. We can simply type another value for the feature 

, right below “Comparison value”. Let’s change the age to 30 and leave the remaining features as they were, to see if the model would predict correctly in this case.

Yup! Now our model predicted the user would be retained and the Age is strongly nudging the model’s prediction in the right direction! Moreover, notice that the other features received different scores as well because we are essentially explaining a new point.

If you are still skeptical about the feasibility of the practical implementation of SHAP, you’re right, you should be. 

Two gaps need to be addressed to allow the use Shapley values

First, computing the Shapley values in the way that we presented would require the ML models to behave the way the powerful function from management did. We should be able to evaluate the model’s output for any subset of input features.

In the churn classifier example, this would mean evaluating the model’s prediction if we knew only the 

The problem is that if we train a model using all the features, we are not able, in theory, to evaluate it using only a subset of them.

The second problem is the summation we have to perform to compute the weighted average. If we have 

 terms to compute each feature’s scores.

In the churn classifier from the previous example, where there were 10 features, this is equal to 512, which is large, but not problematic at all for today’s computers.

What if we had 400 features? What about 1000?

 and can very quickly become prohibitive in practical situations.

Fortunately, the practical implementations of SHAP contain some clever workarounds for both of these issues. We won’t go into details of each in this post, to avoid getting overly technical. Let’s limit ourselves to saying that SHAP overcomes the first issue by using conditional expectations and assuming feature independence and it overcomes the second issue by cleverly using sampling.

Even though it might not seem like it at first, LIME (which we presented in a 

) and SHAP are very, very connected. In fact, they are 

both solutions to the same optimization problem

, but with slightly different choices made along the way.

This is one of the key results shown in the paper 

“A Unified Approach to Interpreting Model Predictions”

, by Scott Lundberg and Su-In Lee, which draws clear connections between various explainability methods, bundling all of them into the category of 

Again, we leave the details for a future blog post, and meanwhile, the interested reader should check out the above paper for details.

From game theory to machine learning

SHAP demystified: understand what Shapley values are and how they work

Working with machine learning (ML) models can be challenging: not only the problems are complex, but coordinating the efforts of different teams around all the activities that contribute to the development and deployment of a model 

can waste a lot of resources if not done properly

This is why organizations need processes. They must follow a clear and well-defined development pipeline, where the transition from one stage to the next is motivated by rational decisions.

From a bird’s eye view, most organizations follow the same pipeline, similar to the one shown above. However, the set of activities and prioritized objectives at each stage matter a lot. Consequently, despite their high-level similarity, teams fall somewhere within a broad spectrum, with teams shipping high-quality models that impress their users in one end and teams that blindly deploy models that fail silently in production in the other.

, two of the stages that heavily dictate to which end of the spectrum a team is closer to. Although these two stages share the goal of supporting teams to ship the best possible model, they 

First, we discuss the purpose and kinds of activities comprised in model monitoring. We then motivate the necessity of error analysis with a thought-provoking question. To make matters more concrete, we briefly go through a recent famous case that illustrates the consequences of having a model that fails in production. At the end of this post, you will find a table that compares error analysis and model monitoring side-by-side, which might help you identify gaps in your ML development pipeline.

And hey — if you are already familiar with the difference between these two stages and want to start conducting systematic error analysis, 

Model monitoring: the last line of defense

The world is constantly changing, so the scenarios and patterns that a model picked up during the training stages may, sooner or later, become outdated. Therefore, it is fundamental that models are monitored in production. If not, there is a high risk that they will start failing silently and, over time, become less and less useful as 

Monitoring is an ample field, with a wide variety of activities that strive to cover different aspects of the ML system serviced in production.

, making sure that all of the scaffolding around the model continues to work properly over time. Some examples are running checks that ensure the data pipelines are functioning correctly and checking if the data schema that the model is expecting suffered modifications.

, to verify its performance, measure health metrics, and check if the set of statistical assumptions over which the model is built is still valid. This is where ML engineers might look for

 (such as data or concept drifts) and spot model performance trends.

The above list of activities is far from exhaustive, but they illustrate a few key characteristics of monitoring. As it is possible to note, model monitoring can be a lifesaver, as it is the 

 against errors and changes in the environment that might trigger significant modifications in the system as a whole.

Although ML has idiosyncrasies that differentiate it from traditional software systems, one thing that both fields have in common is that 

the cost of fixing an error grows exponentially with time

, as shown in the chart below (modified from 

Currently, a lot of organizations spend profusely on monitoring solutions and over-rely on them to catch mistakes and ensure their models are trustworthy. Needless to say that a lot of resources end up being wasted in this process.

Wouldn’t it be better if engineers were able to 

catch errors proactively rather than retroactively

To be able to do so, they must conduct rigorous and systematic error analysis.

Error analysis is the attempt to analyze 

. It embraces the process of isolating, observing, and diagnosing erroneous ML predictions, thereby 

helping understand pockets of high and low performance of the model

 is an umbrella term that is comprised of many activities. Each activity serves a different purpose and helps practitioners not only identify the models’ possible failure modes but also construct a much more comprehensive view of model quality.

Some activities under the umbrella of error analysis include making use of 

, to understand why a model is behaving a certain way; leveraging the power of 

, to ensure model robustness and invariances under different scenarios; performing 

 to check how the model performance differs for different subgroups of the data; making use of 

 to possibly identify biases before the model is shipped; and taking 

 seriously, with reproducibility and rigor in the leading roles.

If you look at the ML development pipeline diagram we presented at the beginning, you might notice that error analysis is there, right after model training. Unfortunately, this is a stage that is 

As we pointed out earlier, being able to catch the models’ mistakes before servicing them in production has an enormous potential to save resources. Mature organizations understand that error analysis should lie at the heart of their development pipeline. The insights that arise from it serve as a compass pointing in the right direction. As Stanford professor Andrew Ng puts it, “if you do error analysis well, it will tell you what’s the most efficient use of your time to improve performance”.

Let’s go through an example to answer this question.

Zillow had become the go-to place when it came to house valuation. The 

 was an estimate of the price of a house produced by a model from a set of features provided by the user. After a while, Zillow created its iBuyer arm and 

automatically bought houses based on the Zestimate

The idea worked for some time and flipping houses relying on the Zestimate seemed to be a profitable business. Eventually, though, the scenario changed and things stopped working.

 and selling them at loss. As a consequence, it shut its iBuyer arm and said it would 

No one knows for sure who is at fault in Zillow’s iBuyer case. One thing is clear, though: ML models are deployed in dynamic environments, and simply monitoring them might not be enough to guarantee that they are working properly.

last line of defense, but it is not a silver bullet

There is often confusion between model monitoring and error analysis. Hopefully, by now you notice that although they share the goal of supporting teams ship the best possible model, they 

Below is a table that highlights some of their key differences. This is a resource that might help you identify gaps in your development pipeline.

Show me your ML development pipeline and I'll tell you who you are

Error analysis x Model monitoring: how are they different?

The journey toward building trustworthy machine learning (ML) systems begins by placing error analysis at the core of the development process. After all, ML practitioners won’t be able to avoid the mistakes their models make unless they start looking beyond aggregate metrics, such as accuracy or precision.

 models fail. It embraces the process of isolating, observing, and diagnosing erroneous ML predictions, thereby helping understand pockets of high and low performance of the model.

 that encompasses various activities. In this post, we explore some of them, motivating their importance and providing guidelines to help practitioners incorporate them into their development pipelines. More specifically, we will look into how error cohort analysis, explainability, counterfactual analysis, and systematic testing can be used to increase trust in ML.

And hey — if you are already familiar with the different activities under the umbrella of error analysis, feel free to head straight to 

 to start validating, debugging, and testing your models!

Error cohort analysis is probably one of the most powerful activities within error analysis in ML. Let's understand it with an example.

Imagine you have trained a classifier that predicts whether a user will churn or continue using your platform based on a set of features, such as age, gender, ethnicity, geography, and others. Now, you want to know your model’s performance, so you assess its accuracy on a validation set. The accuracy you obtain is equal to 90%. That’s great and you are feeling proud of your work!

The 90% accuracy, as an aggregate metric, summarizes the performance of your model across your whole validation set. It is a useful first metric to look at, but it certainly doesn’t convey the complete story of how your model behaves.

Is that accuracy sustained across different subgroups of the data? For example, how does your model perform for users aged between 25 and 35? What about for users based outside the US?

Notice that from a business perspective, the answers to these questions might be very relevant, so you need to be confident that your model is coherent enough to answer them.

What you will most likely find out is that the accuracy of your model is not uniform across different cohorts of the data. Furthermore, you may even encounter some data pockets with low accuracies and specific failure modes.

If you looked only at the aggregate metric (the accuracy, in this case), you would have a myopic view of your model’s performance and think that it was satisfactory. This is why analyzing different cohorts of the data is critical to building trust in your model and not being surprised by failure modes only after your model is serviced in production.

By looking solely at aggregate metrics, it is not possible to arrive at any answers.

Being able to produce explainable predictions lies at the center of trustworthy ML. These explanations can be global or local and each one provides a distinct perspective to practitioners and businesses.

Global explanations help reveal which features contributed the most to the (mis)predictions made by the model over a dataset. In the churn classifier from the previous section, for example, one might find out that the users’ geography is one of the most important features to predict whether a user will churn or not. Note that this information can be directly translated into business insights. A marketing team, for instance, might decide to create specific campaigns targeting the users from a particular location. It also increases the ML practitioners’ confidence that the model is behaving properly, without over-indexing to certain features.

Local explanations provide insights into the individual predictions and help practitioners get to the root cause of problematic predictions their models are making. The justifications provided by local explanations build confidence that the model is taking into consideration reasonable data to make its predictions.

For example, in a sentiment analysis task in natural language processing (NLP), where a phrase is categorized as positive or negative, if a model predicts that the sentence “I’m having a great day” is positive because of the word “having” and “day” and not because of the word “great”, is it really working?

Not really. Ideally, the model, in this case, should be predicting a positive label particularly because of the word “great”, which is a strong positive word. Predicting the right label is only half of the story.

Explainability is an active area of research. Some models are intrinsically more interpretable than others. For example, a linear regression is more interpretable than a neural network, as in the former, the model’s weights are evident and can be seen as proxies for feature importance, while in the latter, the high number of parameters and cascaded nonlinearities often make it hard to understand what’s going on. There are other approaches that are applicable to a wider set of problems, such as 

 (Local interpretable model-agnostic explanations) and 

As Marco Tulio, a researcher at Microsoft Research and one of LIME’s creators, puts it, “Understanding the reasons behind predictions is quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model.”

When used in combination with a systematic error cohort analysis, explanations can be an important step toward understanding why the model might not be performing so well for a specific subset of the data.

Will a model change its prediction if the values of a set of features are varied in an unforeseen way?

Posts from Gustavo Cid

What is data-centric AI: 3 reasons to pay attention to it

The trend in AI

SHAP demystified: understand what Shapley values are and how they work

From game theory to machine learning

Error analysis x Model monitoring: how are they different?

Show me your ML development pipeline and I'll tell you who you are

Error analysis in machine learning: going beyond predictive performance

Going beyond predictive performance

Detecting data integrity issues in machine learning

Methods and tools for data quality assurance

Baseline models demystified: a practical guide

Start simple to get far

A beginner’s guide to evaluating machine learning models beyond aggregate metrics

A high accuracy is not enough

How LIME works | Understanding in 5 steps

Leveraging how LIME works to build trustworthy ML

Debugging models with the bias-variance trade-off

Systematically boosting model performance

The roads toward explainability

Shedding light on black-box ML models

Ensemble learning 101

Stacked models, bagging, and boosting

The representation of meaning

The idea that revolutionized a whole field

Mental models for ML products

Avoid missing the forest for the trees with mental models

Dealing with class imbalance - Part 2

Evaluating models on unbalanced datasets

Dealing with class imbalance - Part 1

Learning with unbalanced datasets

3 differences between ML in production and in academia

Going beyond ML models and algorithms

Testing and its many guises

Understand the need for testing and learn three ML testing frameworks to help you ship with confidence

Model evaluation in machine learning

Understanding the true purpose of model evaluation in the quest for high-quality models

Gustavo Cid

Website

Writing Openlayer posts since

Number of posts