 is an essential concept in software development. Version control tools, such as 

, are used to manage and track source codes. Developers and organizations alike value source code because it is the backbone of their product(s), from which they earn revenue. 

 is the application of version control concepts within the machine learning process.

Machine learning and artificial intelligence offer an edge to businesses and developers over their competitors, motivating the competition to push to build the best model as it provides a business advantage. As developers optimize for speed and accuracy, minimizing errors also becomes a priority. Model versioning is a great tool for all of those purposes.

In this article, you’ll learn more about the importance of model versioning in machine learning by exploring a demo on how to apply it when building ML models and reviewing best practices to follow, limitations to be aware of, and some of the best tools currently available.

Model versioning derives its core processes from version control in software development. The difference is that software development runs mainly on code, while machine learning runs on both code and data, which is the backbone of the best-performing models.

Model versioning, therefore, is the process of tracking and managing changes made to source code and data, including metrics, parameters, and hyperparameters. Popular 

 have failed to track changes made to data and are limited in the amount of data they’re able to host due to storage limits. Solutions such as 

 (Git-LFS) can accept a large amount of data but still fail to track data and models.

Applying model versioning when building machine learning applications is important because it allows:

referencing previous versions of the model;

reverting changes quickly, making building models less risky;

easily reproducing and sharing models among development teams.

ML models are generally built by developers across a variety of teams, which requires a high level of collaboration. Model versioning then becomes necessary as it permits teams to do the following:

 Building machine learning models is nondeterministic. In other words, training the same algorithm on the same data, as done in the past, can produce a different result due to the change in parameters and hyperparameters. Tracking data and everything involved in the model-building process becomes possible with model versioning, allowing for the convenient reproduction of models.

 Working on machine learning projects requires collaborative efforts depending on the size and complexity of the project. Model versioning allows teams to share models by saving files in a remote storage location and recording/tracking each file that produced a model. Teams can constantly update and share models among themselves.

 Industries that deal with sensitive information, such as healthcare, need data governance to ensure that data is handled with care and that it remains consistent and accurate. Model versioning allows for proper model auditing to ensure compliance with government laws like the GDPR.

In order to prevent having bugs in production, which can break the system, here are some best practices to follow:

 Regularly reviewing your model will help you spot errors that could otherwise cause it to break or produce inaccurate predictions. Be sure to keep an eye out for outliers and data points that can weaken or change your model’s performance.

Regularly delete models that are not in use:

 Keeping obsolete models uses up space and causes redundancy. Deleting these models provides clarity and ensures that your resources are invested only in models that are currently in use.

 Testing models before deployment is crucial to help you identify errors that may otherwise lead to downtime, operation costs, and for a commercial product, loss of users.

Model versioning is still in its early stages, with tools like DVC and MLflow made available for public use in 

, respectively, compared to Git, which was built in 

. Thus, tools used for versioning—especially open source tools—are not yet built to support 

 model versioning. For instance, some prioritize tracking model changes over robust storage options (or vice versa). Similarly, some model versioning tools come with pipeline management, leading to data redundancy in cases where your team already has a pipeline system in place.

To overcome these limitations, organizations like 

 have built their own internal versioning systems, while others opt to use third-party solutions like 

To tackle the challenges of model versioning and solve the limitations of versioning software such as Git, several open-source and enterprise solutions have been built. Here are a few useful tools to consider:

: Openlayer is a collaborative quality assurance platform for machine learning that allows you to track and version your models with ease. It helps you discover errors when building models, boost model performance, and test your model before deployment. If you’re building an end-to-end machine learning project, it may be just the right tool for you.

: Data Version Control (DVC) is an open-source tool used for versioning data sets. DVC commands are similar to Git’s, and both are used in tandem when building models. DVC versions large data sets, while Git stores the .dvc files used to retrieve data from DVC remote storage. DVC supports popular remote storage options like Amazon S3 and Google Cloud Storage and is a great open-source option to version your data.

: LakeFS is an open-source platform similar to DVC that provides Git-like commands for performing model versioning operations. It also integrates storage options like AWS and GCP to store data. LakeFS minimizes data duplication via a copy-on-write mechanism and is well suited for enterprise use due to its high performance over data lakes of any size.

Model versioning is a valuable part of the development process for machine learning projects because of the need for collaboration, tracking changing codes, and monitoring the model’s performance over time. If you’re considering what tool might be right for you and your team, here are a few guidelines to keep in mind. The ideal tool will have the following features:

 Model versioning tools should be easy for you and your team to get familiar with. Machine learning is complex enough without these tools—using them should make building models simpler, not more complicated.

 Whatever tool you choose to use should integrate well with your tech stack for smooth building. Fortunately, most tools built for model versioning are language-agnostic, so finding one that meets your needs shouldn’t be difficult.

 Choosing a tool or platform that supports the kind of data you are working with ensures faster model development. Versioning tabular-based data is different from versioning audio, video, or image files. You’ll want to pick a tool that supports the data type you work with now—as well as any others you plan on working with in the future—to avoid needing to switch platforms later on.

If you’re in the market for a great ML platform that allows you to expertly version and track your models, be sure to consider 

 today to learn about how Openlayer can help you with error analysis, synthetic data generation, testing, deploying models, and more.

* A previous version of this article listed the company name as Unbox, which has since been rebranded to Openlayer.

Oghenefejiro Esosuota

Systematically iterating through ML models 

The importance of model versioning in machine learning

With the introduction of the internet in our daily lives, the amount of data available has increased hugely. Furthermore, the incremental growth of computer capabilities has allowed for the collecting and storing of this ever-increasing amount of data, opening up all kinds of possibilities for extracting information. It is in this context that data science has come to play a fundamental role.

Data science as we currently conceive it is a 

 and has evolved significantly throughout the years. It is a field closely related to statistics, data analysis, computer science, and many others—from machine learning and mathematics to domain expertise, communication, and data visualization. It’s all about processing and understanding huge amounts of data with the purpose of extracting information from it.

The volume of data created, captured, copied, and consumed worldwide

 increased from 2 zettabytes in 2010 to 79 zettabytes in 2021.

The possibility of processing that enormous quantity of data has made the role of the data scientist to be considered by some the 

. However, more recently, the shortage of data scientists has become a constraint—it’s no longer enough to rely solely upon dedicated data scientists.

Enter the citizen data scientist (CDS), an individual who participates in advanced data analytics but whose primary role exists in a related business field—not in statistics or analytics. In this article, you’ll learn more about citizen data scientists and their role in the data science process. You’ll explore distinctions in their responsibilities versus those of dedicated data scientists and review both the benefits and drawbacks of involving CDSs in your organization. Finally, you’ll learn some of the best practices that can help CDSs flourish in your workplace.

The term “big data” has been used since the ’90s as a consequence of the data explosion. This massive amount of information being stored promises the ability to extract valuable insights with the use of several convenient tools. Artificial intelligence, computer science, statistics, and other fields provide the tools and methods, while data scientists provide the knowledge to use them.

Over time, big data and artificial intelligence became an objective for commercialization. New and powerful tools appeared—like MS Azure Machine Learning Studio, Apache Spark, Amazon SageMaker, Google Cloud AutoML, IBM Watson Studio, KNIME, and so on—to simplify and commercialize the methods and strategies for dealing with big amounts of data. These platforms provide interfaces that help users apply complicated calculus to extract interesting information, visualize the results, and produce reports. They sell services that allow people without a deep knowledge of mathematics, statistics, and other sciences to apply artificial intelligence methods to their specific situations. You can also find courses and advisers on those platforms to help you learn and troubleshoot along the way.

, a citizen data scientist can be defined as “a person who creates or generates models that use advanced diagnostic analytics or predictive and prescriptive capabilities, but whose primary job function is outside the field of statistics and analytics.” CDSs are an important link between business users and data scientists.

 mentioned that engineers that have experience with statistics and modeling but whose roles don’t require math expertise often fill the role of CDS. They can extract relevant information from the analytics tools they are using and delve deeply into data, applying visualization tools to best analyze the data streams.

Businesses have realized they do not need data scientists for every single data role; other professionals with the right skill set and training are sufficient for specific tasks. Organizations can make better use of data and reduce costs by hundreds of thousands of dollars using CDSs to handle exploratory analysis, visualization, and the creation of actionable insights.

CDSs are more closely connected to the daily problems of businesses or industries than data scientists typically are, which allows them to more easily integrate the results obtained through the mathematical models to create valuable solutions.

Do dedicated data scientists still have a place?

Of course, dedicated data scientists (DSs) still have a place, as their skills are generally far more advanced than those of a CDS. CDSs serve a role between the business users and the DS doing advanced analytics.

DSs focus on the complex and specialized analysis of data for which they have studied and prepared for. They have the advanced mathematical expertise to manage the end-to-end process: exploratory data analysis, model programming, and evaluation, result interpretation and communication, and of course, deployment and maintenance.

Data scientists are still needed, but perhaps not as many of them as were previously. Nick Elprin, co-founder and CEO of 

: “You don’t go to the lifeguard if you need surgery done.” The world needs both lifeguards and doctors.

How involved in the data science process are citizen data scientists?

According to Gartner’s definition, citizen data scientists could create or generate models that use advanced diagnostic analytics or predictive and prescriptive capabilities. However, as their primary job function is outside the field of statistics and analytics, CDSs typically focus on other things like developing exploratory analysis, visualization, and providing business and industry expertise in the data science process.

Considering that the shortage of data scientists persists, in addition to the high salaries associated with their position, many companies make an effort to use CDSs as substitutes as much as possible.

, founder and CEO of DotData, “Model development, as well as model operationalization, can be significantly simplified by automation. New data science automation platforms will enable enterprises to deploy, operate, and maintain data science processes in production, with minimal efforts, helping companies maximize their AI and ML investments, and their current data team.” CDSs use such tools to accomplish the companies’ goals, taking advantage of online courses designed to provide the basic skills of a citizen data scientist, such as SQL, Tableau, Power BI, and others. Many organizations rely heavily on these automated tools to process big data and create models to gain additional insights.

What are the benefits of having citizen data scientists in your business?

As you’ve seen, citizen data scientists can be suitable substitutes for data scientists when it comes to certain tasks. There are many advantages to going with this option. First, CDSs do not require advanced mathematics and statistics knowledge; therefore, it tends to be easier to recruit them. Armed with powerful software, they solve specific business problems with a “point and click” interface, and in doing so, data preparation costs are reduced by hundreds of thousands of dollars.

CDSs also have an in-depth understanding of the relevant business problems, meaning they have a unique ability to discover insights. They usually reside in a line of business such as sales, marketing, finance, or human resources. Their business experience and awareness of business priorities allow them to provide better business expertise than many data scientists. Furthermore, CDSs can perform repetitive and redundant tasks in the analytics workflow, releasing data scientists from those tasks and thus reducing costs.

What are the limitations and drawbacks of having citizen data scientists?

CDSs often have corporate or industry work experience as well as basic computer and math skills. However, because they typically don’t have a deep knowledge of the mathematical basis and programming of the models they use, they depend on automated machine learning tools. It’s important that they receive proper education and training on these tools in order to ensure that the tools are used correctly and that errors are avoided—it would be easy to set a model incorrectly, leading to unexpected results that may be difficult to detect. Company management must enforce regulations and work procedures to help prevent such errors from occurring.

For cases where there are prepackaged software tools available to easily apply a technique without deep knowledge about it, CDSs should be able to apply the corresponding approach. When it comes to developing new strategies, though, there’s rarely a simple solution. In those cases, deeper domain knowledge is often necessary, making a dedicated data scientist a better option.

, “But for any problem that’s going to be really competitively differentiating for a business or require deep domain expertise or inventing something new, I think that’s going to be hard for citizens to attack that problem... There’s a risk of people building models where they don’t have a deep understanding of the statistical fundamental for models that have risk associated with them.”

What are the best practices that can help citizen data scientists flourish in the workplace?

For CDSs to flourish in your workplace, they need to understand the company’s needs, think outside the box, have an analytical mind, and be able to draw meaningful conclusions.

Software developers and engineers are typically good candidates for this role. It is advantageous to work in an adjacent field like backend software development or engineering due to the skills required in those positions, such as math, computer science, and coding. Otherwise, CDSs can prepare themselves by taking courses to develop skills with Tableau, KNIME, SQL, Excel, or Python.

The company should also provide flexible and secure environments that allow CDSs to work and collaborate with data scientists and data engineers. Administrations should classify data sets considering security and data-protection criteria. At the same time, they should prepare some data sets specifically for CDS training to avoid the risk of damaging anything. CDSs should analyze the results of applying different tools to the same data and then compare the results they obtain. They should be able to understand their results and their insights, use graphical interfaces to present the results, and participate in the overall analysis of the data.

CDSs should learn the basics of data science tools—especially those available in their workplace that simplify the tasks of applying machine learning to real-world problems, detecting anomalies in the output of machine learning algorithms, and following the regulations established by relevant governing bodies.

Finally, it is convenient for CDSs to know techniques such as regression analysis and predictive analytics. They should be able to tell when a model is overfitting and have some coding ability, as well as experience using actual data and knowledge of Excel and SQL.

In this article, you learned about how the internet and its big data explosion have brought many opportunities to gather valuable information. However, specialists (i.e., data scientists) are needed to extract this information from the data. As there are not enough data scientists to cover the demand, a new role has appeared: the citizen data scientist (CDS). They do not have the same expertise as a dedicated data scientist, but they are equipped to use automated machine learning tools and offer additional value in their analytical abilities and their unique understanding of the company’s needs. They do not require high salaries or even a master’s degree or Ph.D. Rather, they can be reskilled or upskilled by taking courses to train in any number of relevant areas as needed.

While using CDSs offers many advantages, there are limitations related to the depth of their understanding of mathematics, statistics, and modeling. They should be prepared to use prepacked software tools to deal with commoditized use cases, but they are not typically equipped to attack significant new problems.

For best results, administrations utilizing CDSs should provide appropriate environments for CDS to grow in, build teams that allow CDSs to collaborate with traditional data scientists, and offer courses to train them in relevant skills.

Andres Soto Villaverde

Spreading the data knowledge

Why every company needs citizen data scientists

Machine learning models, especially deep neural networks, are trained using large amounts of data. However, for many machine learning use cases, real-world data sets do not exist or are prohibitively costly to buy and label. In such scenarios, synthetic data represents an appealing, less expensive, and scalable solution.

Additionally, several real-world machine learning problems suffer from 

—that is, where the distribution of the categories of data is skewed, resulting in disproportionately fewer observations for one or more categories. Synthetic data can be used in such situations to balance out the underrepresented data and train models that generalize well in real-world settings.

 is now increasingly used for various applications, such as computer vision, image recognition, speech recognition, and time-series data, among others. In this article, you will learn about synthetic data, its benefits, and how it is generated for different use cases.

. It’s generated with algorithms as well as machine learning models to have similar statistical properties as the real-world data sets. For data-hungry deep learning models, the availability of large training data sets is a massive bottleneck that can often be solved with synthetic data.

Additionally, synthetic data can be used for myriad business problems where real-world data sets are missing or underrepresented. Several industries—like consumer tech, finance, healthcare, manufacturing, security, automotive, and robotics—are already benefiting from the use of synthetic data. It helps avoid the key bottleneck in the machine learning lifecycle of the unavailability of data and allows teams to continue developing and iterating on innovative data products.

For example, building products related to natural language processing (NLP), like search or language translation, is often problematic for low-resource languages. 

 has been successfully used to generate parallel training data for training deep learning models for neural machine translation.

Generating synthetic data for machine learning

There are several standard approaches for 

Statistical approaches based on sampling from the source data distribution

Deep neural network–based methods such as 

The choice of methods for synthetic data generation depends on the type of data to be generated, with statistical methods being more common for numerical data and deep learning methods being commonly used for unstructured data like images, text, audio, and video. In the following sections, you’ll learn more about the different types of synthetic data and then explore some techniques for generating it.

Synthetic data can be classified into different types based on their usage and the data format. Generally, it falls into one of two categories:

, where only a specific set of the training data is generated artificially

, where the entire training data set consists of synthetic data

Partially synthetic data finds its application in use cases where sensitive data needs to be replaced in the original training data set. Fully synthetic data sets are used in domains like finance and healthcare, where privacy and compliance concerns restrict the use of original data.

Popular types of synthetic data, classified according to the data type, include the following:

Synthetic media including images, audio, and video

Synthetic text finds its use in applications like language translation, content moderation, and product reviews. Synthetic images are used extensively for purposes like training self-driving cars, while synthetic audio and video data is used for applications including speech recognition, virtual assistants, and 

. Synthetic time-series data are used in financial services to represent the temporal aspect of financial data, like stock price. Finally, synthetic tabular data is used in domains like 

Generating synthetic data can be very simple, such as adding noise to data samples, and can also be highly sophisticated, requiring the use of state-of-the-art models like generative adversarial networks. In this section, you’ll review two chief methods for generating synthetic data for machine learning and deep learning applications.

In statistics, data samples can be assumed to be generated from a probability distribution with certain characteristic statistical features like mean, variance, skew, and so on. For instance, in the case of anomaly detection, one assumes that the nonanomalous samples belong to a certain statistical distribution while the anomalous or outlier samples do not correspond to this data distribution.

Consider a hypothetical machine learning example of predicting the salaries of data scientists with certain years of experience at top tech companies. In the absence of real-world salary data, which is a topic considered taboo, synthetic salary data can be generated from a distribution defined by the few real-world salary public reports on platforms like Glassdoor, LinkedIn, or Quora. This can be used by recruiters and hiring teams to benchmark their own salary levels and adjust the salary offers to new hires.

As the complexity of the data increases, statistical-sampling-based methods are not a good choice for synthetic data generation. Neural networks, especially deep neural networks, are capable of making better approximations of complex, nonlinear data like faces or speech. A neural network essentially represents a transformation from a set of inputs to a complex output, and this transformation can be applied on synthetic inputs to generate synthetic outputs. Two popular neural network architectures for generating synthetic data are variational autoencoders and generative adversarial networks, which will be discussed in detail in the next sections.

Variational autoencoders are generative models that belong to the 

 class of unsupervised models. They learn the underlying distribution of a data set and subsequently generate new data based on the learned representation.

 that learns an efficient latent representation of the source data distribution and a 

 that aims to transform this latent representation back into the original space. The advantage of using VAEs is that the quality of the generated samples can be quantified objectively using the reconstruction error between the original distribution and the output of the decoder. VAEs can be trained efficiently through an objective function that minimizes the reconstruction error.

VAEs represent a strong baseline approach for generating synthetic data. However, VAEs suffer from a few disadvantages. They are not able to learn efficient representations of heterogeneous data and are not straightforward to train and optimize. These problems can be overcome using generative adversarial networks.

 are a relatively new class of generative deep learning models. Like VAEs, GANs are based on simultaneously training two neural networks but via an adversarial process.

A generative model, G, is used to learn the latent representation of the original data set and generate samples. The discriminator model, D, is a supervised model that learns to distinguish whether a random sample came from the original data set or is generated by G. The objective of the generator G is to maximize the probability of the discriminator D, making a classification error. This adversarial training process, similar to a zero-sum game, is continued until the discriminator can no longer distinguish between the original and synthetic data samples from the generator.

GANs originally became popular for synthesizing images for a variety of 

 problems, including image recognition, text-to-image and image-to-image translation, super resolution, and so on. Recently, GANs have proven to be highly versatile and useful for generating 

In this section, you will go through a brief tutorial to generate synthetic images using deep learning methods. Specifically, two images will be segmented using a pre-trained neural network module. Then the images will be annotated to describe the objects that the model was able to identify. There are many use cases for this technique, including identifying specific objects or separating images for other purposes.

To achieve the results described above, you can use a number of different frameworks. However, for the purposes of this tutorial, you will use 

. PyTorch is an open-source machine learning framework that is similar to 

 in many ways. It has a rich ecosystem of tools and libraries specialized in computer vision, NLP, and more. PixelLib, on the other hand, is a library designed to perform image segmentation using only few lines of code. An advantage of these tools is that they are written in Python, which makes them easy to use for development.

 If you want to clone the project used in this tutorial and follow along in your own editor, you can access the GitHub repository 

To follow this tutorial, you’ll need the following:

A local machine or VM (recommended) running Ubuntu 20.04-LTS.

At least 4GB of RAM and 4GB of swap space, assuming you’re using a VM. To learn more about increasing swap in Ubuntu, check out 

 permissions and SSH access to the remote server (VM only).

 Although both PyTorch and PixelLib can run on macOS and Windows, it is best to use the recommended setup with Ubuntu 20.04-LTS, as it minimizes compatibility issues between Python libraries. Furthermore, the model that will be used in this tutorial is currently only available for Linux.

The best practice when working with projects written in Python is to use isolated virtual environments. This ensures that you won't break the base Python compiler your OS uses. That said, in this tutorial, you will use 

 since it is one of the most popular virtual environment packages for Python. Additionally, 

 will be installed to handle all dependencies and 

, and the rest of the global dependencies by running the following command:

3. Create a directory for the virtual environment. This example will create a hidden directory called 

. In this example, the name of the environment will also be 

5. Activate the virtual environment by running the following command:

You will know that the virtual environment is enabled if you see its name preceding the path on the command line, like this:

Now that you have your virtual environment up and running, you can safely install all the necessary dependencies using 

1. Start by installing PyTorch using the following command:

Although it seems trivial, this step is one of the most crucial. It's vitally important that you use specific versions of each package depending on your OS and hardware. Fortunately, there is a handy configurator at 

 that can be used to generate the proper code:

, a Python API library that assists in loading, parsing, and visualizing image annotations in 

3. Install the PixelLib library by running the following command:

At this point, you have a basic framework ready to generate synthetic images. If you haven’t already, go ahead and clone the following repository with the sample code that will be used in this tutorial:

Once cloned, you will notice a structure like the following:

 is the directory where the raw images will be stored. For this tutorial, two images will be used.

 is the directory where the processed images will be stored.

 is the directory where the pre-trained models used by PixelLib will be saved. For this tutorial, only 

Before continuing, you’ll have to download PointRed in the respective directory. To do this, navigate to the 

Now all that’s left is to analyze the code of 

As mentioned above, one of the most common use cases for ML is to generate synthetic images from one or more base images. In that sense, the framework you have installed allows real-time instance segmentation of objects in images and videos using just a few lines of code.

 using your preferred text editor. You will see the following code:

Let's analyze what the code shown above does:

 variable is declared to abbreviate the code that follows.

. The model to be used for image segmentation is invoked. In this case, it is PointRed.

 function is called to perform the segmentation of objects in the images at the 

 location. Additionally, it's indicated that the processed images should be saved at the location 

 parameter tells PixelLib to enclose each object in a box and annotate it with its respective ID.

Before running the code, take a look at the raw images in the 

 directory. Both images were chosen to test PixelLib's ability to correctly segment and identify objects and people in each image.

 is located, and with the virtual environment still activated, run the code:

Sundeep Teki

Solving the data gap

How to generate synthetic data for machine learning projects

Data drift refers to the phenomenon where the distribution of live, real-world data differs or “drifts” from the distribution of data used to train a machine learning model. When data drift occurs, the performance of machine learning models in production degrades, resulting in inaccurate predictions. This reduction in the model’s predictive power can adversely impact the expected business value from the investment in training. If data drift is not identified in time, the machine learning model may become stale and eventually useless.

In this article, you’ll learn more about 

, exploring why and in what ways it occurs, its impact, and how it can be mitigated and prevented.

Machine learning models operate in a dynamic environment but are trained on data from a fixed, statistical distribution. Data drift can occur due to a variety of reasons, including seasonal variations, new product features, changes in customer behavior, or even rare events like the Covid-19 pandemic.

Data drift is a critical challenge for production machine learning systems. It occurs when the statistical distribution of the target or real-world data diverges significantly from the statistical properties of the data on which the model was trained. This hurts model performance on new unseen data during real-world inference, leading to inaccurate predictions, poor customer experience, and monetary and reputational costs for the business.

If undetected, data drift can cause multiple problems besides the obvious loss in model performance. It leads to greater 

 challenges and technical burden for teams such as identifying data drift, conducting root cause analysis for input features correlated with data drift, data labeling, active learning, retraining, and redeploying the updated models to production. This is a significant investment of time and resources that can be avoided if machine learning models are closely monitored and a strategy for detecting and fixing data drift is in place.

It is common to assume that a loss in model performance may be due to data drift. However, before arriving at this conclusion, it is important to assess 

. Target data distributions could change due to a new set of users, a feature or product update, or even something as simple as a bug or formatting error in the code or data. After data-quality issues are ruled out, data drift can be examined in more detail.

Fundamentally, data drift implies a change in the statistical distribution of target data from that of the training data. Thus, the simplest way to identify data drift is to compare summary statistics (like mean, variance, Kullback–Leibler divergence, and so on) of a carefully sampled subset of the target data against the training data. Other statistical measures include comparing the number of outliers in the two distributions or using the 

. Analyzing the correlations between input features and model predictions for both the data distributions can also shed some light.

Model-based machine learning techniques can also be used to identify data drift. A sample of data from the reference or training distribution can be labeled as 0, and an equivalent sample from the target distribution can be labeled as 1. Based on this input data, a simple binary classification model can be trained to discriminate between the two types of data distributions. If the model can distinguish between the two data sets, this implies that data drift is present. Alternatively, if the model fails to discriminate between the data sets, then no data drift is evident. Using a machine learning–based approach captures nonlinear relationships better and can help catch data drift where the above statistical methods might fail.

What are the different kinds of data drift?

The change in the statistical distribution between the target and training data manifests in different 

 that are observed in real-world machine learning systems:

 refers to a data drift that is correlated with a shift in the independent variables or input features. The relationship between the features and target variables is unchanged, but the change in a few features leads to covariate drift. 

 can occur due to sample selection bias and is frequently observed in nonstationary environments.

 is associated with a change in the relationship between the independent variables and the target variables. For instance, a particular machine learning model may suffer from concept drift when it is launched in a new geography where the customer behavior is markedly different from the behavioral data from the original geography that was used to train the models. Although the set of input features and the data distributions may remain the same, the model may not make any useful predictions and is rendered obsolete.

EvidentlyAI, "Machine Learning Monitoring, Part 5: Why You Should Care About Data and Concept Drift"

As shown in the image above, typical data drift can be captured by analyzing the shift in the statistical characteristics of the target data distribution relative to the reference data distribution. Statistical tests can be performed to evaluate whether the data drift is actually statistically significant or meaningful enough to intervene.

 is to continuously monitor the machine learning model in production and plot performance metrics, like precision, recall, or F1-score, over time for both the reference data as well as the target real-world data. Any significant change in the model metric that diverges beyond established performance thresholds may be indicative of data drift.

How can you mitigate and prevent data drift?

Once data drift is confirmed as real and significant after rigorous analysis and statistical tests as described above, it is important to address it sooner than later. Here are a few strategies for doing so.

 the new target data is the first step toward addressing data drift. A carefully selected batch of the test data can be sampled and sent to subject matter experts for data annotation. Thereafter, this labeled target data from the modified data distribution can be incorporated into the original data distribution to ameliorate the impact of data drift.

With newly labeled target data, the model can be retrained on data from both the original distribution and the test distribution. As the new model is now trained to recognize data from the modified target distribution, it typically does a better job in production than the original model. However, a model might need to be trained multiple times, depending on the rate of data drift, to capture the new patterns from the test data.

With repeated model retraining, the training pipeline, model architecture, and hyperparameters may remain the same, with the only difference being the change in training data. However, if data drift is not taken care of with periodic retraining, it might be prudent to train the model from scratch with a fresh approach and insights learned from efforts focused on evaluating and mitigating data drift.

The new model may be trained differently from the original one in a number of ways:

Reweighting the more recent data samples;

Conducting error analysis to identify any segments on which the model fails consistently;

Testing different model architectures depending on the machine learning use case;

Evaluating model-adaptation strategies to fit the model to a new target domain.

 of machine learning model performance is critical to keep track of the quality of the model in production. Model performance metrics like true positives, false positives, precision, recall, F1-score, and AUC-ROC curves can be periodically assessed. After thresholds for such performance metrics are carefully selected, alerts can be triggered using platforms like 

 or by using third-party-managed MLOps platforms.

Apart from output metrics, other things to monitor include any data issues or inconsistencies, bias in training data, and explainability metrics.

The phenomenon of data drift afflicts most machine learning models in production, arising from various reasons due to the dynamic nature of real-world data, seasonal trends, changes in product features, software- or data-related issues, changes in customer behavior due to new competition or legislation, and even rare black swan events like the Covid-19 pandemic.

Data drift can be of different types, depending on whether the relationship between the independent features and the target variables changes or not. This article has equipped you to know what data drift looks like and provided a list of best practices for identifying and mitigating it before it becomes a major MLOps challenge and renders the machine learning model unfit for its intended business purpose.

Openlayer named a top 100 AI company of 2022 by CB Insights!

Openlayer

Product

Use cases

Developer

Company

Blog

Blog

Model quality

Machine learning

The importance of model versioning in machine learning

Oghenefejiro Esosuota

Data science

Why every company needs citizen data scientists

Andres Soto Villaverde

Data science

Machine learning

How to generate synthetic data for machine learning projects

Sundeep Teki

Data science

Data quality

Surefire ways to identify data drift

Sundeep Teki

Model quality

Machine learning

Debugging models with the bias-variance trade-off

Gustavo Cid

Machine learning

Understanding each piece in the ML infrastructure stack

Gourav Singh Bais

Company

Openlayer named to the 2022 CB Insights AI 100 list of most innovative startups

Openlayer team

Explainability

The roads toward explainability

Gustavo Cid

Data quality

Data science

Data labeling and relabeling

Sundeep Teki

Error analysis

Explainability

Systematic error analysis

Gustavo Cid

Model quality

Data science

Baseline models

Gustavo Cid

Model quality

Data-centric

Understanding and measuring data quality

Sundeep Teki

Machine learning

Ensemble learning 101

Gustavo Cid

Data science

Machine learning

The challenge of becoming a full-stack data scientist

Roel Peters

Machine learning

NLP

The representation of meaning

Gustavo Cid

Big data

10 examples of using Python for big data analysis

Christopher Collins

Machine learning

Mental models for ML products

Gustavo Cid

Explainability

Anchor your predictions

Matheus Dias

Machine learning

Model quality

Dealing with class imbalance - Part 2

Gustavo Cid

Data-centric

Model quality

Why data-centric AI

Gustavo Cid