Baseline models play a key role in every machine learning (ML) development pipeline. Let's start with an example to understand why.

Imagine you work at a company that built an online platform that has lots of active users. You know that some users love the platform and intend to continue using it indefinitely. However, after some time, other users exit the platform to never come back, i.e., churn.

Moreover, you know that by observing some of the users’ characteristics, such as age, gender, geography, and others, you could train a model that predicts whether a given user will be retained or exit. This binary classifier can be quite useful for different teams inside the organization and hopefully, if the model is good enough, specific actions can be taken in time to retain users that were likely to churn.

You understand the problem and already have labeled data. As a data scientist or ML engineer, it’s all in your hands now!

It can be tempting to start developing a complex solution right away, but that is not what your next step should be. You need to start building a baseline model!

A good baseline model takes you a long way to build high-quality ML solutions. The problem is that frequently, one of two things happen. Either a baseline is neglected completely, or a weak baseline is found in a rush, making it useless.

In many scientific fields, progress is incremental, and ML is no different. “When we lose accurate baselines, we lose our ability to accurately measure our progress over time,” as put by Smerity in their 

In this post, we explore the use of baseline models in ML. First, we define what baseline models are. Then, we justify why they are useful as you develop ML models and provide a step-by-step to help you use baseline models right. Finally, we provide an example that illustrates what can go wrong when baseline models are neglected.

 from Carnegie Mellon University, “a baseline [model] is a simple model that provides reasonable results on a task and does not require much expertise and time to build.”

 model. It is important to start simple so that intuition can be built and you can start truly understanding the task at hand. Furthermore, working with simple models is particularly suitable to find bugs and test assumptions.

Second, the results produced by the baseline model should be 

. They are not expected to be perfect, and you will likely not use the baseline as your final model. However, if you pick any simple model without much thought, you risk wasting your time on a model that won’t help you at all.

Finally, it is important that building the baseline 

 The baseline model is a starting point that will help you tremendously in the next steps of ML development, but it is not your final solution.

If time is scarce, then why should you spend time building a model that will likely not serve as a final solution?

ML development is iterative. The insights that arise from one cycle inform the decisions of the next. Moreover, the speed at which practitioners iterate and learn through the stages of ML development highly dictates how well they transition from prototype to product.

Since baseline models are quick to develop, they quickstart practitioners on the ML development cycle. It doesn’t matter if the baseline model won’t be the final modeling approach. The insights that arise from its use are precious and can potentially save a lot of resources down the road.

Baselines also are a great way to increase the understanding of the data, find bugs, and test assumptions. As discussed in 

, data is the key ingredient behind every ML solution. The process of constructing and refining a baseline model reveals a lot to practitioners in a much more digestible environment of limited complexity. 

 go as far as to say that “a baseline takes only 10% of the time to develop, but will get us 90% of the way to achieve reasonably good results.”

Finally, baseline models help you measure progress. ML solutions should not be more complex than necessary, and the only way to assure this is by rigorously measuring progress against baseline models. We have discussed the role of baselines for model evaluation in details in our post on 

model evaluation beyond aggregate metrics

Now that you understand what are baseline models and are convinced that they are useful, it’s time to learn how to use them in practice. We break it down into five steps, but not all problems require all of them.

After you understood the problem and have a dataset to start with, the first step is often conducting exploratory data analysis (EDA). This is where you would go through your raw data to get familiar with the relationships and patterns that your solution will eventually learn to explore.

What is often also useful at this stage is performing a few back-of-the-envelope calculations that should give you lower bounds on the performance. This is often done by calculating what random (chance) performance would be.

For example, if you have a binary classification task with a perfectly balanced dataset, the chance model (which outputs labels uniformly at random) would achieve an accuracy of 50%. Therefore, any modeling approach you try should at least surpass this.

, so it is not rare to have a starting accuracy in the 80-90% range for models that only output the majority class.

You should spend some time calculating your most important performance metrics (precision, recall, F1, error rates, etc.) for a random model. Anything moving forward must surpass it.

A lot of the problems currently being solved with ML had previous rule-based solutions. Before jumping to ML, when available, you should try out these solutions based on domain expertise to gauge how they perform.

It is not guaranteed that there will exist rule-based solutions to the problem you are striving to solve. However, depending on the data type, they often come from the field of signal processing or make use of auxiliary data.

It’s time to pick a simple ML model to start. Remember, the idea is to start iterating quickly through the different stages of ML development and not to find a perfect solution.

There are mainly two aspects to consider when picking simple ML models: representation and model architecture.

The representation is related to how the data that shall be fed to the model is represented. This is less of an issue for tabular data, but if you are working with text, time series, or images, for example, there are many ways to represent them before feeding them to ML models. At this stage, favor simpler representations based on domain expertise (feature engineering) instead of learned representations (word embeddings or starting from raw convolutions).

As for the model architecture, common approaches followed by practitioners are: linear regression, logistic regression, and gradient-boosted trees. Some practitioners also start with famous pre-trained models from literature, such as 

 for natural language processing (NLP), although it harms a bit the intuition development aspect of using a baseline model.

After you started with a simple model, it’s time to push it to its limits. This means improving the baseline model performance as much as you can by testing assumptions, debugging, regularizing, tuning hyperparameters, etc.

This process will make you familiar with the data and reveal the limitations of the simple modeling approach. It will also set a strong performance baseline that you will need to surpass moving forward.

Complex models should not be built for the sake of complexity. Complex models should be used for their representational capacity. In the previous step, you’ve squeezed all the juice out of your baseline model and this should give you a decent idea about what are its limitations.

The objective now is to increase the complexity one component at a time to overcome the mapped limitations. In this process, properly conducting error analysis is key so that at every step, you are mindful of the trade-offs you are making.

. ML models can often be seen as statistical summaries of the data. Consequently, finding more data that looks a certain way is frequently preferable to increasing model complexity.

What can go wrong when baseline models are neglected?

To make matters more concrete, let’s look at an example of what can happen in practice when good baseline models are not used.

 used deep learning to forecast the spatial distribution of aftershocks of large seismic events. This is an important problem because the correct forecast of these locations can greatly help us not only better understand earthquakes but also improve the response to such events.

The paper used a 13k parameter neural network which, allegedly, greatly improved the performance if compared to classical estimation methods.

The issue is that a year later, in 2019, 

 showed that the same results could be achieved with a 2-parameter logistic regression. According to the latter, “this demonstrates that so far the proposed deep learning strategy does not provide any new insight (predictive or inferential) in this domain.”

 comparing the performance of the neural net and logistic regression.)

This illustrates the importance of using good baseline models. Surpassing a weak baseline is useless, as it doesn’t indicate progress nor helps us assess if the modeling approach followed is adequate.

The rapid progress in the field makes it tempting to jump straight to complex modeling approaches. However, these approaches are often hard to debug, hard to interpret, and are an overkill for a lot of the problems. This is why it is important to keep both feet on the ground and start with baseline models.

Nowadays, there are a few consolidated models that are known to provide good results in practice. For example, as pointed out by 

, a senior data scientist, it can be much more productive to spend time on the data than on model architecture after starting out with a baseline:

Progress in ML is incremental, so it is key that baselines are used to measure if we are moving in the right direction with our ML models. Remember, a good baseline model gets you far and a weak baseline is meaningless.

Gustavo Cid

Start simple to get far

Baseline models demystified: a practical guide

, and as part of our new identity, we released a new website, 

While developing the new website and app, we found a few techniques that helped us build faster using Next.js + Vercel, Sanity, and Emotion. We are big fans of the Next.js and Vercel stack, and you can read more about our experience in 

In this post, we want to share what we learned along the way, from design to implementation, performance to SEO.

We use Next.js as a framework for both our website and app to rapidly create performant and scalable UI. Next.js comes with built-in support for all the common features needed for any full-stack application. It makes it simple and quick to handle everything from routing and data-fetching to performance and SEO optimizations.

One of the benefits of Next.js is its built-in filesystem-based routing. Every file or folder in the 

 directory becomes a route on the server. For each page, you can optionally export a function 

 at the file level to fetch any data on the frontend server from your backend API (which, by the way, you can similarly create with Next by adding routes to 

We decided, however, to omit any server-side rendering and instead fetch our data on the client after the request is fulfilled. This is recommended if server-side requests are not 

 in order to bring the initial page load down to near zero.

Another decision we made was to separate components from logic as much as possible. For example, we placed a lot of our logic into a separate 

 directory. This has the benefit of readability as well as testability - these functions are decoupled and able to be easily unit-tested.

There are many optimizations that Next.js and Vercel provide that are worth mentioning, but we feel that the best features are image optimization, client-side navigation, and deployment previews.

Image optimization allows for static images to be optimized based on device size. Any images that we placed in the 

 folder is rendered as multiple images of different sizes for mobile, tablet, and desktop screens, making content delivery faster for smaller devices.

For images that cannot be included in the 

 folder (like images from a CMS), we can still optimize them for mobile screens using Sanity's 

For any link between pages within our app, we can use the 

. Next.js will then transition between pages using only Javascript, without sending a new request. This allows for instant page transitions without losing the traditional page structure needed for search engine optimization (SEO).

And the results are dramatic. Since most websites on the web do not use this technique, the difference in speed is very noticeable, and it makes the site feel much leaner as a result. We get the performance of a single-page app (SPA) with the visibility of a traditional site.

Vercel offers deployment previews on every PR pushed to your Github repo, even on the free tier. This feature is perfect for teams that need to collaborate on the design and implementation of a website before it goes live.

It allows teammates to post comments at specific pages and locations within the site. You can even tag each other or add images for context. As a dev tool, you can mark comments as resolved once they are finished, effectively serving as a shared to-do list for team feedback.

For even quicker feedback, Vercel also ships with the 

 command which can launch a one-off deployment from the code in your current directory.

This allows developers to test out their changes on a mobile phone, for example, before pushing to the remote repo.

Tools like these are a superpower for collaborative feedback, dramatically reducing the time to fix bugs and ensuring a seamless user experience before launching a public site.

, which is a popular content management system (CMS) that comes with its own online editing tool. This allows writers to add posts to the site without touching any code.

With content management comes the quest of caching. Does each page visit send a separate request to the CMS, or can it be cached in the backend? Since the blog only updates each time we add or update a post, it does not make sense to re-render the entire blog index on each request.

Instead, Next.js provides something called 

 (ISR), which lets us customize caching. By adding the “revalidate” property to getStaticProps, we can tell Next.js that we want our page to be "static,” meaning rendered before user requests, but also “incremental,” meaning updated every so often to keep it fresh. In our case, this update happens at most every 10 seconds. To a user visiting the page, there will be no additional backend delay due to the pre-rendering. But our blog will still reflect the latest content in Sanity (save for a small, acceptable delay).

 markup and styles. This is a natural tool when writing React code, since your layout is in Javascript (or Typescript, in our case). It also encourages 

, two tools to reduce the complexity of code.

). While developing, we can open the two files side-by-side to easily get a picture of all the relevant details for that component. Another win for readability and developer speed!

With Emotion, we try to use inheritance patterns as much as possible. We create several base components, from foundational flex or grid containers to more complex components that can be easily extended as needed.

One big benefit of using Emotion is the performance improvements when organizing our styles into modular pieces. Emotion will automatically cache reusable styled components into a separate class name (something like 

). Anywhere that we reuse that style, its class is already stored by the browser for quick retrieval.

Of course, these hashes tell us nothing about where the style came from or what it is doing, so we use the 

 during development to append the folder and filename.

This makes visual debugging from the browser’s web inspector much faster.

Another benefit of using Emotion with Typescript is that it gives us Intellisense on CSS properties.

 (as opposed to CSS strings), each style block is a Javascript object, so the keys and values can be statically typed. This speeds up development because we get feedback on typos from within the editor, before we render in the browser. Small improvements like this add up over time.

, which means that all styled components can take an implicit theme property without having to explicitly pass it through from a parent component, like this:

To enable themes, we just need to wrap our app in a ThemeProvider:

Since our mobile site had a lot of common markup with the desktop version, it made sense to also include an "isMobile" property to our theme. We could simply update the Theme type like so:

Now, all our styles have access to this information, without having to pass an additional prop throughout the tree:

Whether you are creating a new website from scratch or reworking a legacy site, you want to be able to build fast. It is natural to leave optimization for a later time or to skip some industry best practices. But knowing about the best tools and techniques beforehand can make development faster in the long run. There are many small decisions that add up to huge savings when it comes to getting fast feedback or reducing the cost of maintenance.

Using popular tools like Next.js + Vercel, Sanity, and Emotion are a great way to bring in some of these best practices from the start, since they are all developed by web experts with many years of experience in the field. We also highly recommend 

 for state and network query management, but more on that for a later post about the stack behind our 

All of these tools are essentially free to start using now, so give them a whirl if you aren’t already! If you want to see some more industry-leading tools in action, the 

We hope you found this post useful in your own development. Feel free to reach out to us on social media with any comments or feedback. We’d love to hear what you think!

Rex Garland

Developing fast and reliably

How we built our public site with Next.js, Sanity and Emotion

How do you decide if you have a high-quality machine learning (ML) model in your hands?

The insights that help you answer this question should come from the model evaluation stage. However, more often than not, people interpret “model evaluation” as a synonym for computing overall aggregate metrics on a validation set (such as accuracy, or F1).

The problem is that seeing aggregate metrics as proxies for model quality can be very misleading. In this post, we will show three ways you can evaluate models beyond aggregate metrics.

We could spend all day presenting the deceptions of relying too much on overall aggregate metrics. Model evaluation should assess model quality, and the quality of a model goes way beyond a single number. However, in this post, we will focus on three main problems and how each can be overcome.

The first problem of relying too much on aggregate metrics is that they are hard to interpret in isolation. This often leads to misinterpretation of model capabilities and difficulty getting management buy-in — since aggregate metrics are usually not directly tied to business value.

The second is that overall aggregate metrics provide a very low-resolution image of model performance. Consequently, model evaluation overly based on them is prone to selecting models that will, later, fail silently in production.

Finally, aggregate metrics tell us nothing about 

 our models perform the way they do. A team that only looks at aggregate metrics might be deceived by a model that performs well but, in fact, relies on spurious data from the training set to make its predictions.

To overcome these problems, one must expand their model evaluation process. That’s what we will explore in the next section.

Model evaluation beyond aggregate metrics

Imagine you are building an ML model that predicts house prices. After training the model, you obtain a mean squared error (MSE) of 120,000 dollars^2 on the validation set.

Now you need to decide: is such an MSE good or not?

Note how answering one of the main questions from model evaluation is not as trivial as it seems. Even for a standard metric such as the MSE, it is impossible to answer this question by just staring at a single number. More context is needed.

Benchmarks are one piece of this additional context. To adequately evaluate models, we should strive to iteratively compare their performance to benchmarks, which serve as goalposts along the way. They help us ground the evaluation process and ensure we are moving in the right direction.

Back to our example, if the model you are developing is expected to replace an existing system (ML powered or not), the natural benchmark would be the performance of such a system. As you progress in the model evaluation process, you should often ask yourself if the current model surpasses your benchmark. Otherwise, you risk pouring energy into something that ultimately won’t be used.

Alternatively, if you are developing the model from scratch, it is important to start with simple benchmarks and progressively increase their complexity over time. Usually, a good place to start is with the performance of the random (chance) model. Then, moving to rule-based estimates. To simple models using a subset of features. And so on. This is similar to the approach we explored in detail in our 

Benchmarks are also useful during the last mile of model development. At that stage, it is common to discuss the results from model evaluation with people from a business background. Using the standard ML aggregate metrics might not be the best choice to convince them of how great your model is. First, because most aggregate metrics are not intuitive. Second, they might not be directly tied to business metrics, which is what a company cares about at the end of the day.

In these situations, well-crafted benchmarks that are easy to compute and understand are extremely valuable. How these benchmarks look will vary greatly between problems, and they can even be the human performance for the task at hand. After all, the argument in favor of a model that surpasses the performance of real estate agents is much stronger than any MSE score in our example.

Aggregate metrics are convenient because they summarize a lot of information into a single number. However, as a consequence, they provide a low-resolution picture of what’s going on with our model performance.

Hidden behind a high accuracy, for example, are usually many data cohorts where our model performance is not so stellar. Model evaluation based mostly on overall aggregate metrics is prone to choosing models that fail silently on some data pockets.

Consider a churn classifier that predicts whether a user is likely to churn from demographic and usage data. The code for this example is 

. The resulting model’s accuracy on the validation set is equal to 0.71.

Now, if we start to dig deeper and explore different data cohorts, we see that there are underperforming subpopulations. For example, if we evaluate our model on the fraction of the dataset where the feature 

, we notice that the performance is equal to 0.52. This is significantly lower than the overall accuracy.

There are also many other data cohorts defined by feature value combinations that result in high error rates, as shown in the image below. In this case, we are using an algorithm to automatically find the data cohorts that might be problematic for the model.

Identifying such subpopulations is critical to evaluate models. Otherwise, our view of model performance can be myopic.

ML models are trained to pick up patterns in the training data. The problem is that datasets contain good and bad patterns. Signal as well as noise. Thus, if we are not thorough during model evaluation, we can end up with models that rely on spurious information.

Models that seem to be performing well on the surface might be overfitting to noise in the training data. This happens because aggregate metrics tell us nothing about 

That’s one of the key roles explainability takes.

, we can start to shed light on black box models. They help us identify which features contributed the most to the (mis)predictions made by the model. Therefore, we can ensure that our models are making predictions based on reasonable information by inspecting the feature importance scores.

, “understanding the reasons behind predictions is quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model.”

For instance, in the image below, we note how the churn classifier we mentioned in the previous section seems to attribute a lot of importance to the features 

. Using domain expertise, we can decide if this is expected or if the model is biased.

Tabular data is not the only one that benefits from explainability techniques for model evaluation. For example, in natural language processing (NLP) problems, they are just as important. In a sentiment analysis task, where a phrase is categorized as positive or negative, if a model predicts that the sentence “I’m having a great day” is positive because of the word “having” and “day” and not because of the word “great”, is it really a good model?

Not really. Ideally, the model should be predicting a positive label because of the word “great,” which is a strong positive word. Predicting the right label is only half of the story.

The process followed during model evaluation can make or break an ML application. In this post, we explored some of the potential problems of evaluating models mostly using overall aggregate metrics. By incorporating benchmarks, data cohort analysis, and explainability into your evaluation stack, you can detect many issues that otherwise would go unnoticed. These three are just the first step of model evaluation beyond aggregate metrics. If you don’t want to miss out, make sure to join our newsletter!

A high accuracy is not enough

A beginner’s guide to evaluating machine learning models beyond aggregate metrics

Being able to produce explainable predictions lies at the center of trustworthy machine learning (ML). LIME, which is short for Local Interpretable Model-agnostic Explanations, is a technique that can be used to explain the predictions made by any black-box classifier.

In this post, we start by discussing why explainability matters in ML. Then, we understand how LIME works by going through the five steps followed by the algorithm to explain a model prediction. Finally, we illustrate a simple use of LIME in churn prediction and give pointers to references for further exploration, if this is a topic you are interested in.

trade-off between interpretability and predictive performance 

when choosing between different ML models for a task.

The consequences of such a trade-off appear in the types of ML approaches chosen by the industry. For example, in highly regulated fields where there must always be a way to justify the model’s predictions, interpretable models — such as shallow decision trees or linear models with sparse features — are often chosen over more powerful alternatives.

LIME is a technique that can be used with

. Its usefulness is based on the fact that instead of having to work only with intrinsically interpretable models and their limited capabilities, one can use any classifier and then generate explanations for individual predictions afterward, thus benefiting from 

To produce the explanations, LIME feeds the black-box model with small variations of the original data sample and probes how the model’s predictions change. From these variations, it learns an interpretable model which approximates the black-box classifier in the vicinity of the original data sample. Locally, the interpretable model provides a faithful approximation of the black-box model, even though it is likely not a globally good approximator.

Don’t worry if you didn’t understand everything by just reading the previous description. Let's go through the five steps followed by LIME to generate an explanation.

Step 1: Choosing the sample to be explained

Consider the binary classification task from two features. Furthermore, let’s say we have a black-box classifier illustrated below, where class 0 is in red and class 1 is in blue.

The first step in using LIME is choosing the data sample to be explained

. For example, this could be a point from your validation set that your model made a mistake or, really, any sample that you are interested in investigating further.

In this case, we are interested in understanding 

 the black-box classifier predicted that a specific sample, x', belongs to class 0.

Now, the actual LIME algorithm kicks in. It starts by 

 x', which results in multiple new samples, depicted as the black dots in the image below.

Step 3: Labelling the perturbed samples with the black-box model

Then, these perturbed samples are given to the black-box model, to see how it would predict them. These labeled samples will serve as a dataset to train an interpretable model.

There is an additional catch. Since one of the premises that LIME builds upon is that the interpretable model can be a good local approximation, we are mostly interested in the perturbed samples that are close to x'. Therefore, to learn the interpretable model, LIME 

weights the samples according to their proximity

 to x', so samples close to x' are given a large weight, and samples far from x' are given a small weight.

Finally, LIME is ready to learn the interpretable model!

The default interpretable model used by LIME is a linear model. It is learned from the weighted samples and provides explanations for a particular prediction, in this case, x'. Even though the black-box model might be nonlinear, we expect that close to x', the decision boundary can be reasonably approximated by a line.

A linear model with sparse features, such as the one learned by LIME, is 

. The model’s weights can be inspected and seen as proxies for feature importance. Therefore, what LIME returns as the feature scores is actually the linear model’s weights.

For the sake of simplicity, in this post, we focused on giving a high-level overview of how LIME works for tabular data with only two features, avoiding the mathematical details from the problem formulation. The generalization from two features to many is direct, we just wouldn’t be able to draw nice figures.

Moreover, LIME works not only for tabular data but also 

as well. There are a few modifications required so that the data representation over which the explanations are provided remains easily interpretable to humans. All of these modifications are discussed in more detail in the 

 and the interested reader is encouraged to check it out.

LIME is a groundbreaking technique that changed the landscape of ML interpretability.

Let’s go through a simple example that illustrates the usefulness of explainability in a real-world situation. The code and data for this example are 

Imagine you have trained a model that predicts whether a user is likely to churn from demographic and usage data. The resulting model often mistakenly predicts a user is going to churn when, in fact, it doesn't. 

want to understand a few of your model’s mispredictions.

Let's inspect the explainability scores generated by LIME for a row in the validation set. 

Note how for this example, the feature that contributed the most to the prediction (shown in green) is the 

Inspecting more rows where the model makes exactly the same mistake, we notice that when 

, the model doesn't attribute a lot of importance to the 

. This suggests that the model might have picked up a bias from the training set.

Actionable insights arise naturally once we leverage the power of explanations

. Moreover, explanations build trust that your model is behaving as expected and not simply over-indexing to certain features, for instance.

Interpretability and explainability are active areas of research. Even though the current techniques are already proving to be useful, there are still several open questions to be resolved. 

 (which stands for SHapley Additive exPlanations) is another common explainability technique that is similar to LIME. 

Interpretable machine learning: a guide for making black-box models explainable

” by Christoph Molnar is a great resource online that covers many practical aspects surrounding the topic. If you want to start thinking from first principles about interpretability, Zachary Lipton’s “

* A previous version of this article listed the company name as Unbox, which has since been rebranded to Openlayer.

Leveraging how LIME works to build trustworthy ML 

How LIME works | Understanding in 5 steps

Today marks an important milestone for all of us at Openlayer. We’re delighted to announce our $4.8M seed round, led by Quiet Capital, with participation from Y Combinator, Picus Capital, Hack VC, Liquid2 Ventures, Soma Capital, Mantis VC, and Ground Up VC. We’re also fortunate to partner with some awesome angels like Max Mullen, Mike Krieger, Guillermo Rauch, Gokul Rajaram, Yuri Sagalov and Jonathan Swanson.

Our journey started in early 2020, when the three founders were working on a set of exciting AI projects at Apple. 

We started to realize that, for each of these projects, most of our time was spent solving challenges around evaluating models. We grew frustrated that we encountered the same problem over and over: there was no standardized framework, tool, or guidance for testing and collaborating on models. Prior to deployment, engineers were mostly left to their own devices to manually dig through failing rows in an effort to understand and improve models.

 founders Rishab Ramanathan, Vikas Nair, and Gabriel Bayomi

The work of evaluating models has historically been tedious, inefficient, and insufficient. It’s a guessing game. Because of this, often times errors are left uncaught until after they are deployed and start to affect users. This results in rollbacks, stretched development cycles, and potentially disastrous outcomes. We realized the inefficiency and quickly became passionate about building a best-in-class framework for evaluating models to solve these problems.

With the capital raised, we have built a team of likeminded engineers, researchers, and designers from Amazon, Apple, and Harvard Design who are passionate about paving this road for the next generation of AI development. Together, we created Openlayer — a one-of-a-kind machine learning validation and testing platform that enables users to efficiently detect, diagnose, and correct model failures. The product is a culmination of extensive design thought, engineering, and collaboration with a select group of customers, including Fortune 500s and startups. Our customers use Openlayer to boost model performance, streamline development processes, and ship with confidence.

Our focus is now on releasing the product to the public, and expanding to deliver similar value to even more ML teams. The increasing complexity, versatility, and prevalence of models we’re seeing present an ever-expanding surface area of potential vulnerability. LLMs, like ChatGPT, are capable of producing highly accurate and coherent outputs, but they can still make unacceptable errors or generate outputs that may be inappropriate or biased. Our platform will help users better understand these errors, identify their root causes, and refine their models accordingly. We believe that everyone should be empowered to effortlessly ship reliable, performant and ethical models to the world.

Getting here has required the collective effort and support of so many people, and we’d like to thank all of you — our earliest clients, as well as our team, investors, friends and family. You’ve given us the courage to build something truly transformative. We can’t wait to show you.