, and as part of our new identity, we released a new website, 

While developing the new website and app, we found a few techniques that helped us build faster using Next.js + Vercel, Sanity, and Emotion. We are big fans of the Next.js and Vercel stack, and you can read more about our experience in 

In this post, we want to share what we learned along the way, from design to implementation, performance to SEO.

We use Next.js as a framework for both our website and app to rapidly create performant and scalable UI. Next.js comes with built-in support for all the common features needed for any full-stack application. It makes it simple and quick to handle everything from routing and data-fetching to performance and SEO optimizations.

One of the benefits of Next.js is its built-in filesystem-based routing. Every file or folder in the 

 directory becomes a route on the server. For each page, you can optionally export a function 

 at the file level to fetch any data on the frontend server from your backend API (which, by the way, you can similarly create with Next by adding routes to 

We decided, however, to omit any server-side rendering and instead fetch our data on the client after the request is fulfilled. This is recommended if server-side requests are not 

 in order to bring the initial page load down to near zero.

Another decision we made was to separate components from logic as much as possible. For example, we placed a lot of our logic into a separate 

 directory. This has the benefit of readability as well as testability - these functions are decoupled and able to be easily unit-tested.

There are many optimizations that Next.js and Vercel provide that are worth mentioning, but we feel that the best features are image optimization, client-side navigation, and deployment previews.

Image optimization allows for static images to be optimized based on device size. Any images that we placed in the 

 folder is rendered as multiple images of different sizes for mobile, tablet, and desktop screens, making content delivery faster for smaller devices.

For images that cannot be included in the 

 folder (like images from a CMS), we can still optimize them for mobile screens using Sanity's 

For any link between pages within our app, we can use the 

. Next.js will then transition between pages using only Javascript, without sending a new request. This allows for instant page transitions without losing the traditional page structure needed for search engine optimization (SEO).

And the results are dramatic. Since most websites on the web do not use this technique, the difference in speed is very noticeable, and it makes the site feel much leaner as a result. We get the performance of a single-page app (SPA) with the visibility of a traditional site.

Vercel offers deployment previews on every PR pushed to your Github repo, even on the free tier. This feature is perfect for teams that need to collaborate on the design and implementation of a website before it goes live.

It allows teammates to post comments at specific pages and locations within the site. You can even tag each other or add images for context. As a dev tool, you can mark comments as resolved once they are finished, effectively serving as a shared to-do list for team feedback.

For even quicker feedback, Vercel also ships with the 

 command which can launch a one-off deployment from the code in your current directory.

This allows developers to test out their changes on a mobile phone, for example, before pushing to the remote repo.

Tools like these are a superpower for collaborative feedback, dramatically reducing the time to fix bugs and ensuring a seamless user experience before launching a public site.

, which is a popular content management system (CMS) that comes with its own online editing tool. This allows writers to add posts to the site without touching any code.

With content management comes the quest of caching. Does each page visit send a separate request to the CMS, or can it be cached in the backend? Since the blog only updates each time we add or update a post, it does not make sense to re-render the entire blog index on each request.

Instead, Next.js provides something called 

 (ISR), which lets us customize caching. By adding the “revalidate” property to getStaticProps, we can tell Next.js that we want our page to be "static,” meaning rendered before user requests, but also “incremental,” meaning updated every so often to keep it fresh. In our case, this update happens at most every 10 seconds. To a user visiting the page, there will be no additional backend delay due to the pre-rendering. But our blog will still reflect the latest content in Sanity (save for a small, acceptable delay).

 markup and styles. This is a natural tool when writing React code, since your layout is in Javascript (or Typescript, in our case). It also encourages 

, two tools to reduce the complexity of code.

). While developing, we can open the two files side-by-side to easily get a picture of all the relevant details for that component. Another win for readability and developer speed!

With Emotion, we try to use inheritance patterns as much as possible. We create several base components, from foundational flex or grid containers to more complex components that can be easily extended as needed.

One big benefit of using Emotion is the performance improvements when organizing our styles into modular pieces. Emotion will automatically cache reusable styled components into a separate class name (something like 

). Anywhere that we reuse that style, its class is already stored by the browser for quick retrieval.

Of course, these hashes tell us nothing about where the style came from or what it is doing, so we use the 

 during development to append the folder and filename.

This makes visual debugging from the browser’s web inspector much faster.

Another benefit of using Emotion with Typescript is that it gives us Intellisense on CSS properties.

 (as opposed to CSS strings), each style block is a Javascript object, so the keys and values can be statically typed. This speeds up development because we get feedback on typos from within the editor, before we render in the browser. Small improvements like this add up over time.

, which means that all styled components can take an implicit theme property without having to explicitly pass it through from a parent component, like this:

To enable themes, we just need to wrap our app in a ThemeProvider:

Since our mobile site had a lot of common markup with the desktop version, it made sense to also include an "isMobile" property to our theme. We could simply update the Theme type like so:

Now, all our styles have access to this information, without having to pass an additional prop throughout the tree:

Whether you are creating a new website from scratch or reworking a legacy site, you want to be able to build fast. It is natural to leave optimization for a later time or to skip some industry best practices. But knowing about the best tools and techniques beforehand can make development faster in the long run. There are many small decisions that add up to huge savings when it comes to getting fast feedback or reducing the cost of maintenance.

Using popular tools like Next.js + Vercel, Sanity, and Emotion are a great way to bring in some of these best practices from the start, since they are all developed by web experts with many years of experience in the field. We also highly recommend 

 for state and network query management, but more on that for a later post about the stack behind our 

All of these tools are essentially free to start using now, so give them a whirl if you aren’t already! If you want to see some more industry-leading tools in action, the 

We hope you found this post useful in your own development. Feel free to reach out to us on social media with any comments or feedback. We’d love to hear what you think!

Rex Garland

Developing fast and reliably

How we built our public site with Next.js, Sanity and Emotion

How do you decide if you have a high-quality machine learning (ML) model in your hands?

The insights that help you answer this question should come from the model evaluation stage. However, more often than not, people interpret “model evaluation” as a synonym for computing overall aggregate metrics on a validation set (such as accuracy, or F1).

The problem is that seeing aggregate metrics as proxies for model quality can be very misleading. In this post, we will show three ways you can evaluate models beyond aggregate metrics.

We could spend all day presenting the deceptions of relying too much on overall aggregate metrics. Model evaluation should assess model quality, and the quality of a model goes way beyond a single number. However, in this post, we will focus on three main problems and how each can be overcome.

The first problem of relying too much on aggregate metrics is that they are hard to interpret in isolation. This often leads to misinterpretation of model capabilities and difficulty getting management buy-in — since aggregate metrics are usually not directly tied to business value.

The second is that overall aggregate metrics provide a very low-resolution image of model performance. Consequently, model evaluation overly based on them is prone to selecting models that will, later, fail silently in production.

Finally, aggregate metrics tell us nothing about 

 our models perform the way they do. A team that only looks at aggregate metrics might be deceived by a model that performs well but, in fact, relies on spurious data from the training set to make its predictions.

To overcome these problems, one must expand their model evaluation process. That’s what we will explore in the next section.

Model evaluation beyond aggregate metrics

Imagine you are building an ML model that predicts house prices. After training the model, you obtain a mean squared error (MSE) of 120,000 dollars^2 on the validation set.

Now you need to decide: is such an MSE good or not?

Note how answering one of the main questions from model evaluation is not as trivial as it seems. Even for a standard metric such as the MSE, it is impossible to answer this question by just staring at a single number. More context is needed.

Benchmarks are one piece of this additional context. To adequately evaluate models, we should strive to iteratively compare their performance to benchmarks, which serve as goalposts along the way. They help us ground the evaluation process and ensure we are moving in the right direction.

Back to our example, if the model you are developing is expected to replace an existing system (ML powered or not), the natural benchmark would be the performance of such a system. As you progress in the model evaluation process, you should often ask yourself if the current model surpasses your benchmark. Otherwise, you risk pouring energy into something that ultimately won’t be used.

Alternatively, if you are developing the model from scratch, it is important to start with simple benchmarks and progressively increase their complexity over time. Usually, a good place to start is with the performance of the random (chance) model. Then, moving to rule-based estimates. To simple models using a subset of features. And so on. This is similar to the approach we explored in detail in our 

Benchmarks are also useful during the last mile of model development. At that stage, it is common to discuss the results from model evaluation with people from a business background. Using the standard ML aggregate metrics might not be the best choice to convince them of how great your model is. First, because most aggregate metrics are not intuitive. Second, they might not be directly tied to business metrics, which is what a company cares about at the end of the day.

In these situations, well-crafted benchmarks that are easy to compute and understand are extremely valuable. How these benchmarks look will vary greatly between problems, and they can even be the human performance for the task at hand. After all, the argument in favor of a model that surpasses the performance of real estate agents is much stronger than any MSE score in our example.

Aggregate metrics are convenient because they summarize a lot of information into a single number. However, as a consequence, they provide a low-resolution picture of what’s going on with our model performance.

Hidden behind a high accuracy, for example, are usually many data cohorts where our model performance is not so stellar. Model evaluation based mostly on overall aggregate metrics is prone to choosing models that fail silently on some data pockets.

Consider a churn classifier that predicts whether a user is likely to churn from demographic and usage data. The code for this example is 

. The resulting model’s accuracy on the validation set is equal to 0.71.

Now, if we start to dig deeper and explore different data cohorts, we see that there are underperforming subpopulations. For example, if we evaluate our model on the fraction of the dataset where the feature 

, we notice that the performance is equal to 0.52. This is significantly lower than the overall accuracy.

There are also many other data cohorts defined by feature value combinations that result in high error rates, as shown in the image below. In this case, we are using an algorithm to automatically find the data cohorts that might be problematic for the model.

Identifying such subpopulations is critical to evaluate models. Otherwise, our view of model performance can be myopic.

ML models are trained to pick up patterns in the training data. The problem is that datasets contain good and bad patterns. Signal as well as noise. Thus, if we are not thorough during model evaluation, we can end up with models that rely on spurious information.

Models that seem to be performing well on the surface might be overfitting to noise in the training data. This happens because aggregate metrics tell us nothing about 

That’s one of the key roles explainability takes.

, we can start to shed light on black box models. They help us identify which features contributed the most to the (mis)predictions made by the model. Therefore, we can ensure that our models are making predictions based on reasonable information by inspecting the feature importance scores.

, “understanding the reasons behind predictions is quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model.”

For instance, in the image below, we note how the churn classifier we mentioned in the previous section seems to attribute a lot of importance to the features 

. Using domain expertise, we can decide if this is expected or if the model is biased.

Tabular data is not the only one that benefits from explainability techniques for model evaluation. For example, in natural language processing (NLP) problems, they are just as important. In a sentiment analysis task, where a phrase is categorized as positive or negative, if a model predicts that the sentence “I’m having a great day” is positive because of the word “having” and “day” and not because of the word “great”, is it really a good model?

Not really. Ideally, the model should be predicting a positive label because of the word “great,” which is a strong positive word. Predicting the right label is only half of the story.

The process followed during model evaluation can make or break an ML application. In this post, we explored some of the potential problems of evaluating models mostly using overall aggregate metrics. By incorporating benchmarks, data cohort analysis, and explainability into your evaluation stack, you can detect many issues that otherwise would go unnoticed. These three are just the first step of model evaluation beyond aggregate metrics. If you don’t want to miss out, make sure to join our newsletter!

Gustavo Cid

A high accuracy is not enough

A beginner’s guide to evaluating machine learning models beyond aggregate metrics

Being able to produce explainable predictions lies at the center of trustworthy machine learning (ML). LIME, which is short for Local Interpretable Model-agnostic Explanations, is a technique that can be used to explain the predictions made by any black-box classifier.

In this post, we start by discussing why explainability matters in ML. Then, we understand how LIME works by going through the five steps followed by the algorithm to explain a model prediction. Finally, we illustrate a simple use of LIME in churn prediction and give pointers to references for further exploration, if this is a topic you are interested in.

trade-off between interpretability and predictive performance 

when choosing between different ML models for a task.

The consequences of such a trade-off appear in the types of ML approaches chosen by the industry. For example, in highly regulated fields where there must always be a way to justify the model’s predictions, interpretable models — such as shallow decision trees or linear models with sparse features — are often chosen over more powerful alternatives.

LIME is a technique that can be used with

. Its usefulness is based on the fact that instead of having to work only with intrinsically interpretable models and their limited capabilities, one can use any classifier and then generate explanations for individual predictions afterward, thus benefiting from 

To produce the explanations, LIME feeds the black-box model with small variations of the original data sample and probes how the model’s predictions change. From these variations, it learns an interpretable model which approximates the black-box classifier in the vicinity of the original data sample. Locally, the interpretable model provides a faithful approximation of the black-box model, even though it is likely not a globally good approximator.

Don’t worry if you didn’t understand everything by just reading the previous description. Let's go through the five steps followed by LIME to generate an explanation.

Step 1: Choosing the sample to be explained

Consider the binary classification task from two features. Furthermore, let’s say we have a black-box classifier illustrated below, where class 0 is in red and class 1 is in blue.

The first step in using LIME is choosing the data sample to be explained

. For example, this could be a point from your validation set that your model made a mistake or, really, any sample that you are interested in investigating further.

In this case, we are interested in understanding 

 the black-box classifier predicted that a specific sample, x', belongs to class 0.

Now, the actual LIME algorithm kicks in. It starts by 

 x', which results in multiple new samples, depicted as the black dots in the image below.

Step 3: Labelling the perturbed samples with the black-box model

Then, these perturbed samples are given to the black-box model, to see how it would predict them. These labeled samples will serve as a dataset to train an interpretable model.

There is an additional catch. Since one of the premises that LIME builds upon is that the interpretable model can be a good local approximation, we are mostly interested in the perturbed samples that are close to x'. Therefore, to learn the interpretable model, LIME 

weights the samples according to their proximity

 to x', so samples close to x' are given a large weight, and samples far from x' are given a small weight.

Finally, LIME is ready to learn the interpretable model!

The default interpretable model used by LIME is a linear model. It is learned from the weighted samples and provides explanations for a particular prediction, in this case, x'. Even though the black-box model might be nonlinear, we expect that close to x', the decision boundary can be reasonably approximated by a line.

A linear model with sparse features, such as the one learned by LIME, is 

. The model’s weights can be inspected and seen as proxies for feature importance. Therefore, what LIME returns as the feature scores is actually the linear model’s weights.

For the sake of simplicity, in this post, we focused on giving a high-level overview of how LIME works for tabular data with only two features, avoiding the mathematical details from the problem formulation. The generalization from two features to many is direct, we just wouldn’t be able to draw nice figures.

Moreover, LIME works not only for tabular data but also 

as well. There are a few modifications required so that the data representation over which the explanations are provided remains easily interpretable to humans. All of these modifications are discussed in more detail in the 

 and the interested reader is encouraged to check it out.

LIME is a groundbreaking technique that changed the landscape of ML interpretability.

Let’s go through a simple example that illustrates the usefulness of explainability in a real-world situation. The code and data for this example are 

Imagine you have trained a model that predicts whether a user is likely to churn from demographic and usage data. The resulting model often mistakenly predicts a user is going to churn when, in fact, it doesn't. 

want to understand a few of your model’s mispredictions.

Let's inspect the explainability scores generated by LIME for a row in the validation set. 

Note how for this example, the feature that contributed the most to the prediction (shown in green) is the 

Inspecting more rows where the model makes exactly the same mistake, we notice that when 

, the model doesn't attribute a lot of importance to the 

. This suggests that the model might have picked up a bias from the training set.

Actionable insights arise naturally once we leverage the power of explanations

. Moreover, explanations build trust that your model is behaving as expected and not simply over-indexing to certain features, for instance.

Interpretability and explainability are active areas of research. Even though the current techniques are already proving to be useful, there are still several open questions to be resolved. 

 (which stands for SHapley Additive exPlanations) is another common explainability technique that is similar to LIME. 

Interpretable machine learning: a guide for making black-box models explainable

” by Christoph Molnar is a great resource online that covers many practical aspects surrounding the topic. If you want to start thinking from first principles about interpretability, Zachary Lipton’s “

* A previous version of this article listed the company name as Unbox, which has since been rebranded to Openlayer.

Leveraging how LIME works to build trustworthy ML 

How LIME works | Understanding in 5 steps

Today marks an important milestone for all of us at Openlayer. We’re delighted to announce our $4.8M seed round, led by Quiet Capital, with participation from Y Combinator, Picus Capital, Hack VC, Liquid2 Ventures, Soma Capital, Mantis VC, and Ground Up VC. We’re also fortunate to partner with some awesome angels like Max Mullen, Mike Krieger, Guillermo Rauch, Gokul Rajaram, Yuri Sagalov and Jonathan Swanson.

Our journey started in early 2020, when the three founders were working on a set of exciting AI projects at Apple. 

We started to realize that, for each of these projects, most of our time was spent solving challenges around evaluating models. We grew frustrated that we encountered the same problem over and over: there was no standardized framework, tool, or guidance for testing and collaborating on models. Prior to deployment, engineers were mostly left to their own devices to manually dig through failing rows in an effort to understand and improve models.

 founders Rishab Ramanathan, Vikas Nair, and Gabriel Bayomi

The work of evaluating models has historically been tedious, inefficient, and insufficient. It’s a guessing game. Because of this, often times errors are left uncaught until after they are deployed and start to affect users. This results in rollbacks, stretched development cycles, and potentially disastrous outcomes. We realized the inefficiency and quickly became passionate about building a best-in-class framework for evaluating models to solve these problems.

With the capital raised, we have built a team of likeminded engineers, researchers, and designers from Amazon, Apple, and Harvard Design who are passionate about paving this road for the next generation of AI development. Together, we created Openlayer — a one-of-a-kind machine learning validation and testing platform that enables users to efficiently detect, diagnose, and correct model failures. The product is a culmination of extensive design thought, engineering, and collaboration with a select group of customers, including Fortune 500s and startups. Our customers use Openlayer to boost model performance, streamline development processes, and ship with confidence.

Our focus is now on releasing the product to the public, and expanding to deliver similar value to even more ML teams. The increasing complexity, versatility, and prevalence of models we’re seeing present an ever-expanding surface area of potential vulnerability. LLMs, like ChatGPT, are capable of producing highly accurate and coherent outputs, but they can still make unacceptable errors or generate outputs that may be inappropriate or biased. Our platform will help users better understand these errors, identify their root causes, and refine their models accordingly. We believe that everyone should be empowered to effortlessly ship reliable, performant and ethical models to the world.

Getting here has required the collective effort and support of so many people, and we’d like to thank all of you — our earliest clients, as well as our team, investors, friends and family. You’ve given us the courage to build something truly transformative. We can’t wait to show you.

Openlayer

Read about our financing round led by Quiet Capital

Openlayer raises $4.8m seed round to build guardrails for AI

 is an essential concept in software development. Version control tools, such as 

, are used to manage and track source codes. Developers and organizations alike value source code because it is the backbone of their product(s), from which they earn revenue. 

 is the application of version control concepts within the machine learning process.

Machine learning and artificial intelligence offer an edge to businesses and developers over their competitors, motivating the competition to push to build the best model as it provides a business advantage. As developers optimize for speed and accuracy, minimizing errors also becomes a priority. Model versioning is a great tool for all of those purposes.

In this article, you’ll learn more about the importance of model versioning in machine learning by exploring a demo on how to apply it when building ML models and reviewing best practices to follow, limitations to be aware of, and some of the best tools currently available.

Model versioning derives its core processes from version control in software development. The difference is that software development runs mainly on code, while machine learning runs on both code and data, which is the backbone of the best-performing models.

Model versioning, therefore, is the process of tracking and managing changes made to source code and data, including metrics, parameters, and hyperparameters. Popular 

 have failed to track changes made to data and are limited in the amount of data they’re able to host due to storage limits. Solutions such as 

 (Git-LFS) can accept a large amount of data but still fail to track data and models.

Applying model versioning when building machine learning applications is important because it allows for the following:

Referencing previous versions of the model

Reverting changes quickly, making building models less risky

Easily reproducing and sharing models among development teams

Using a hands-on approach to learning about model versioning will aid your understanding of the concept. In this guide, 

 will be used to demonstrate how model versioning works. MLflow is a platform used for tracking, reproducing, deploying, and managing models in a central repository.

To get started, install MLflow and scikit-learn using one of the following methods:

: These install scikit-learn and MLflow separately.

: This installs MLflow with other dependencies, including scikit-learn.

Then, run the file in a terminal pointing to the file directory using 

 to compare the models produced. View the page at 

You should see the MLflow interface displaying logged details about the model:

. Restart MLflow using the same command, 

. MLflow automatically saves the new metrics and parameters, logs them, and displays them on the localhost, as pictured here:

MLflow tracks logged objects, such as parameters and metrics, by creating project directory folders containing the result of each run in separate folders with unique 

 names to distinguish them and make it easier to reference particular objects as needed.

Each time you run the code, a new folder is created. Within these hash-named folders are a different set of folders, such as artifacts and metrics, which contain the logged objects. Here’s an example of what the file structure looks like:

ML models are generally built by developers across a variety of teams, which requires a high level of collaboration. Model versioning then becomes necessary as it permits teams to do the following:

 Building machine learning models is nondeterministic. In other words, training the same algorithm on the same data, as done in the past, can produce a different result due to the change in parameters and hyperparameters. Tracking data and everything involved in the model-building process becomes possible with model versioning, allowing for the convenient reproduction of models.

 Working on machine learning projects requires collaborative efforts depending on the size and complexity of the project. Model versioning allows teams to share models by saving files in a remote storage location and recording/tracking each file that produced a model. Teams can constantly update and share models among themselves.

 Industries that deal with sensitive information, such as healthcare, need data governance to ensure that data is handled with care and that it remains consistent and accurate. Model versioning allows for proper model auditing to ensure compliance with government laws like the GDPR.

In order to prevent having bugs in production, which can break the system, here are some best practices to follow:

 Regularly reviewing your model will help you spot errors that could otherwise cause it to break or produce inaccurate predictions. Be sure to keep an eye out for outliers and data points that can weaken or change your model’s performance.

Regularly delete models that are not in use:

 Keeping obsolete models uses up space and causes redundancy. Deleting these models provides clarity and ensures that your resources are invested only in models that are currently in use.

 Testing models before deployment is crucial to help you identify errors that may otherwise lead to downtime, operation costs, and for a commercial product, loss of users.

Model versioning is still in its early stages, with tools like DVC and MLflow made available for public use in 

, respectively, compared to Git, which was built in 

. Thus, tools used for versioning—especially open-source tools—are not yet built to support 

 model versioning. For instance, some prioritize tracking model changes over robust storage options (or vice versa). Similarly, some model versioning tools come with pipeline management, leading to data redundancy in cases where your team already has a pipeline system in place.

To overcome these limitations, organizations like 

 have built their own internal versioning systems, while others opt to use third-party solutions like 

To tackle the challenges of model versioning and solve the limitations of versioning software such as Git, several open-source and enterprise solutions have been built. Here are a few useful tools to consider:

: Openlayer is a collaborative quality assurance platform for machine learning that allows you to track and version your models with ease. It helps you discover errors when building models, boost model performance, and test your model before deployment. If you’re building an end-to-end machine learning project, it may be just the right tool for you.

: Data Version Control (DVC) is an open-source tool used for versioning data sets. DVC commands are similar to Git’s, and both are used in tandem when building models. DVC versions large data sets, while Git stores the .dvc files used to retrieve data from DVC remote storage. DVC supports popular remote storage options like Amazon S3 and Google Cloud Storage and is a great open-source option to version your data.

: LakeFS is an open-source platform similar to DVC that provides Git-like commands for performing model versioning operations. It also integrates storage options like AWS and GCP to store data. LakeFS minimizes data duplication via a copy-on-write mechanism and is well suited for enterprise use due to its high performance over data lakes of any size.

Model versioning is a valuable part of the development process for machine learning projects because of the need for collaboration, tracking changing codes, and monitoring the model’s performance over time. If you’re considering what tool might be right for you and your team, here are a few guidelines to keep in mind. The ideal tool will have the following features:

 Model versioning tools should be easy for you and your team to get familiar with. Machine learning is complex enough without these tools—using them should make building models simpler, not more complicated.

 Whatever tool you choose to use should integrate well with your tech stack for smooth building. Fortunately, most tools built for model versioning are language-agnostic, so finding one that meets your needs shouldn’t be difficult.

 Choosing a tool or platform that supports the kind of data you are working with ensures faster model development. Versioning tabular-based data is different from versioning audio, video, or image files. You’ll want to pick a tool that supports the data type you work with now—as well as any others you plan on working with in the future—to avoid needing to switch platforms later on.

If you’re in the market for a great ML platform that allows you to expertly version and track your models, be sure to consider 

 today to learn about how Unbox can help you with error analysis, synthetic data generation, testing, deploying models, and more.