We are often in awe that things worked out at all when we look back on history and see how the foundations of the technologies we use today were developed. For example, think about how the software development process looked like as recently as a few decades ago. Isn’t it impressive that whole companies were built by engineers editing source code with partly manual version control?

A lot of great software was built like this, but today, 

no one would dare to say they preferred the old way

 — particularly considering the widespread availability of tools like Git that make development much smoother.

. On the one hand, the world as a whole is convinced of its potential and its applications are already influencing billions of lives. On the other, the way models are developed and deployed in the industry is often far from ideal.

Engineers and data scientists usually receive datasets as CSV files, which they store or access locally through their machines; they experiment with different models on a few Jupyter Notebooks and then perform tests with scripts that they came up with half-haphazardly.

Not only that but engineers are often siloed. It is hard for them to share their work with others and we cannot help but note the similarities with the early days of software development when we see Jupyter notebooks and analysis docs bouncing back and forth with names such as 

How can we expect AI to live up to its full potential if the development process still looks like this?

It is not surprising that from time to time, we see, in the media, pieces about products powered by AI 

making obvious mistakes, exhibiting biases, and even behaving in unethical ways

. For teams shipping such products, in the best cases, the results are a piece of bad PR and upset users; in the worst cases, it can run them out of business.

To overcome such issues, teams must be systematic with their processes.

The challenge is that the complexity and black-box nature of AI/ML has made rigorous evaluation a lot harder than it is in most software development. There is no toolkit that reliably gives developers insight into how and why their models fail.

Openlayer is here to simplify this for both common and long-tail failure scenarios. We are a testing tool that fits into your development and production pipelines to help you ship high-quality models with confidence.

For example, you can promptly detect sudden changes in the data run through your model caused by a change in user behavior. You can also keep track of hallucination scores, to ensure your model responses are always grounded. And that's not all — our platform lets you choose from a comprehensive suite of tests that your model (or agent) needs to clear.

We support seamlessly switching between (1) development mode, which lets you track, version, and compare your tests before you deploy them to production, and (2) monitoring mode, which lets you run tests live in production and receive alerts when things go sideways.

Say you're using an LLM for RAG and want to make sure the output is always relevant to the question. You can set up hallucination tests, and we'll buzz you when the average score dips below your comfort zone.

Or say you're working with a fraud prediction model and are losing sleep over false negatives. With Openlayer, you can quickly write granular tests to measure and monitor the performance of specific cohorts of data and understand why your model made the choices it did on individual data points.

The MLOps landscape is currently fragmented. We’ve seen countless data and ML teams glue together a ton of bespoke and third-party tools to meet basic needs: one for experiment tracking, another for monitoring, and another for CI automation and version control. With LLMOps now thrown into the mix, it can feel like you need yet 

We don’t think you should, so we're building Openlayer to condense and simplify AI evaluation. It’s a collaborative platform that solves long-standing ML problems, while tackling the new crop of challenges presented by Generative AI and foundation models (e.g. prompt versioning, quality control).

We address these problems in a single, consistent way that doesn't require you to learn a new approach. We’ve spent a lot of time ensuring our evaluation methodology remains robust even as the boundaries of AI continue to be redrawn.

We are confident that the way to harness the true potential of AI lies in adopting systematic evaluation methodologies. If you’re interested in building high-quality and impactful AI solutions, you can 

Openlayer

And how to build trustworthy AI

Navigating the chaos: why you don’t need another MLOps tool

Today marks an important milestone for all of us at Openlayer. We’re delighted to announce our $4.8M seed round, led by Quiet Capital, with participation from Y Combinator, Picus Capital, Hack VC, Liquid2 Ventures, Soma Capital, Mantis VC, and Ground Up VC. We’re also fortunate to partner with some awesome angels like Max Mullen, Mike Krieger, Guillermo Rauch, Gokul Rajaram, Yuri Sagalov and Jonathan Swanson.

Our journey started in early 2020, when the three founders were working on a set of exciting AI projects at Apple. 

We started to realize that, for each of these projects, most of our time was spent solving challenges around evaluating models. We grew frustrated that we encountered the same problem over and over: there was no standardized framework, tool, or guidance for testing and collaborating on models. Prior to deployment, engineers were mostly left to their own devices to manually dig through failing rows in an effort to understand and improve models.

 founders Rishab Ramanathan, Vikas Nair, and Gabriel Bayomi

The work of evaluating models has historically been tedious, inefficient, and insufficient. It’s a guessing game. Because of this, often times errors are left uncaught until after they are deployed and start to affect users. This results in rollbacks, stretched development cycles, and potentially disastrous outcomes. We realized the inefficiency and quickly became passionate about building a best-in-class framework for evaluating models to solve these problems.

With the capital raised, we have built a team of likeminded engineers, researchers, and designers from Amazon, Apple, and Harvard Design who are passionate about paving this road for the next generation of AI development. Together, we created Openlayer — a one-of-a-kind machine learning validation and testing platform that enables users to efficiently detect, diagnose, and correct model failures. The product is a culmination of extensive design thought, engineering, and collaboration with a select group of customers, including Fortune 500s and startups. Our customers use Openlayer to boost model performance, streamline development processes, and ship with confidence.

Our focus is now on releasing the product to the public, and expanding to deliver similar value to even more ML teams. The increasing complexity, versatility, and prevalence of models we’re seeing present an ever-expanding surface area of potential vulnerability. LLMs, like ChatGPT, are capable of producing highly accurate and coherent outputs, but they can still make unacceptable errors or generate outputs that may be inappropriate or biased. Our platform will help users better understand these errors, identify their root causes, and refine their models accordingly. We believe that everyone should be empowered to effortlessly ship reliable, performant and ethical models to the world.

Getting here has required the collective effort and support of so many people, and we’d like to thank all of you — our earliest clients, as well as our team, investors, friends and family. You’ve given us the courage to build something truly transformative. We can’t wait to show you.

Posts from Openlayer

Navigating the chaos: why you don’t need another MLOps tool

And how to build trustworthy AI

Openlayer raises $4.8m seed round to build guardrails for AI

Read about our financing round led by Quiet Capital

Openlayer

Website

Writing Openlayer posts since

Number of posts