Developing the basic structure of a RAG pipeline is typically straightforward. However, the real challenge emerges when it comes to fine-tuning it for production and ensuring the quality of its outputs. The selection of appropriate tools and parameters becomes challenging in a landscape filled with numerous options. In this article, I will share insights on developing a strong workflow combining Ragas and Openlayer to assist you in making the best choices for your RAG and ensuring its high quality.

For this article, we’ll use data from arxiv papers about prompt engineering to build our RAG pipeline.

To run the code samples in this post, make sure to 

 install the dependencies, clone the git repo, and export your OpenAI API key as an environment variable.

Compiling a golden test dataset for evaluation is often a burdensome and expensive task, more so at the outset of a project or with shifting data sources. Synthetic generation of high-quality test data presents a viable solution, cutting down curation efforts by 90%. The ideal dataset should feature high-quality, diverse data points, mirroring real production scenarios. 

 adopts a unique, evolution-based synthetic data generation technique, ensuring the creation of diverse and high-quality questions. While it defaults to OpenAI models, Ragas allows the use of any preferred model. Let's see how we can generate 1,000 data points using Ragas.

With the flexibility to adjust the distribution of question types, you can tailor the dataset to better fit your specific requirements. Now that our test dataset is prepared, let’s build a basic RAG pipeline using llama-index.

In this step, I’m using llama-index to build a simple RAG pipeline which I will be evaluating later. For evaluating any RAG using Ragas, apart from the question and the expected answer corresponding to each question that we generated in the earlier step we also need input from our RAG pipeline like contexts retrieved and generated answer for each query. Let’s prepare that now.

Let’s test out RAG by posting a simple question.

Now we have seen that works perfectly, let’s form the evaluation dataset by feeding in each question from test data to our pipeline.

Let’s move on and add this data to Openlayer now that we have all the required data points in for evaluation.

We can start to gauge how well (or poorly) our RAG system is performing by inspecting our dataset rows. The problem is that this evaluation approach is error-prone and lacks comprehensiveness. To ship a production-grade system, we must adopt a more systematic evaluation strategy.

Openlayer is the tool we will use to thoroughly evaluate our RAG system. We will dive into how it solves our problem shortly. First, we need to onboard our artifacts to the Openlayer platform.

Let’s start creating a project on Openlayer:

After we create the project, the next step is to add our dataset and model to the project:

Finally, we can commit and push these artifacts to the Openlayer platform.

Once we have our first commit on Openlayer, it’s time to start evaluating our system by creating tests.

Tests materialize our expectations around our system. For our RAG system, we can start with performance tests using any Ragas metric.

Context recall test

 and measures how good our context retriever is at retrieving all the relevant information required to answer a question. If the context recall is below 0.8 — our threshold — the test fails.

Note how creating tests such as the one we just did enables 

, two pillars of LLM application development.

We created our first test, but we should not stop there. We can set up other Openlayer tests to evaluate our system, ranging from data quality to model performance tests.

After we have tests in place, every commit to Openlayer gets evaluated against this set of well-defined criteria. By doing so, we ensure that we are systematically making progress and avoiding regressions as we work to improve our RAG system.

On our project home page, we can see which tests are passing and which are failing. For example, we can see that the context recall test is currently failing. If we click on the test card, we notice that the test fails because our context recall is below our threshold.

Test diagnostics

We can inspect the scores for individual rows and hypothesize with other team members what is the root cause of the issue. One plausible hypothesis is that our context retriever is retrieving too few contexts, making the context incomplete to answer the question.

One possible path to mitigate the issue is to modify the 

 function. We can increase it, to make our context retriever return more than just two relevant contexts. After re-computing the results, we can push a new commit to Openlayer. By doing so, we can evaluate if the issue was fixed and we can rest assured that we avoided regressions in other domains if the other tests remain passing.

All our efforts so far concentrated on evaluating our system while it was still under development, i.e., before shipping it to production. This is why we started by creating a validation set with Ragas, pushing artifacts to Openlayer, and keeping track of their versions.

However, Openlayer and Ragas are helpful beyond development. Both can be used in a production environment, to monitor our RAG system and help maintain its quality and performance while it serves user requests.

The main difference between development and monitoring is that while in development, we use a validation set to run tests, in monitoring, tests run on production data at a regular cadence.

For example, let’s take a look at the context recall test. We can create a test on Openlayer that checks whether our context recall remains above 0.8 and runs every hour. Openlayer will then use the latest production data published to compute the results. If the test suddenly fails, you get notified immediately, to take corrective action in time.

 Start testing your LLMs in under 2 minutes. 

Posts from Shahul Es

Evaluating RAG pipelines with Ragas and Openlayer

How to use synthetic data to evaluate RAG systems

Shahul Es

Website

Writing Openlayer posts since

Number of posts