# Introducing jevals: typed decisions for agent evals and guardrails

> An open-source Python library for evaluating agent behavior and turning the same checks into runtime guardrails using typed decision models.

Published: 2026-09-23

Today we’re releasing **jevals**, an open-source Python library for agent evals and guardrails built around Jev-style decision models. We’ve also added **Jev as a judge directly in Openlayer**.

The idea is simple: **write an eval once, use it to test your agent, then use the same definition to gate what the agent can do in production.**

A lot of agent evaluation ultimately comes down to a decision:

- Did the agent choose the right tool?
- Is its answer supported by the evidence?
- Did it stay within scope?
- Should this action proceed, be blocked, or go to a human?

LLM judges can answer these questions. But when the output you need is a classification or score, generating text for every judgment introduces unnecessary cost and latency. That becomes especially noticeable when you’re evaluating large volumes of traffic or checking actions inside an agent loop.

Jev takes a different approach.

You provide the relevant context and questions with defined answer types. Jev returns choices, scores, and probabilities, and can evaluate multiple questions in a single request.

[How Jev works](https://vercel.com/changelog/typesafe-ai-jev-now-available-on-ai-gateway)

We built **jevals** to turn that interface into a practical evaluation and guardrail library.

## Get started with jevals {#get-started-with-jevals}

pip install jevals

Then pass in the messages and tool schemas from an agent run:

from jevals import evaluate
from jevals.agent import ToolChoice, Grounded, StayedInScope
from jevals.security import IndirectInjection

result = evaluate(
    {"messages": messages, "tools": tools},
    [
        ToolChoice(),
        Grounded(),
        StayedInScope(),
        IndirectInjection(),
    ],
)

print(result.table())

## 37 evaluations out of the box {#37-evaluations-out-of-the-box}

jevals currently includes **37 evaluations** across agent behavior, response quality, and security.

You can evaluate things like:

- tool selection
- groundedness
- task completion
- scope adherence
- prompt injection
- sensitive information

Custom evaluations can be written in Python or YAML.

## Turn the same eval into a runtime guardrail {#turn-the-same-eval-into-a-runtime-guardrail}

A gate applies a policy to evaluation results and determines whether an action should **proceed, be blocked, or require review**.

For a support agent, that could mean escalating a proposed refund before execution, checking tool results for injected instructions, or redacting sensitive information before passing it onward.

You can also replay saved traces against the same definitions to see how those policies would have behaved.

**The check you use during development can become the check you enforce in production.**

[See guardrail examples](https://github.com/openlayer-ai/jevals#guardrails)

## Why use a decision model instead of an LLM judge? {#why-use-a-decision-model-instead-of-an-llm-judge}

Cost is one reason.

In the small RAG benchmark documented in our README, the estimated model cost for four metrics was:

Configuration

Cost per 1,000 samples

Ragas with GPT-4.1-mini

$2.60

jevals with Jev

$0.03

Those costs are calculated from observed token usage and published pricing.

The benchmark used 20 rows, with differences in metric implementation and an OpenRouter limitation affecting Ragas answer relevancy. It measures the economics of that particular setup, not universal evaluation quality. You should still validate evaluations against your own examples and labels.

[See the full benchmark](https://github.com/openlayer-ai/jevals#numbers)

## Bring your own backend {#bring-your-own-backend}

jevals supports hosted Jev, local Kev and Laya backends, as well as a conventional LLM backend.

You can switch backends while keeping the same evaluation definitions, then recalibrate thresholds for the model you’re using.

This isn’t an argument that LLM judges should disappear. Evaluations that require extended reasoning, nuanced interpretation, or a written critique can still benefit from an LLM judge.

But a large class of agent evaluations are ultimately typed decisions. **We think they should be treated that way.**

## Jev is now available in Openlayer {#jev-is-now-available-in-openlayer}

Alongside the open-source release, **Jev is now available as a judge directly in Openlayer**.

Openlayer also continues to support evaluation of the classifiers themselves, including F1 scores, confusion matrices, drift, and explainability. Those tools remain useful whenever the output is a decision, regardless of how the underlying model was trained.

jevals is currently in alpha, and we want people to break it.

[**Try jevals on your own agent traces**](https://github.com/openlayer-ai/jevals) and tell us which checks work, which fail, and what you want to evaluate next.

## FAQ {#faq}

### What is jevals?

jevals is an open-source Python library for evaluating AI agents and creating runtime guardrails. It includes prebuilt evaluations for agent behavior, response quality, and security, and supports custom evaluations written in Python or YAML.

### How is jevals different from an LLM judge?

Many LLM judges generate freeform text before producing an evaluation result. jevals can use typed decision models such as Jev to directly return structured choices, scores, and probabilities for classification-style evaluations.

### Can jevals be used for AI agent guardrails?

Yes. Evaluation results can be connected to gates that determine whether an agent action should proceed, be blocked, or require human review. The same evaluation definition can therefore be used during testing and in production.

### Does jevals only work with Jev?

No. jevals supports hosted Jev, local Kev and Laya backends, as well as conventional LLM backends. Evaluation definitions can remain the same while the underlying backend changes.

### What can jevals evaluate?

jevals includes evaluations for areas such as tool selection, groundedness, task completion, scope adherence, prompt injection, and sensitive information.

### Is jevals open source?

Yes. jevals is an open-source Python library available on GitHub.