Skip to main content
Each Openlayer test can be tailored to your needs. You can choose the data to run on, adjust thresholds, tweak parameters, and more. Most options have sensible defaults, so setup is usually minimal. This page walks you through all available configurations. Test configuration

​
Configurations

The configurations available are:
The “Data” section allows you to select the data on top of which this test will run.In monitoring mode, it means choosing among your data sources (e.g., production data, staging data, etc.) In development mode, it means selecting among your datasets (e.g., training, validation, etc.)You can also apply filters to narrow the test to specific slices of the data, and turn on Apply to new data sources so the test covers data sources you connect later. When you fill in a filter, the value picker suggests values from your data — see Filter value suggestions.To test or filter on one key inside a structured input or output instead of the whole column, define a custom column for it.For supported tests in monitoring mode, Evaluate controls whether the test evaluates each matching trace as one row or evaluates each matching step as its own row. Whole trace is the default. See Evaluating whole traces or individual steps for eligibility, filtering, and result behavior.In monitoring mode, the section also holds the Evaluation windows settings: the Cadence the test runs on (Scheduled or Manual), plus the Evaluation delay, Include historical data in each run, and Anchor evaluation windows toggles. See Evaluation and delay windows for details.
Some tests include configurable parameters that control their behavior. For example, a test such as the PII detection test lets you specify which type of PII to detect (e.g., credit card numbers, social security numbers, etc.)Custom metrics can also declare their own configurable parameters. See Configurable parameters for details on how to define them.Refer to the specific test page in the Catalog for details on its parameters.For LLM-as-a-judge tests, the Judge settings hold the judge model and the sampling controls that bound how much data the judge scores: Sample size (%) (100 by default), Data limit (rows) (50 by default), and the Sampling method (for example, Random). You can set the same controls for every test in the project under Settings → LLM-as-a-judge.
The “Threshold” section allows you to define the condition for the test to pass.On the Manual tab, you pick a Measurement, an Operator, and a Threshold value — saying, for example, that the test passes if the metric is greater than 0.5.In monitoring mode, most tests also offer an Auto tab. With automatic thresholds, Openlayer automatically learns the distribution of historical data and predicts upper and lower bounds for the metric. The test fails if the observed value falls outside the predicted range.Automatic thresholds are directional: choose Both bounds to fail on either side of the predicted range, or Upper only or Lower only to fail on one side only — useful when a move in a single direction is what concerns you, such as a rising cost.
The “Advanced settings” section holds the test metadata. Here’s where you set the test’s Priority.Cadence, delays, and anchoring live in the “Data” section — see Evaluation and delay windows.

​
Evaluating whole traces or individual steps

In monitoring mode, supported tests include an Evaluate setting:
  • Whole trace evaluates each matching trace as one row. This is the default.
  • Each step evaluates every span that matches the test’s step filters as its own row. Each span receives its own score and explanation, and the test result aggregates those span-level results.
When you select Each step, filters are grouped into Step fields, including Step name and Step type, and Parent trace fields. Use a step filter to narrow which steps are evaluated; Openlayer displays a hint when no step filter does so. For tests that require a column, you can evaluate the step’s Output or full Input. Semantic filters are not supported for step-scoped tests. You choose the evaluation scope when you create the test and cannot change it later. To use a different scope, create another test. Openlayer also rejects step scope in development mode or for an unsupported test or metric. For Metric tests, the available scope follows the selected measurement; when you edit a step-scoped Metric test, only measurements that can evaluate steps are available.

​
Supported tests and measurements

You can use Each step with Character length, String validation, Is JSON, JSON schema validation, Is code, Contains valid URL, Special characters ratio for LLM projects, Contains PII, Contains PHI, Prompt injection, LLM-as-a-judge, and Bias tests. RAGAS tests support step scope only for metrics judged from the step’s input and output. For Metric tests, the following measurements support step scope:
  • Latency: minimum, maximum, mean, median, p90, p95, and p99.
  • Cost: maximum, mean, and total.
  • Tokens: maximum, mean, and total.
  • Correctness, harmfulness, coherence, conciseness, maliciousness, and answer relevancy.
Cost and token measurements evaluate only spans that record usage. Metrics that require trace context or a ground truth remain whole-trace only: faithfulness, hallucination, context utilization, context relevancy, context recall, answer correctness, exact match, quasi-exact match, edit distance, semantic similarity, BLEU 1–4, and JSON score. The same restriction applies to RAGAS metrics that require context or a ground truth.

​
Reading step test results

The test’s Properties panel shows Evaluates: Each step or Evaluates: Whole trace. For a step-scoped test, the Data tab lists one span per row with its score, explanation, and failure highlighting. You can filter these rows by Step ID. Opening a row selects that step in the trace view. In the trace view, the Scores panel lists the results for the selected record or step. Results are labeled with the test, and explanations are available on hover. A results icon in the trace tree identifies steps with scores.

​
Filter value suggestions

When you filter a test on a column, the value picker suggests values Openlayer has already seen in that column, so you can select a value instead of typing it from memory. Every filter value picker can show these suggestions, not just semantic search filters, and the kind of comparison you are making determines which values it suggests — whether you are matching on equality, membership, contains, or an ordering comparison. The same suggestions appear when you filter monitoring records: the Records table’s filter prompts you to “Enter value…”, then lists matching values under “In the last 30 days” with each value’s count beside it. Each suggested value carries a count tag. Hover it to see how many rows matched, as in “12 matches found in recent data”, together with the note “Suggestions are drawn from a recent sample of rows, so the full dataset may contain more matches.” Read the count as a signal of whether a value is worth filtering on rather than as an exact total. Columns with many distinct values do not list suggestions up front. The picker asks you to “Type to search values” instead, and looks up matching values as you type.

​
Pausing tests

In monitoring mode, you can pause tests when you don’t want them to compute results — for example, while you are replaying data or waiting on an upstream fix. Open the overflow menu on a test row and select Pause. While tests are paused, the tests list shows the banner “Tests are paused. You can still create tests, but they will not compute results.” Select Unpause in the banner to start receiving test results again.