January 17, 2024

Cost-per-request, new tests, subpopulation support for data tests, and more precise row filtering

We’re excited to introduce the newest set of tests to hit Openlayer! Make sure column averages fall within a certain range with the Column average test. Ensure that your outputs contain specific keywords per request with our Column contains string test, where the values in Column B must contain the string values in Column A. Monitor and manage your costs by setting Max cost, Mean cost, and Total cost tests.

As additional support for managing costs, we now show you the cost of every request in the Requests page.

You can now filter data when creating integrity or consistency tests so that the results are calculated on specific subpopulations of your data, just like performance goals.

That’s not all, so make sure to read all the updates below. Join our Discord community to follow along on our development journey, and stay tuned for more updates from the changelog! 📩🤝

Features

New tests
- Column average test – make sure column averages fall within a range
- Column contains string test – column B must contain the string in column A
- Cost-related tests – max cost, mean cost, and total cost per evaluation window
View your production data associated with each of your tests in monitoring mode
Support for cost-per-request and cost graph
Filter rows by row-level metrics such as conciseness
Subpopulation support for data goals
The timeline page is back - see how your commits perform on goals over time

Improvements

Ability to update previously published production data by setting existing columns or adding new columns
Sample requests are paginated
Latency rendered in ms in the requests table
Requests filters no longer require selecting a filter type
Suggested tests modal auto-opens after project creation outside of the onboarding
Notifications callout not shown until the project is fully setup
Enabled filtering without datasets in development and monitoring modes
Render cost in requests table
Render monitoring data correctly in test diagnosis modals
Row-level scores and explanations rendered for gpt-based metric tests
Activity log is now collapsible
Individual rows in data tables within the test diagnosis modal can be expanded
Input and output columns rendered next to each other in data tables
New example notebook showing how to send additional columns as metadata with the monitor
Cleaned up example notebooks

Fixes

Irrelevant reserved columns no longer presented in requests table
Column filtering did not dismiss in requests page
Button to create commit from UI was rendered for non-LLM projects
Navigating back from certain pages was broken
Dismissing modals caused the app to become unresponsive
Monitoring onboarding modal did not open
Production tests with subpopulation filters rendered incorrect insights in results graph
Clicking outside of dropdowns within a modal dismissed the whole modal
Improved discoverability of the data points that a test is run on in test diagnosis modal
Subsequent pages of monitoring requests would not always render
Some rows contained latency, cost, and tokens columns even if they were left unspecified
Suggested test modal reappeared unexpectedly
When table columns are very large, other columns were not readable
LLM rubric tests did not show score or explanations in monitoring
Requests pane was not scrollable
Some error states for test creation and results weren’t being shown
Column Value test title was not updating upon threshold change
Default color scheme to system
Added new and updated existing examples of how to incorporate the Openlayer TypeScript client for various use cases
Data table columns no longer cut off

Cost-per-request, new tests, subpopulation support for data tests, and more precise row filtering

Stop guessing. Ship with confidence.

The automated AI evaluation and monitoring platform.