Control how much you're spending with cost-per-request visibility and tests 💸

We’re excited to introduce the newest set of tests to hit Openlayer! Make sure column averages fall within a certain range with the Column average test. Ensure that your outputs contain specific keywords per request with our Column contains string test, where the values in Column B must contain the string values in Column A. Monitor and manage your costs by setting Max cost, Mean cost, and Total cost tests.

As additional support for managing costs, we now show you the cost of every request in the Requests page.

You can now filter data when creating integrity or consistency tests so that the results are calculated on specific subpopulations of your data, just like performance goals.

That’s not all, so make sure to read all the updates below. Join our Discord community to follow along on our development journey, and stay tuned for more updates from the changelog! 📩🤝

New features

• New tests
  • Column average test – make sure column averages fall within a range
  • Column contains string test – column B must contain the string in column A
  • Cost-related tests – max cost, mean cost, and total cost per evaluation window
• View your production data associated with each of your tests in monitoring mode
• Support for cost-per-request and cost graph
• Filter rows by row-level metrics such as conciseness
• Subpopulation support for data goals
• The timeline page is back - see how your commits perform on goals over time

Improvements

• Ability to update previously published production data by setting existing columns or adding new columns
• Sample requests are paginated
• Latency rendered in ms in the requests table
• Requests filters no longer require selecting a filter type
• Suggested tests modal auto-opens after project creation outside of the onboarding
• Notifications callout not shown until the project is fully setup
• Enabled filtering without datasets in development and monitoring modes
• Render cost in requests table
• Render monitoring data correctly in test diagnosis modals
• Row-level scores and explanations rendered for gpt-based metric tests
• Activity log is now collapsible
• Individual rows in data tables within the test diagnosis modal can be expanded
• Input and output columns rendered next to each other in data tables
• New example notebook showing how to send additional columns as metadata with the monitor
• Cleaned up example notebooks

Bug fixes

• Irrelevant reserved columns no longer presented in requests table
• Column filtering did not dismiss in requests page
• Button to create commit from UI was rendered for non-LLM projects
• Navigating back from certain pages was broken
• Dismissing modals caused the app to become unresponsive
• Monitoring onboarding modal did not open
• Production tests with subpopulation filters rendered incorrect insights in results graph
• Clicking outside of dropdowns within a modal dismissed the whole modal
• Improved discoverability of the data points that a test is run on in test diagnosis modal
• Subsequent pages of monitoring requests would not always render
• Some rows contained latency, cost, and tokens columns even if they were left unspecified
• Suggested test modal reappeared unexpectedly
• When table columns are very large, other columns were not readable
• LLM rubric tests did not show score or explanations in monitoring
• Requests pane was not scrollable
• Some error states for test creation and results weren’t being shown
• Column Value test title was not updating upon threshold change
• Default color scheme to system
• Added new and updated existing examples of how to incorporate the Openlayer TypeScript client for various use cases
• Data table columns no longer cut off