A log of all the changes and improvements made to our app

Subscribe to the changelog

July 1st, 2024

Custom metrics, rotating API keys, and new models for direct-to-API calls

Make evals your own with custom metrics 🛠️📊

We understand that you may have metrics that are highly specific to your use case, and you want to use these alongside standard metrics to eval your AI systems. That’s why we built custom metrics. You can now upload any custom metric to Openlayer simply by specifying them in your openlayer.json file. These metrics can then be used in a number of ways on platform; they’ll show up as metric tests that you can run, or as project-wide metrics that will be computed on all of your data. Now, your evals on Openlayer are more comprehensive than ever.

We’ve shipped more exciting features and improvements this month, including the ability to create multiple API keys, and a bunch of new models available for direct-to-API calls, so be sure to read below for a full list of updates!

New features

  • Custom metrics
    • Upload your own custom metrics to Openlayer, which can be used:
      • as project-wide metrics
      • as tests
  • Create multiple Openlayer API keys
    • create new personal Openlayer API keys so that you can rotate API keys
    • rename and delete keys
  • Specify desired metrics in openlayer.json
  • New models available for direct-to-API calls
    • GPT-4o
    • GPT-4 Turbo
    • Claude 3.5 Sonnet
    • Claude 3 Haiku
    • Claude 3 Opus
    • Claude 3 Sonnet
    • Command R
    • Command R Plus
    • Gemini 1.0 Flash
    • Gemini 1.5 Flash
    • Gemini 1.5 Pro


  • test creation page design improvements
  • link to git repository and organization in git settings pages
  • add button to view status of commit during processing in project loading state
  • updated solid danger buttons’ shade of red
  • modal background overlay opacity is no longer too light
  • toast messages no longer overflow the page
  • improved text sizing in various places
  • added error toast when Assistant requests fail
  • navigation polish
  • different icons for different commit sources
  • suggested titles for GPT evaluation tests now reference the criteria name
  • improvements to docs
    • updated Python code snippets with the new SDK syntax
    • tracing for Anthropic models
    • updated example notebook links

Bug fixes

  • commit log processing time does not use relative time
  • new users that were invited to a workspace do not auto-navigate to invites page
  • help breadcrumb is hidden and shows in place of user dropdown options
  • cost values close to 0 rendered as $0.00
  • progress bars did not render in chrome
  • test metadata disappeared entirely when collapsed
  • navigating to project from breadcrumb prevents back navigation
  • switching projects prevented back navigation
  • creating commit from the UI does not generate outputs
  • commit processing icon was broken in navigation dropdowns
  • activity log overflows screen