Improved quality control over your LLM’s responses with annotations and human feedback
Setting up alerts is an essential first step to monitoring your LLMs, but in order to understand why issues arise in production, it’s helpful to have human eyes to review requests.
This process is now easier than ever in Openlayer - you can add annotations to any requests, with custom values. If you’ve set up tracing, you can annotate each individual step of the trace for more granularity.
Every request can also be rated as a thumbs up or thumbs down, making it easy to scan through good and bad responses and figure out where your model is going wrong.
We’ve released some other huge features and improvements this month, so make sure to read the full changelog below!
Features
- •UIAbility to export data from the UI (Now you can download requests data right from the workspace. This is especially helpful if you’ve applied filters and want to download the filtered cohort of data)
- •UIUpdated navigation (Our navigation has a new layout featuring breadcrumbs at the top, making it much easier to navigate between projects and understand the hierarchy)
- •UIAnnotation and human feedback (You can now annotate any request with custom values. You can also give every request a thumbs up or thumbs down to make identifying error patterns even easier)
Improvements
- •TemplatesMore project templates
- •SDKsImproved OpenAI SDK
- •UIImprovements to billing page in settings
- •IntegrationsProject-level Git repository settings now available
- •IntegrationsAbility to now edit branch and root directory in project-level git settings
- •UIWith new navigation, ability to copy project name and inference pipeline ID
- •UIAbility to add ground truths to requests and edit existing ground truths
- •UIData in individual test modals is now filtered by selected evaluation window
Fixes
- •PerformanceSome tests were improperly skipped
- •UIOpenlayer Assistant was broken
- •SecuritySpaces in No PII test caused errors
- •PerformanceIssue with metric tests when input variable names were null
- •UIHovering over graph with no results shows broken tooltip