Evals for LLMs, real-time monitoring, Slack notifications and so much more!
Published October 24, 2023
It’s been a couple of months since we posted our last update, but not without good reason! Our team has been cranking away at our two most requested features: support for LLMs and real-time monitoring / observability. We’re so excited to share that they are both finally here! 🚀
We’ve also added a Slack integration, so you can receive all your Openlayer notifications right where you work. Additionally, you’ll find tons of improvements and bug fixes that should make your experience using the app much smoother.
We’ve also upgraded all Sandbox accounts to a free Starter plan that allows you to create your own project in development and production mode. We hope you find this useful!
Join our Discord for more updates like this and get closer to our development journey!
Features
- PlatformLLMs in development mode (Experiment with and version different prompts, model providers and chains, Create a new commit entirely in the UI with our prompt playground. Connects seamlessly with OpenAI, Anthropic and Cohere, Set up sophisticated tests around RAG (hallucination, harmfulness etc.), regex validation, json schemas, and much more)
- PlatformLLMs in monitoring mode (Seamlessly evaluate responses in production with the same tests you used in development and measure token usage, latency, drift and data volume too)
- ObservabilityAll existing tasks support monitoring mode as well
- UI/UXToggle between development mode and monitoring mode for any project
- SDKsAdd a few lines of code to your model’s inference pipeline to start monitoring production data
- CollaborationSlack & email notifications (Setup personal and team notifications, Get alerted on goal status updates in development and production, team activity like comments, and other updates in your workspace)
- TemplatesSeveral new tests across all AI task types
- TemplatesNew sample project for tabular regression
- EvalsSelect and star the metrics you care about for each project
- SecurityAdd encrypted workspace secrets your models can rely on
Improvements
- UI/UXRevamped onboarding for more guidance on how to get started quick with Openlayer in development and production
- UI/UXBetter names for suggested tests
- UI/UXAdd search bar to filter integrity and consistency goals in create page
- PerformanceReduce feature profile size for better app performance
- UI/UXAdd test activity item for suggestion accepted
- UI/UXImproved commit history allows for better comparison of the changes in performance between versions of your model and data across chosen metrics and goals
- UI/UXAdded indicators to the aggregate metrics in the project page that indicate how they have changed from the previous commit in development mode
- PlatformImproved logic for skipping or failing tests that don’t apply
- UI/UXUpdated design of the performance goal creation page for a more efficient and clear UX
- PlatformAllow specifying MAPE as metric for the regression heatmap
- PerformanceImprovements to data tables throughout the app, including better performance and faster loading times
- UI/UXImproved UX for viewing performance insights across cohorts of your data in various distribution tables and graphs
- UI/UXUpdated and added new tooltips throughout the app for better clarity of concepts
Fixes
- UI/UXDownloading commit artifacts triggered duplicate downloads
- PerformanceFixed lagginess when browsing large amounts of data in tables throughout the app
- UI/UXValid subpopulation filters sometimes rendered empty data table
- UI/UXFixed bugs affecting experience navigating through pages in the app
- UI/UXFixed issues affecting the ability to download data and logs from the app
- UI/UXFiltering by tokens in token cloud insight would not always apply correctly
- UI/UXFixed UI bugs affecting the layout of various pages throughout the app that caused content to be cut off
- SDKsFixed Python client commit upload issues
