SAML Directory Sync, new LLM-as-a-judge models, and website refresh
Published January 16, 2025
We’ve added lots of features and enhancements across our platform, focused on improving performance, expanding functionality, and streamlining workflows. To highlight a few:
🔐 Increased security with SAML SSO directory sync. You can now sync SAML SSO on Openlayer with existing security groups. Openlayer can now more seamlessly fit into your organization’s security policies.
🧑⚖️ New LLM-as-a-judge models. We’ve expanded the models available to act as judges for LLM-as-a-judge tests. You can now use models from Cohere and Vertex AI when running these tests.
🎨 Website refresh. We’ve given our website a brand refresh, including lots of fun animations showcasing the Openlayer product in action and case studies from some of our customers.
Features
- SDKsFaster batch uploads with pyarrow support
- SDKsPush commits to the platform via the Python SDK
- UI/UXTabular view of test results in test modals
- UI/UXAdd pie graph for test results in project home
- EvalsAdd Faithfulness and Answer Correctness metrics for RAG systems
- PlatformUse Cohere, Vertex AI models as options for LLM-as-a-judge metrics
- APIAdd `expand` to inference pipeline GETs so projects and workspaces are included in the response body
- PlatformNew "Viewer" role in workspaces that doesn’t have write, update or delete permissions on resources
- SDKsSupport for async data uploads, and faster upload speeds
- PlatformDirectory sync with SAML
Improvements
- APILower latency for data stream endpoint
- UI/UXUpdate tooltips and rendering of statuses in test cards
- UI/UXMake sections in test modals collapsible
- APIAdd skipped and failing test counts in project version and inference pipeline objects
- APIBetter error messages for invalid data configs when streaming data
- PlatformMore intuitive status messages for skipped tests
- DocumentationAdd code samples in Java
Fixes
- PlatformGenerate outputs step was not failing gracefully
- UI/UXSurface user-facing error messages upon SSO login failures
- UI/UXBetter failure message when password reset link has expired
- PlatformImproved rate limiting
- IntegrationsSlack notifications for create pipeline now includes name
- PlatformAnswer Correctness metric was breaking when output was not a string
