Drift detection
Versioned drift profiles evaluated on a schedule. PSI for distribution shift, SPC for control limits and WECO rules, custom metrics for anything else. Alerts via Slack, OpsGenie, or console.
Scouter
Quality Control for AI - Monitoring, Observability, and Agentic Evaluation
Quick navigation
Section titled “Quick navigation” Installation Install the Python client and connect to a running server.
Server setup Run Scouter locally with Docker or deploy to production.
Monitoring Drift detection for traditional ML
Tracing OpenTelemetry-compatible distributed tracing for AI services.
Agent evaluation End-to-end evaluation platform for agentic systems.
Architecture System overview, data flow, and component diagrams.
What Scouter does
Section titled “What Scouter does”Distributed tracing
Drop-in OpenTelemetry tracer provider. Send spans to Scouter, forward to an external collector, or both. Stored in Delta Lake for analytical queries.
Agent evaluation
Deterministic assertions, LLM judge tasks, and agent-specific checks. Same task graph works offline before a release or online against sampled production traffic.
Bifrost data bridge
High-throughput data transport between Python and Rust. Sub-microsecond record insertion via gRPC or HTTP with configurable queue backends.
Architecture in one paragraph
Section titled “Architecture in one paragraph”The server is Rust. PostgreSQL stores recent operational data. DataFusion handles long-term archival and analytical reads. The Python package wraps Rust types through PyO3, which keeps the API small and the hot path fast.
Reference
Section titled “Reference”- API reference — top-level Python exports
- Architecture — runtime components and data flow
- Tech specs — design notes for queueing, archival, gRPC, and evaluation internals
- Evaluation platform — deep dive on online and offline evaluation