Tracely

Tracely is an open-source (MIT) observability and evaluation platform for LLM and AI agent applications, and a self-hostable alternative to Langfuse, LangSmith and Braintrust.

Every eval tool asks you to hand-author a dataset: invent questions, write ideal answers, keep them current as the product changes. That dataset is a guess about what might break. Production already handed you the real thing — a trace of the exact run that failed, with the exact input, tool calls and model responses.

Traces arrive over OpenTelemetry (OTLP) from the Python SDK or any OTel-compatible instrumentation. Evaluators — deterministic structural checks or LLM-as-a-judge rubrics — grade each trace as it lands. Failures are automatically clustered into issues so you see the pattern rather than a wall of individual bad runs. Any cluster can then be frozen into a hermetic, replayable regression case that runs in CI and blocks the pull request that would reintroduce it.

Key features:

• Full trace and span tracing for agents, tools and LLM calls, with conversation replay
• Automatic evaluation on ingest: structural evaluators and LLM-as-a-judge
• Failure clustering and trend analysis across releases
• Hermetic regression cases replayed in CI, with a pass/fail commit status and PR comment
• Judge-vs-human calibration to measure how much you can trust your evaluators
• Scenario simulation, including adversarial runs
• MCP server, so coding agents can query your traces directly
• Bring your own model key; no vendor lock-in

Self-host the entire stack — API, worker, UI, Postgres, ClickHouse, Redis and object storage — with Docker Compose or a one-click Railway template. A hosted free tier is available at 20,000 traces per month.