Opik

Opik is an open-source platform for evaluating, testing and monitoring LLM applications. Built by Comet.

You can use Opik for:

1. Development:

• Tracing: Track all LLM calls and traces during development and production (Quickstart, Integrations
• Annotations: Annotate your LLM calls by logging feedback scores using the Python SDK or the UI.

2. Evaluation: Automate the evaluation process of your LLM application:

• Datasets and Experiments: Store test cases and run experiments (Datasets, Evaluate your LLM Application)
• LLM as a judge metrics: Use Opik's LLM as a judge metric for complex issues like hallucination detection, moderation and RAG evaluation (Answer Relevance, Context Precision
• CI/CD integration: Run evaluations as part of your CI/CD pipeline using our PyTest integration

3. Production Monitoring: Monitor your LLM application in production and easily close the feedback loop by adding error traces to your evaluation datasets.