
DeepEval
Open-source Python framework for unit-testing LLM apps with pytest-style test cases and ready-made metrics (faithfulness, answer relevancy, hallucination, G-Eval), from Confident AI.
12 open-source projects declare DeepEval in their code; no maker's live product we track uses it yet.
Profile checked
Works with AI agents:llms.txtCLIHow makers use DeepEval
What DeepEval is used for
DeepEval vs the alternatives makers chose
| Option | Makers' products | Free for | At 10,000 users | Open source | |
|---|---|---|---|---|---|
| 50 43% | — | — | No | ||
| 35 30% | ~5,000 users | $29/mo · Core | Yes | ||
| 13 11% | ~1,000 users | $116/mo · Pro | Yes | ||
| 11 9% | ~520 users | $475/mo · Developer | No | ||
| 6 5% | — | — | No | ||
| 3 3% | — | — | No | ||
| 0 0% | Any size | — | Yes |
For LLM observability & evals: how many of the 117 makers' products we know the LLM observability & evals of chose each option, and what each costs at 10,000 users.
Costs are the cheapest plan that fits, from each vendor's pricing page, at the usage our stack planner assumes for a product with that many users. Free for: the users its cheapest plan a business can use stays free for.
When to use DeepEval, and when not
Something else fits better if
- You want tracing, prompts and evals in one tool you can self-host for free: Langfuse.
- You already use OpenTelemetry or want vendor-neutral instrumentation: Arize Phoenix.
- You want request logs and costs today by changing one base URL: Helicone.
- Your app is built on LangChain or LangGraph: LangSmith.
- Your main question is whether a prompt or model change made outputs better or worse: Braintrust.
How DeepEval itself is built
21 tools, from its own code, website and Product Hunt page.
Reliability and open issues
The most wanted open issue on GitHub has 16 👍.
Most wanted on GitHub
- Don't push for confidently.ai platform, or give way to silence it👍 16 · opened Mar 2025 · active Aug 2026
- Remove unconditional, uncontrollable output statements👍 12 · opened May 2024 · active Mar 2026
- ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.👍 10 · opened Aug 2024 · active Feb 2026
Who uses it
No maker's live product on record yet, and 12 open-source projects that declare it in their code.
We haven't found a maker's live product that uses DeepEval yet — only the open-source projects below, which declare it in their code.
None of these in makers' own words.
Open source: a project that declares DeepEval as a dependency in its public code — verifiable, but not necessarily a live product.
Alternatives to DeepEval
All alternatives by situation →Questions makers ask about DeepEval
What are the alternatives to DeepEval?
For LLM observability & evals, the other options makers' products we track chose most are Datadog (50), Langfuse (35), Helicone (13), LangSmith (11), Braintrust (6), Weights & Biases (3). Open source or self-hostable among them: Langfuse, Helicone, Weights & Biases. Each alternative, with the situation it fits, is on the DeepEval alternatives page.
Is DeepEval free?
Yes — there is a free tier a small product can run on; paid use starts at $200/mo (Confident AI Starter). source ↗
Is DeepEval open source or self-hostable?
Open source, and you can self-host it. source ↗
Can AI coding agents work with DeepEval?
It serves an llms.txt docs index; it has an official CLI (deepeval).
Who uses DeepEval?
No maker's live product we track yet; 12 open-source projects declare it in their code. source ↗