DeepEval

Open-source Python framework for unit-testing LLM apps with pytest-style test cases and ready-made metrics (faithfulness, answer relevancy, hallucination, G-Eval), from Confident AI.

Visit deepeval.com ↗Pricing & calculator Alternatives

Works with AI agents:llms.txt CLI

Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

Where it fits

LLM Observability & Evals · Eval and test CLIWriting LLM evals as pytest-style tests in Python, with ready-made metrics for hallucination, faithfulness and answer relevancy.Trade-off: The metrics use an LLM as a judge, so every run costs tokens and scores vary slightly; dashboards and history are in the paid Confident AI platform.

How DeepEval itself is built

19 tools, from its own code, website and Product Hunt page.

Agent Development Kit

Who uses it

No maker's live product on record yet, and 3 open-source projects that declare it in their code.

Declared in code 3How evidence is collected →

We haven't found a maker's live product that uses DeepEval yet — only the open-source projects below, which declare it in their code.

Sort

beeai-framework Build production-ready AI agents in both Python and Typescript.LLM Observability & EvalsIn its code · source ↗

deepteam DeepTeam is a framework to red team LLMs and AI agents.LLM Observability & EvalsIn its code · source ↗

waku-agent Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop…LLM Observability & EvalsIn its code · source ↗

Open source: a project that declares DeepEval as a dependency in its public code — verifiable, but not necessarily a live product.

Reliability and open issues

Most wanted on GitHub

Don't push for confidently.ai platform, or give way to silence it👍 16 · opened Mar 2025 · active Aug 2026
Remove unconditional, uncontrollable output statements👍 12 · opened May 2024 · active Mar 2026
ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.👍 10 · opened Aug 2024 · active Feb 2026

Open on 2026-10-04 in confident-ai/deepeval, with activity in the last year — issues and feature requests by 👍.

Alternatives to DeepEval

All alternatives by situation →

LangfuseTracing, prompt management and evals in one tool you can run on your own server for free.Open sourceFree tier33 makers · LLM observability & evals

Arize PhoenixOpenTelemetry-based tracing and evals you can start locally in a notebook, then self-host or move to its cloud.Self-hostableFree tier

HeliconeGetting request logs, costs and latency by changing one base URL, with no SDK instrumentation.Open sourceFree tier13 makers · LLM observability & evals

LangSmithApps built on LangChain or LangGraph, where tracing works with almost no setup.Free tier11 makers · LLM observability & evals

BraintrustEval-driven work — scoring outputs and comparing prompts and models side by side in experiments.Free tier5 makers · LLM observability & evals

OpikTracing, evaluations and prompt optimization in an Apache-licensed platform you self-host with Docker or Kubernetes, or use on Comet's cloud.Open sourceFree tier

PromptfooRunning prompts and models against test cases from the command line or CI, including red-team tests for prompt injection and data leaks.Open sourceFree tier

RagasScoring RAG pipelines — how faithful answers are to the retrieved context and how relevant that context is — and generating test questions from your documents.Open sourceFree tier

Questions makers ask about DeepEval

Is DeepEval free?

Yes — there is a free tier a small product can run on; paid use starts at $200/mo (Confident AI Starter). source ↗

Is DeepEval open source or self-hostable?

Open source, and you can self-host it. source ↗

Can AI coding agents work with DeepEval?

It serves an llms.txt docs index; it has an official CLI (deepeval).

Who uses DeepEval?

No maker's live product we track yet; 3 open-source projects declare it in their code. source ↗