Updated
Agent building blocks

LLM Observability & Evals

See what your prompts, models and agents actually did in production, and test changes against real examples before you ship them.

For your framework or language:Next.js
Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

Trends

Promptfoo gained the most ground in package downloads: 4.3× against the median npm package over a year.

Public counts each month: pull requests on GitHub, package downloads, stars, Product Hunt followers. They show direction, not market share.

ToolDownloadsStarsProduct Hunt
Datadog9.3M/wk−28% vs npm——
LangSmith6.2M/wk−8% vs npm—858 followers
Langfuse1.7M/wk+57% vs npm★ 35.7k2.3k followers
Braintrust1.6M/wk+106% vs npm——
Promptfoo695.7k/wk4.3× vs npm★ 26k—
DeepEval588.6k/wk★ 18.8k—
Ragas239k/wk★ 16k—
OpenLLMetryAcquired119.5k/wk+80% vs npm★ 7.5k—
Arize Phoenix80.4k/wk+172% vs npm★ 11.9k—
Laminar49.2k/wk3× vs npm★ 3.4k—
Opik36.9k/wk+50% vs npm★ 22.6k—
Lunary10k/wk−74% vs npm——
MLflow9.1k/wk★ 28.3k5 followers
Patronus AIArchived7.3k/wk——
PromptLayer4.8k/wk−62% vs npm★ 784—
AgentaArchived3k/wk★ 4.8k—
GalileoAcquired1.3k/wk★ 22—
HumanloopArchived714/wk——
RespanRenamed665/wk★ 1—
Athina AIArchived263/wk——
Parea AIArchived184/wk——
AgentOpsMaintenance only84/wk★ 5.9k—
HoneyHive73/wk★ 3—

The real choice

Open-source platform you can self-host, or a hosted eval product. Open-source tools keep traces on your own servers and cost nothing to run yourself, but you maintain them. Hosted eval platforms give you polished experiment and scoring workflows for a monthly bill, with your data in their cloud.

What makers chose81 live products, each with a source. Not market share.

Pick by situation

Tap the ones that are you — the tools that fit light up below.
  • You want tracing, prompts and evals in one tool you can self-host for freeLangfuse
  • You already use OpenTelemetry or want vendor-neutral instrumentationArize Phoenix
  • You want request logs and costs today by changing one base URLHelicone
  • Your app is built on LangChain or LangGraphLangSmith
  • Your main question is whether a prompt or model change made outputs better or worseBraintrust

Compare the tools

Grouped by the side of the choice they answer, not ranked. Open a row for when to use it, the trade-off and what makers say.

ToolBest for
Open-source platform2
Langfuseplatform · open source · free tierTracing, prompt management and evals in one tool you can run on your own server for free.35+58 open source$29Core1.7M/wk+57% vs npm

Use it whenYou want to own your trace data or keep costs flat as volume grows.

Trade-offSelf-hosting means running Postgres, ClickHouse, Redis and object storage alongside it; otherwise it's a usage-based cloud plan.

llms.txtCLI

Used by Agentplace.io, Conduit, Finyuus, Flowstep, giselle and 30 more · in 58 open-source projects.

Marc gave me an in-person onboarding in SF - I found an issue in our LLM provider config just 30 minutes after the onboarding thanks to Langfuse. 10/10 recommendation
stagewise, the makerSep 2026 ↗
Arize Phoenixservice · self-hostable · free tierOpenTelemetry-based tracing and evals you can start locally in a notebook, then self-host or move to its cloud.31 open source—80.4k/wk+172% vs npm

Use it whenYou already use OpenTelemetry or want vendor-neutral instrumentation.

Trade-offSource-available under the Elastic License rather than a permissive open-source license.

llms.txt

No maker's product we track shows it for this yet · in 31 open-source projects.

Proxy logging1
Heliconeplatform · open source · free tierGetting request logs, costs and latency by changing one base URL, with no SDK instrumentation.13$116Pro3.8k/wk

Use it whenYou want visibility today and your app makes direct model calls.

Trade-offProxy logging sees individual requests well but agent steps and eval workflows less deeply; acquired by Mintlify in March 2026, so check its roadmap before building on it.

llms.txt

Used by Alai, Codebuff, Fume, Persana, Pretty Prompt and 8 more.

Helicone AI offers open-source observability tools tailored for developers working with LLMs. It simplifies debugging and optimization, providing valuable insights into AI model performance.
Persana, the makerSep 2026 ↗
Hosted eval platform2
LangSmithplatform · free tierApps built on LangChain or LangGraph, where tracing works with almost no setup.11+62 open source$475Developer6.2M/wk−8% vs npm

Use it whenYou already use the LangChain stack.

Trade-offPaid per seat beyond the free tier; self-hosting is enterprise-only.

llms.txt

Used by Astrid, DryMerge, GitLaw, Intryc, NeuralAgent and 6 more · in 62 open-source projects.

LangSmith’s real-time analytics and versioning keep our AI agents rock-solid -- so everything just works better.
Watchman AI, the makerSep 2026 ↗
Braintrustplatform · free tierEval-driven work — scoring outputs and comparing prompts and models side by side in experiments.6+11 open source—1.6M/wk+106% vs npm

Use it whenYour main question is "did this change make outputs better or worse".

Trade-offClosed source, and the paid tier is priced for teams rather than hobby projects.

llms.txtMCPCLI

Used by Brew, Hey Noah, Arcade, Blacksmith, Linear and 1 more · in 11 open-source projects.

Cost: the cheapest plan that fits traces 100k traces, from list prices. Try your own numbers →

Cost as you grow

Each contender's cheapest usable plan as usage rises.

$0$1,000$5,000$20,0001k10k50k100k500k1M5M10M
LangSmithHeliconeArize PhoenixLangfusex: traces per month (trace) · cheapest usable plan at each point, list prices · try your own numbers

Who switches to what

Public pull requests on GitHub since Oct 2024 whose title says "X to Y" — real code changes, by developers in general rather than makers only. Pick a flow to see its pull requests.

LangSmith → Langfuse9 PRs · Compare →

Before you choose

How to approach it

Start by logging every request with its prompt, output, latency and cost — a day of real traces teaches more than any benchmark. Save the bad outputs you find as a small dataset, and rerun it every time you change a prompt or model. Add automated scoring only after you've read enough traces to know what "good" means for your app.

Common mistakes
  • Logging full prompts and outputs that contain users' personal data to a third-party service without masking it or listing the vendor as a subprocessor.
  • Changing a prompt in production with no saved examples to rerun, so a fix for one case silently breaks five others.
  • Trusting an LLM-as-judge score before checking that it agrees with your own judgment on a few dozen examples.

Other options

Real choices most makers here won't need to weigh.

DatadogHosted eval platformTraces, token costs, errors and quality and safety evaluations for LLM apps and agents, next to the APM and logs you already have in Datadog.50 makers' productsWeights & BiasesHosted eval platformTracing and evaluating LLM apps and agents with W&B Weave, in the same account you use for experiment tracking and a model registry when you train or fine-tune models.3 makers' productsRespanHosted eval platformRenamedTracing, monitoring, prompt management and LLM-judge evals for agents, with a built-in gateway that logs every call it routes.2 makers' productsOpikOpen-source platformTracing, evaluations and prompt optimization in an Apache-licensed platform you self-host with Docker or Kubernetes, or use on Comet's cloud.PromptfooEval and test CLIRunning prompts and models against test cases from the command line or CI, including red-team tests for prompt injection and data leaks.DeepEvalEval and test CLIWriting LLM evals as pytest-style tests in Python, with ready-made metrics for hallucination, faithfulness and answer relevancy.RagasEval and test CLIScoring RAG pipelines — how faithful answers are to the retrieved context and how relevant that context is — and generating test questions from your documents.MLflowOpen-source platformTracing, evaluating and versioning prompts for LLM apps in the same open-source platform many teams already use for ML experiments.LaminarOpen-source platformTracing, evals and debugging built for multi-step agents, self-hosted or hosted.AgentOpsHosted eval platformMaintenance onlyTracing, replaying and costing multi-agent runs from a Python SDK that hooks into CrewAI, AutoGen, OpenAI and other frameworks.OpenLLMetryOpen-source platformAcquiredOpen-source OpenTelemetry instrumentation that traces model, vector store and framework calls and sends them to any OpenTelemetry backend.HoneyHiveHosted eval platformTracing, online evaluation of live traffic, offline experiments on datasets, annotation queues and prompt versioning for agents.GalileoHosted eval platformAcquiredScoring every production request with small Luna evaluator models distilled from LLM-judge metrics, then turning those evals into guardrails.PromptLayerHosted eval platformA prompt registry non-engineers can edit and deploy without a redeploy, with request logs, traces and dataset evals tied to prompt versions.LunaryHosted eval platformLogging LLM calls with user and conversation analytics, prompt templates and human review.Confident AIHosted eval platformDashboards, history and regression tracking for DeepEval test runs, plus tracing, online evals and red-teaming in one hosted platform.HumanloopHosted eval platformShut down · Sep 2025Teams managing prompts, running evaluations and watching LLM apps in production.Patronus AIHosted eval platformShut downEvaluating and optimizing LLM apps and agents with research-based evaluators.AgentaOpen-source platformShut down · Jul 2026An open-source, self-hostable place to manage prompts, run evaluations and trace LLM apps.Parea AIHosted eval platformShut downExperiment tracking, evaluations and human annotation of LLM app logs.Athina AIHosted eval platformShut downMonitoring LLM apps in production with automatic hallucination detection and evals.

Decided alongside

What the 81 makers' products here chose for their other decisions.