Open-source platformBest forTracing, prompt management and evals in one tool you can run on your own server for free.
Trade-offSelf-hosting means running Postgres, ClickHouse, Redis and object storage alongside it; otherwise it's a usage-based cloud plan.
Used by 35 makers' products for this · LangSmith vs Langfuse →
Proxy loggingBest forGetting request logs, costs and latency by changing one base URL, with no SDK instrumentation.
Trade-offProxy logging sees individual requests well but agent steps and eval workflows less deeply; acquired by Mintlify in March 2026, so check its roadmap before building on it.
Used by 13 makers' products for this · LangSmith vs Helicone →
Hosted eval platformBest forEval-driven work — scoring outputs and comparing prompts and models side by side in experiments.
Trade-offClosed source, and the paid tier is priced for teams rather than hobby projects.
Used by 6 makers' products for this · LangSmith vs Braintrust →
Open-source platformSelf-hostableFree tierFrom $50/mollms.txt Best forOpenTelemetry-based tracing and evals you can start locally in a notebook, then self-host or move to its cloud.
Trade-offSource-available under the Elastic License rather than a permissive open-source license.
No maker product tracked for this yet · LangSmith vs Arize Phoenix →
Hosted eval platformBest forTraces, token costs, errors and quality and safety evaluations for LLM apps and agents, next to the APM and logs you already have in Datadog.
Trade-offBilled as another Datadog product on top of your plan, and only worth it if the rest of your monitoring lives there; formerly called LLM Observability.
Used by 50 makers' products for this
Hosted eval platformSelf-hostableFree tierFrom $60/mo
Best forTracing and evaluating LLM apps and agents with W&B Weave, in the same account you use for experiment tracking and a model registry when you train or fine-tune models.
Trade-offSelf-hosting needs a licensed W&B Server on Kubernetes with ClickHouse; the free plan ingests 1 GB of trace data a month; now sold as CoreWeave Forge, which names its Agent Lens product as Weave's successor for agent tracing.
Used by 3 makers' products for this
Hosted eval platformFree tierFrom $199/mo
Best forTracing, monitoring, prompt management and LLM-judge evals for agents, with a built-in gateway that logs every call it routes.
Trade-offCloud-only below Enterprise; the free plan keeps logs 7 days, and Team starts at $199/mo billed yearly.
Used by 2 makers' products for this
Open-source platformBest forTracing, evaluations and prompt optimization in an Apache-licensed platform you self-host with Docker or Kubernetes, or use on Comet's cloud.
Trade-offSelf-hosting means running several services; smaller community than Langfuse.
No maker product tracked for this yet
Eval and test CLIBest forRunning prompts and models against test cases from the command line or CI, including red-team tests for prompt injection and data leaks.
Trade-offA testing tool rather than production tracing, so you pair it with something that logs live traffic.
No maker product tracked for this yet
Eval and test CLIOpen sourceFree tierFrom $200/mo (Confident AI Starter)llms.txtCLI Best forWriting LLM evals as pytest-style tests in Python, with ready-made metrics for hallucination, faithfulness and answer relevancy.
Trade-offThe metrics use an LLM as a judge, so every run costs tokens and scores vary slightly; dashboards and history are in the paid Confident AI platform.
No maker product tracked for this yet
Eval and test CLIBest forScoring RAG pipelines — how faithful answers are to the retrieved context and how relevant that context is — and generating test questions from your documents.
Trade-offA Python library, not a platform — no tracing of live traffic or UI, and LLM-judged metrics cost tokens per run.
No maker product tracked for this yet · LangSmith vs Ragas →
Open-source platformBest forTracing, evaluating and versioning prompts for LLM apps in the same open-source platform many teams already use for ML experiments.
Trade-offGrew out of classic ML tooling, so the UI and concepts are broader than an LLM-only tool; you run the tracking server yourself unless you use a managed version.
No maker product tracked for this yet · LangSmith vs MLflow →
Open-source platformOpen source
Best forTracing, evals and debugging built for multi-step agents, self-hosted or hosted.
Trade-offSelf-hosting means running its services yourself; the hosted version saves that work.
No maker product tracked for this yet
Hosted eval platformOpen sourceFree tierFrom $40/mo
Best forTracing, replaying and costing multi-agent runs from a Python SDK that hooks into CrewAI, AutoGen, OpenAI and other frameworks.
Trade-offThe Python SDK hasn't had a PyPI release since August 2025, so check that it supports your framework versions.
No maker product tracked for this yet
Open-source platformOpen sourceFree tier
Best forOpen-source OpenTelemetry instrumentation that traces model, vector store and framework calls and sends them to any OpenTelemetry backend.
Trade-offInstrumentation only — you still need a backend to store and view traces; Traceloop, its maker, is joining ServiceNow.
No maker product tracked for this yet
Hosted eval platformSelf-hostableFree tierFrom Custom (Enterprise)
Best forTracing, online evaluation of live traffic, offline experiments on datasets, annotation queues and prompt versioning for agents.
Trade-offThe free plan stops at 10,000 events a month and 30 days of retention, and the only paid step is a custom-priced Enterprise plan.
No maker product tracked for this yet
Hosted eval platformSelf-hostableFree tierFrom $100/mo (billed annually)
Best forScoring every production request with small Luna evaluator models distilled from LLM-judge metrics, then turning those evals into guardrails.
Trade-offNow owned by Cisco and priced through Splunk, billed annually; built for enterprise teams more than small ones.
No maker product tracked for this yet
Hosted eval platformFree tierFrom $49/mo
Best forA prompt registry non-engineers can edit and deploy without a redeploy, with request logs, traces and dataset evals tied to prompt versions.
Trade-offHosted only below Enterprise; the free plan logs 2,500 requests a month, and usage past plan limits is billed per request.
No maker product tracked for this yet
Hosted eval platformSelf-hostableFrom $20/user/mo
Best forLogging LLM calls with user and conversation analytics, prompt templates and human review.
Trade-offNo free workspace for new sign-ups ($20 per user a month), and self-hosting is now an Enterprise feature.
No maker product tracked for this yet
Hosted eval platformFree tierFrom $200/mo
Best forDashboards, history and regression tracking for DeepEval test runs, plus tracing, online evals and red-teaming in one hosted platform.
Trade-offThe free plan allows 5 test runs a week and one project; the next step is $200 a month.
No maker product tracked for this yet
Best forTeams managing prompts, running evaluations and watching LLM apps in production.
Trade-offShut down on 2025-09-08 after the team joined Anthropic; all data was deleted, and Humanloop pointed users to Langfuse or Braintrust.
No maker product tracked for this yet
Hosted eval platformBest forEvaluating and optimizing LLM apps and agents with research-based evaluators.
Trade-offPivoted to a research lab; there is no self-serve evaluation product on its site.
No maker product tracked for this yet
Open-source platformOpen source
Best forAn open-source, self-hostable place to manage prompts, run evaluations and trace LLM apps.
Trade-offPivoted on 2026-07-21 to a workspace for building agents; no longer an observability product.
No maker product tracked for this yet
Hosted eval platformBest forExperiment tracking, evaluations and human annotation of LLM app logs.
Trade-offThe site returns HTTP 402 and the SDK stopped in February 2025; apparently shut down without notice.
No maker product tracked for this yet
Hosted eval platformBest forMonitoring LLM apps in production with automatic hallucination detection and evals.
Trade-offThe site and app are down and the SDK stopped in May 2025; apparently shut down without notice.
No maker product tracked for this yet