How Makers Build
Check my stackPlan my stack
Menu

Promptfoo

Open-source CLI and library for testing prompts, models and RAG pipelines against test cases, and for red-teaming LLM apps.

Visit promptfoo.dev ↗Pricing & calculatorAlternatives
Works with AI agents:llms.txtCLI
Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

Where it fits

LLM Observability & Evals · Eval and test CLIRunning prompts and models against test cases from the command line or CI, including red-team tests for prompt injection and data leaks.Trade-off: A testing tool rather than production tracing, so you pair it with something that logs live traffic.

How Promptfoo itself is built

4 tools, from its own code, website and Product Hunt page.

Cloudflare Web AnalyticsFly.ioNext.jsPostmark

Who uses it

No maker's live product on record yet, and 8 open-source projects that declare it in their code.

Declared in code 8How evidence is collected →

We haven't found a maker's live product that uses Promptfoo yet — only the open-source projects below, which declare it in their code.

Open source: a project that declares Promptfoo as a dependency in its public code — verifiable, but not necessarily a live product.

Alternatives to Promptfoo

All alternatives by situation →
LangfuseTracing, prompt management and evals in one tool you can run on your own server for free.Open sourceFree tier33 makers · LLM observability & evals
Arize PhoenixOpenTelemetry-based tracing and evals you can start locally in a notebook, then self-host or move to its cloud.Self-hostableFree tier
HeliconeGetting request logs, costs and latency by changing one base URL, with no SDK instrumentation.Open sourceFree tier13 makers · LLM observability & evals
LangSmithApps built on LangChain or LangGraph, where tracing works with almost no setup.Free tier11 makers · LLM observability & evals
BraintrustEval-driven work — scoring outputs and comparing prompts and models side by side in experiments.Free tier5 makers · LLM observability & evals
OpikTracing, evaluations and prompt optimization in an Apache-licensed platform you self-host with Docker or Kubernetes, or use on Comet's cloud.Open sourceFree tier
DeepEvalWriting LLM evals as pytest-style tests in Python, with ready-made metrics for hallucination, faithfulness and answer relevancy.Open sourceFree tier
RagasScoring RAG pipelines — how faithful answers are to the retrieved context and how relevant that context is — and generating test questions from your documents.Open sourceFree tier

Questions makers ask about Promptfoo

Is Promptfoo free?

Yes — there is a free tier a small product can run on; paid use starts at Contact sales. source ↗

Is Promptfoo open source or self-hostable?

Open source, and you can self-host it. source ↗

Can AI coding agents work with Promptfoo?

It serves an llms.txt docs index; it has an official CLI (promptfoo).

Who uses Promptfoo?

No maker's live product we track yet; 8 open-source projects declare it in their code. source ↗