Updated

Promptfoo

Open-source CLI and library for testing prompts, models and RAG pipelines against test cases, and for red-teaming LLM apps.

8 open-source projects declare Promptfoo in their code; no maker's live product we track uses it yet.

Open sourceSelf-hostableFree tierPaid from Contact sales

Profile checked

Works with AI agents:llms.txtCLI
Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

How makers use Promptfoo

What Promptfoo is used for

AI Guardrails & Security · Red-team testing →Red-team test suites for prompt injection, jailbreaks and data leaks, run from the command line or CI.Trade-off: A test tool, not a runtime filter; you still need a guardrail on live traffic.
Lakera1Nightfall1Prompt Security1NeMo Guardrails0Guardrails AI0Promptfoo0Makers' products on each, for AI guardrails & security
LLM Observability & Evals · Eval and test CLI →Running prompts and models against test cases from the command line or CI, including red-team tests for prompt injection and data leaks.Trade-off: A testing tool rather than production tracing, so you pair it with something that logs live traffic.
Datadog50Langfuse35Helicone13LangSmith11Braintrust6Promptfoo0Makers' products on each, for LLM observability & evals

Promptfoo vs the alternatives makers chose

OptionMakers' productsFree forAt 10,000 usersOpen source
Datadog50 43%——No
Langfuse35 30%~5,000 users$29/mo · CoreYes
Helicone13 11%~1,000 users$116/mo · ProYes
LangSmith11 9%~520 users$475/mo · DeveloperNo
Braintrust6 5%——No
Weights & Biases3 3%——No
Promptfoo0 0%Any size—Yes

For LLM observability & evals: how many of the 117 makers' products we know the LLM observability & evals of chose each option, and what each costs at 10,000 users.

Costs are the cheapest plan that fits, from each vendor's pricing page, at the usage our stack planner assumes for a product with that many users. Free for: the users its cheapest plan a business can use stays free for.

When to use Promptfoo, and when not

Pick Promptfoo if

Something else fits better if

  • You want tracing, prompts and evals in one tool you can self-host for free: Langfuse.
  • You already use OpenTelemetry or want vendor-neutral instrumentation: Arize Phoenix.
  • You want request logs and costs today by changing one base URL: Helicone.
  • Your app is built on LangChain or LangGraph: LangSmith.
  • Your main question is whether a prompt or model change made outputs better or worse: Braintrust.
From each decision's guide, written from what makers' products chose and why.

How Promptfoo itself is built

26 tools, from its own code, website and Product Hunt page.

Who uses it

No maker's live product on record yet, and 8 open-source projects that declare it in their code.

We haven't found a maker's live product that uses Promptfoo yet — only the open-source projects below, which declare it in their code.

Open source: a project that declares Promptfoo as a dependency in its public code — verifiable, but not necessarily a live product.

Alternatives to Promptfoo

All alternatives by situation →

Questions makers ask about Promptfoo

What are the alternatives to Promptfoo?

For LLM observability & evals, the other options makers' products we track chose most are Datadog (50), Langfuse (35), Helicone (13), LangSmith (11), Braintrust (6), Weights & Biases (3). Open source or self-hostable among them: Langfuse, Helicone, Weights & Biases. Each alternative, with the situation it fits, is on the Promptfoo alternatives page.

When should you use Promptfoo?

You want to test your app for prompt injection and data leaks before release, in CI (AI guardrails & security). Something else fits better if: you want tracing, prompts and evals in one tool you can self-host for free, pick Langfuse; you already use OpenTelemetry or want vendor-neutral instrumentation, pick Arize Phoenix.

Is Promptfoo free?

Yes — there is a free tier a small product can run on; paid use starts at Contact sales. source ↗

Is Promptfoo open source or self-hostable?

Open source, and you can self-host it. source ↗

Can AI coding agents work with Promptfoo?

It serves an llms.txt docs index; it has an official CLI (promptfoo).

Who uses Promptfoo?

No maker's live product we track yet; 8 open-source projects declare it in their code. source ↗