promptfoo
Open sourceUpdated recentlyPricing may have changed — not yet verifiedOpen-source CLI for LLM red-teaming, eval matrices and CI gating.
by promptfoo · Primary workflow: Local-first LLM evaluation and adversarial testing in CI
scored
No verified editorial score yet. Scores appear only after documented evaluation.
What it is
promptfoo evaluates prompt/model configurations against test cases entirely locally if desired, producing matrices of outputs and scores. Red-team plugins probe injection/jailbreak classes; YAML configs make it pipeline-friendly.
Editorial basis: vendor documentation and public information reviewed by our editors; written originally for this catalog. Hands-on testing has not yet been performed. Last editorial update 2026-08-26.
Editorial assessment
- Security-conscious teams avoiding cloud eval
- Prompt regression gates before deploy
- GUI-only stakeholders
- Managed annotation workflows out of the box
- Runs fully local against your keys
- Red-team suite covers OWASP LLM threat classes
- CLI ergonomics over polish
- Reporting simpler than SaaS peers
Hands-on evidence
Vendor information reviewed; hands-on testing not yet completed.Vendor information reviewed; hands-on testing not yet completed.
When we test this tool ourselves, this section will document the test date, exact scenario, setup experience, example output, integration result and discovered limitations — with reproducibility notes so you can replicate it.
Read our evidence policy →Pricing
Open source
FreeUnverified
Pricing reflects publicly listed information at research time and is marked unverified until an editor re-checks the vendor’s pricing page. Always confirm on the official site.
Integrations & compatibility
- Deployment
- CLI, Self-hosted CI
- Platforms
- Windows, macOS, Linux
- Languages / SDKs
- TypeScript, JavaScript, Python
- API available
- Yes
- Learning curve
- Low
- Documentation quality
- GOOD
- Team fit
- Solo, Small team, Mid-size
- Support
- See vendor
Security, privacy & AI transparency
Orchestrates third-party LLMs as judges/red-teamers; configuration transparent in repo.
Vendor response
Vendors can respond to listings and reviews through the vendor portal. Responses are moderated and labeled. No response has been received for this listing yet.