apis.yml

aid: prompt-foo
name: Promptfoo
description: >-
  Promptfoo is an open-source CLI and TypeScript/Node.js library for
  evaluating, red-teaming, and security-testing LLM applications,
  agents, and RAG pipelines. It runs deterministic prompt evals with
  model-graded and rule-based assertions, generates dynamic adversarial
  attack probes across 50+ vulnerability categories (prompt injection,
  jailbreaks, RAG poisoning, PII leakage, harmful content, business
  rule violations), and integrates with CI/CD via a GitHub Action and
  a `code-scans` command for pull-request review. Promptfoo supports
  dozens of LLM providers — OpenAI, Anthropic Claude, Google Gemini,
  AWS Bedrock/SageMaker, Azure OpenAI, Mistral, Cohere, Groq,
  DeepSeek, Together, Fireworks, OpenRouter, LiteLLM, Vercel and
  Cloudflare AI gateways — plus local runtimes (Ollama, LocalAI,
  llama.cpp, vLLM, llamafile, Docker Model Runner) and custom
  HTTP/WebSocket/Python/JavaScript/Go/Ruby/Shell providers. It also
  ships an MCP server (`promptfoo mcp`), a Model Audit scanner
  (`scan-model`) for malicious ML artifacts, and a hosted Enterprise
  tier with team sharing, continuous monitoring, SSO, and a
  centralized compliance dashboard aligned to OWASP LLM Top 10, NIST
  AI RMF, MITRE ATLAS, and the EU AI Act. The project is MIT-licensed,
  has 21k+ GitHub stars, and is now part of OpenAI while remaining
  open source.
type: Index
position: Provider
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
tags:
  - LLM Evaluation
  - LLM Red Teaming
  - LLM Security
  - AI Security
  - Prompt Engineering
  - Vulnerability Scanning
  - Adversarial Testing
  - Jailbreak Testing
  - Prompt Injection
  - RAG
  - Agents
  - MCP
  - Model Audit
  - OWASP LLM Top 10
  - NIST AI RMF
  - MITRE ATLAS
  - EU AI Act
  - CI/CD
  - Open Source
  - TypeScript
url: https://raw.githubusercontent.com/api-evangelist/prompt-foo/refs/heads/main/apis.yml
created: '2026-05-25'
modified: '2026-05-25'
specificationVersion: '0.20'
apis: []
common:
  - type: TrustCenter
    url: security/prompt-foo-trust-center.yml
  - type: DomainSecurity
    url: security/prompt-foo-domain-security.yml
  - type: Website
    url: https://www.promptfoo.dev
  - type: Portal
    url: https://www.promptfoo.dev
  - type: Documentation
    url: https://www.promptfoo.dev/docs/intro/
  - type: GettingStarted
    url: https://www.promptfoo.dev/docs/getting-started/
  - type: Documentation
    name: CLI Reference
    url: https://www.promptfoo.dev/docs/usage/command-line/
  - type: Documentation
    name: Red Team Quickstart
    url: https://www.promptfoo.dev/docs/red-team/quickstart/
  - type: Documentation
    name: Providers
    url: https://www.promptfoo.dev/docs/providers/
  - type: Documentation
    name: Assertions & Metrics
    url: https://www.promptfoo.dev/docs/configuration/expected-outputs/
  - type: Documentation
    name: API Reference
    url: https://www.promptfoo.dev/docs/api-reference/
  - type: Pricing
    url: https://www.promptfoo.dev/pricing/
  - type: Blog
    url: https://www.promptfoo.dev/blog/
  - type: StatusPage
    url: https://status.promptfoo.app
  - type: TrustCenter
    url: https://trust.promptfoo.dev
  - type: GitHubOrganization
    url: https://github.com/promptfoo
  - type: SourceCode
    name: promptfoo (CLI + library)
    url: https://github.com/promptfoo/promptfoo
  - type: SDKs
    name: Python wrapper for the promptfoo CLI
    url: https://github.com/promptfoo/promptfoo-python
  - type: Tools
    name: Promptfoo GitHub Action
    url: https://github.com/promptfoo/promptfoo-action
  - type: Tools
    name: ModelAudit — ML model security scanner
    url: https://github.com/promptfoo/modelaudit
  - type: Tools
    name: MCP Agent Provider (custom provider for testing MCP servers)
    url: https://github.com/promptfoo/mcp-agent-provider
  - type: Tools
    name: evil-mcp-server (red-team MCP target)
    url: https://github.com/promptfoo/evil-mcp-server
  - type: CodeExamples
    name: example-app (industry/model/security demo app)
    url: https://github.com/promptfoo/example-app
  - type: CodeExamples
    name: demo-app
    url: https://github.com/promptfoo/demo-app
  - type: Library
    name: js-rouge (ROUGE summarization metric)
    url: https://github.com/promptfoo/js-rouge
  - type: Package
    name: npm — promptfoo
    url: https://www.npmjs.com/package/promptfoo
  - type: Package
    name: PyPI — promptfoo
    url: https://pypi.org/project/promptfoo/
  - type: Package
    name: Homebrew — promptfoo
    url: https://formulae.brew.sh/formula/promptfoo
  - type: HelmChart
    name: Helm chart for self-hosted Promptfoo
    url: https://github.com/promptfoo/promptfoo/tree/main/helm/chart/promptfoo
  - type: License
    name: MIT License
    url: https://github.com/promptfoo/promptfoo/blob/main/LICENSE
  - type: Forums
    name: Discord community
    url: https://discord.gg/promptfoo
  - type: Twitter
    url: https://twitter.com/promptfoo
  - type: LinkedIn
    url: https://www.linkedin.com/company/promptfoo
  - type: ChangeLog
    url: https://github.com/promptfoo/promptfoo/releases
  - type: Features
    data:
      - Open-source CLI and Node.js/TypeScript library for LLM evaluation and red teaming
      - >-
        `promptfoo eval` — run deterministic prompt/model evals with caching,
        concurrency, and live reload
      - >-
        `promptfoo redteam` — generate and run dynamic adversarial probes
        across 50+ vulnerability categories
      - >-
        `promptfoo view` — local browser UI for browsing eval and red-team
        results
      - >-
        `promptfoo share` — publish a shareable URL for an eval or model audit
      - >-
        `promptfoo generate` — synthesize datasets, red-team tests, and
        assertions
      - "`promptfoo optimize` — improve prompts against a target provider"
      - "`promptfoo scan-model` — security-scan ML model files (ModelAudit)"
      - >-
        `promptfoo code-scans` — scan code changes for LLM security
        vulnerabilities in IDEs and CI/CD
      - >-
        `promptfoo mcp` — expose promptfoo tools as a Model Context Protocol
        server
      - "`promptfoo retry`, `list`, `export`, `import`, `validate`, `debug`, `cache`, `auth`"
      - >-
        Assertions: rule-based (equals, contains, regex, javascript, python,
        cost, latency) and model-graded (llm-rubric, classifier, factuality,
        answer-relevance, similarity)
      - >-
        Red-team plugins covering prompt injection, jailbreaks, PII leakage,
        harmful content, bias, business-rule violations, RAG poisoning, and
        agent/tool abuse
      - >-
        Attack strategies including multi-turn (Crescendo), GOAT (Meta), and
        iterative jailbreak techniques
      - >-
        Framework alignment: OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, EU AI
        Act
      - >-
        50+ providers: OpenAI, Anthropic, Google Gemini, AWS Bedrock,
        SageMaker, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek, Together,
        Fireworks, Perplexity, OpenRouter, LiteLLM, Vercel AI Gateway,
        Cloudflare AI Gateway
      - >-
        Local runtimes: Ollama, LocalAI, llama.cpp, vLLM, Docker Model Runner,
        llamafile
      - >-
        Custom providers via HTTP, WebSocket, Python, JavaScript, Go, Ruby,
        and Shell
      - GitHub Action (`promptfoo/promptfoo-action`) for PR-level eval gating
      - Self-hostable via Helm chart and on-premise enterprise deployment
      - >-
        Hosted Enterprise tier: team sharing, continuous monitoring,
        centralized compliance dashboard, SSO, granular permissions, managed
        cloud, SLA-backed support
      - SOC 2 and ISO 27001 certified; trust center at trust.promptfoo.dev
      - >-
        Distributed on npm (`promptfoo`), PyPI (`promptfoo`), and Homebrew
        (`brew install promptfoo`)
      - MIT-licensed; 21k+ GitHub stars; now part of OpenAI
    sources:
      - https://www.promptfoo.dev
      - https://github.com/promptfoo/promptfoo
      - https://www.promptfoo.dev/docs/usage/command-line/
      - https://www.promptfoo.dev/docs/red-team/quickstart/
      - https://www.promptfoo.dev/docs/providers/
      - https://www.promptfoo.dev/pricing/
    updated: '2026-05-25'
maintainers:
  - FN: Kin Lane
    email: kin@apievangelist.com