AI Security Tools Compared: Which Layer promptfoo, garak, PyRIT, MCP Scanners and UltraProbe Each Test
Updated 2026-10-12. Capabilities, licences and languages are taken from each project's own GitHub README and website.
These tools do not replace each other; each tests a different layer. promptfoo, garak and PyRIT are dynamic red-teaming tools: they send attacks to a running model or AI app and see how it actually responds. MCP scanners such as Snyk Agent Scan, Cisco MCP Scanner and mcpscan.ai look at the tool servers an AI is connected to. NeMo Guardrails and Lakera Guard are runtime guardrails after launch. UltraProbe covers one layer only: before deployment, it reads the system prompt and tool descriptions you wrote and checks whether 25 defenses are present, without calling a model or sending attacks.
So the useful question is not which tool is best, but which layer is still untested. The table shows the split; the sections below cover when each kind fits, and what UltraProbe does and does not do.
Who tests what
| Tool | Layer | What it actually tests | Needs a model or external API | Licence | Language |
|---|---|---|---|---|---|
| UltraProbe | Static audit, pre-deploy | The text of system prompts and tool descriptions, checked for 25 missing defenses | No; runs offline | MIT | TypeScript (Node CLI + SDK) |
| promptfoo | Evals and dynamic red teaming | Generates attacks, sends them to your model, agent or RAG app, and grades the responses | Yes: a target; red teaming also uses a model to generate and grade attacks | MIT | TypeScript |
| garak | Dynamic LLM vulnerability scanner | Runs probes (injection, jailbreaks, data leakage, toxicity and more) against a model; detectors flag failures | Yes: a target model | Apache-2.0 | Python |
| PyRIT | Red-teaming framework | Lets red teamers script attack strategies, including multi-turn attacks, with converters and scorers | Yes: a target, often a scoring model too | MIT | Python |
| Snyk Agent Scan | Agent, MCP server and skill scanner | Discovers installed MCP servers and skills, retrieves tool descriptions, checks for prompt injection, tool poisoning, tool shadowing and toxic flows | Yes: a Snyk token; component data including tool descriptions goes to Snyk's analysis API | Apache-2.0 | Python |
| Cisco MCP Scanner | MCP server scanner | Scans MCP tools, prompts and resources with YARA rules, LLM analysis or the Cisco AI Defense API; can also scan server source code | YARA mode runs locally; LLM and API modes need keys | Apache-2.0 | Python |
| knostic MCP-Scanner | MCP exposure discovery | Uses Shodan to find MCP servers exposed on the internet, then verifies them and enumerates their tools | Yes: a Shodan API key | MIT | Python |
| mcpscan.ai | Hosted MCP source-code scan | Takes the GitHub URL of an MCP server and checks its source code for vulnerabilities such as command injection | Hosted service that fetches the repo | No open-source licence stated on the site | Web service |
| NeMo Guardrails | Runtime guardrails | Applies programmable input and output rails to live conversations | Runs inside your app; many rails call a model | Apache-2.0 | Python |
| Lakera Guard | Runtime guardrails (commercial) | Screens inputs in real time through an API for prompt injection and other attacks | Yes: the Lakera API | Commercial | API service |
Dynamic red teaming: promptfoo, garak, PyRIT
All three need a target that answers: a staging copy of your model, chatbot or agent. They send attacks, collect responses and judge them, so they answer the most practical question: does this system hold up right now. The cost is that you need a running environment, each run spends tokens, and model output is random enough that the same test can give different results on a rerun.
promptfoo is an eval and red-teaming CLI and library; one config can drive functional regression tests and security tests, and it plugs into CI/CD. Its README states that the project is now part of OpenAI and remains MIT licensed. It suits teams that already manage test cases in config files and want security tests in the same pipeline.
garak is NVIDIA's LLM vulnerability scanner; its README compares it to nmap or Metasploit for LLMs. Point it at a model and by default it runs every probe it knows, which suits broad scans with little setup.
PyRIT is Microsoft's red-teaming framework rather than a one-click scanner: attack strategies, multi-turn attacks, input converters and scoring are composed in Python. It suits teams with dedicated red teamers testing complex scenarios.
MCP server scanning: Snyk Agent Scan, Cisco MCP Scanner, knostic, mcpscan.ai
Once an assistant is connected to files, databases or Git through MCP (Model Context Protocol), there is another layer of risk: tool descriptions can be poisoned, parameters can be injected, and tools can hold more permission than they need. These tools look at the servers and tools themselves, not at your prompt.
Snyk Agent Scan (formerly Invariant Labs' mcp-scan) discovers the MCP servers and agent skills installed on your machine, retrieves tool descriptions and checks them for prompt injection, tool poisoning, tool shadowing and toxic flows. Two things to know: its README warns that scanning can start servers and execute the commands in your config, and recommends a sandbox; and analysis sends component data, including tool descriptions, to Snyk's API.
Cisco MCP Scanner has three engines that can run alone or together: local YARA rules, LLM analysis and the Cisco AI Defense API. It can also scan pre-exported JSON offline, or analyse the behaviour of an MCP server's source code. For CI or air-gapped networks, YARA plus offline mode needs no external key.
knostic MCP-Scanner and mcpscan.ai are often mentioned together but do different jobs. knostic uses Shodan to find MCP servers exposed on the internet, which is useful for exposure research, not for auditing your own server. mcpscan.ai is a web service: give it the GitHub URL of an MCP server and it checks the source code for vulnerabilities such as command injection, where a parameter flows straight into a shell command.
Runtime guardrails: NeMo Guardrails, Lakera Guard
The first two kinds test before launch; this kind blocks after launch. NeMo Guardrails is NVIDIA's open-source toolkit for programmable input and output rails in a conversation flow. Lakera Guard is a commercial API that screens content for prompt injection and other attacks before it reaches the model. Both add latency and cost to each request, in exchange for a checkpoint that still works in production.
Where UltraProbe fits: what it does and does not do
UltraProbe is a pre-deploy static audit of prompt defenses. It reads the text of a system prompt or tool description and checks whether 25 defenses are present: 12 mapped to the OWASP LLM Top 10 and 13 mapped to ASI01 to ASI09 of the OWASP Agentic Top 10.
- No model calls, no API key: the same text always gives the same result in milliseconds, so it fits in pre-commit and CI, with SARIF output for GitHub Code Scanning.
- Reads Chinese: the rules match defensive wording in both English and Traditional Chinese, so a prompt written in Chinese is not scored as undefended just because the rules only know English.
- Tool descriptions count: an AI decides whether to call a tool by reading its description, so MCP tool descriptions are part of the prompt and can be checked with the same rules.
- Free and open source: the web version at /en/probe needs no sign-up; the CLI is the npm package ultraprobe, source on GitHub, MIT licensed.
What it does not do: it sends no attacks, so it cannot prove a model actually resists them (that is what red teaming is for); it does not connect to, run or read the code of an MCP server, and cannot detect a tool swapped at runtime (that is what MCP scanners are for); pattern matching can miss unusually phrased defenses or credit a sentence that merely mentions a keyword; and ASI10 (rogue agents) is runtime governance, outside its scope.
What each of the 25 defenses checks, and which OWASP entry it maps to, is listed in the vector table in the GitHub README.
A combination that covers the layers
- While writing prompts and tool descriptions: UltraProbe in pre-commit or CI, so a missing defense blocks the change.
- Once there is a staging environment: red-team it with promptfoo, garak or PyRIT to confirm it actually holds.
- For every MCP server you connect: scan it in a sandbox with Snyk Agent Scan or Cisco MCP Scanner, and review the permissions it asks for.
- After launch: runtime guardrails plus least privilege, with human approval for high-risk actions such as payments, deleting files or sending email.
For how the attack gets in, read What is prompt injection. For MCP, see the section on what MCP servers can get wrong in We audited 7 official MCP servers.
The website AI-visibility half
UltraProbe's other half is a website check that combines SEO, AEO and AAO into an AI Visibility Score (AVS). That is not the same thing as Semrush's AI Visibility or Ahrefs' Brand Radar, which track how often a brand is mentioned in AI answers; UltraProbe checks whether your pages themselves can be read. The difference is explained in brand-mention tracking vs website technical scanning, and a comparison with another free scanner is in UltraProbe vs TWTools.
Frequently asked questions
- What open-source AI security scanners are there?
- By layer: for dynamic red teaming, promptfoo (MIT), garak (Apache-2.0) and PyRIT (MIT); for MCP server and agent component scanning, Snyk Agent Scan and Cisco MCP Scanner (both Apache-2.0); for runtime guardrails, NeMo Guardrails (Apache-2.0); for a pre-deploy static audit of prompt defenses, UltraProbe (MIT). They test different layers and are usually combined.
- How do I test whether my chatbot is vulnerable to prompt injection?
- Two steps. First, run a static audit such as UltraProbe to check whether the system prompt contains defenses; it takes seconds and spends no tokens. Second, use promptfoo, garak or PyRIT to send real injection attacks to a staging copy of the chatbot and see how it responds. The first tells you what is missing; the second tells you whether it actually holds.
- What security tests should an LLM app pass before launch?
- At least four: a defense audit of prompts and tool descriptions; red teaming against staging, covering injection, jailbreaks and data leakage; a scan of every MCP server and plugin; and a permission review of what each tool can read or write and which actions need human approval. The OWASP LLM Top 10 and OWASP Agentic Top 10 work as checklists.
- What are the security risks of an MCP server, and how do I audit one?
- Three main kinds: poisoned tool descriptions (instructions to the AI hidden in a description), parameter injection (user input flowing straight into a shell command or query), and over-permission (a tool can reach more than it needs). Scan tool descriptions and config with Snyk Agent Scan or Cisco MCP Scanner, and the server's source code with mcpscan.ai or a code scanner. UltraProbe only checks whether tool descriptions contain defensive language; it does not scan the server.
- Can UltraProbe replace promptfoo or garak?
- No, they test different layers. UltraProbe sends no attacks and only checks the written defenses, so it suits running on every prompt change; promptfoo and garak send attacks and observe the model, so they suit regular runs against staging. Use both.
- Is there an AI security scanner that understands Chinese prompts?
- UltraProbe is built by Ultra Lab in Taiwan; its defense rules match wording in both English and Traditional Chinese, and the web version has a Traditional Chinese interface, free with no sign-up. It is a static audit, so testing how the model actually behaves still needs a red-teaming tool.
Scan your system prompt free · Source on GitHub · Trust Center