Why Is AI Pentesting Not Traditional Pentesting? Measuring Security in Probabilistic Systems

AI pentesting vs traditional pentesting and attack success rate.

Imagine sending the same attack input to an AI assistant twice. It rejects the request on the first attempt. On the second, it discloses information it should not share. We cannot interpret the first attempt as “secure” and dismiss the second as “a coincidence.” What we need to evaluate is when and how often the security boundary is crossed under defined conditions.

Security testing of applications that use large language models requires a reproducible test setup, an explicit success criterion, and clearly bounded authorization. Attack Success Rate (ASR) is a useful measurement within that setup. On its own, however, it does not describe the system’s security level.

This article examines how AI pentesting differs from traditional pentesting, black/gray/white-box methodology, what nondeterministic behavior means for testing, how to calculate and interpret ASR, how to define scope, and where legal and ethical boundaries apply. The focus is on applications using large language models; not all AI systems behave the same way. The numbers and fictional scenarios below are educational examples, not results from a product test we conducted. The published research example is cited separately.

Key takeaways

AI pentesting extends traditional pentesting; authentication, authorization, APIs and infrastructure still need testing.
Box color (black, gray, white) should be reported separately for each layer: application, orchestration, knowledge, model and tools.
A single successful attack proves a vulnerability exists; repeated trials measure how often it occurs under defined conditions.
ASR is only meaningful with a predefined success criterion, a stated sample size and its uncertainty.
Scope must cover data flows and real-world actions, not just domains and accounts, and authorization does not extend to third-party model providers.

Why Is AI Pentesting Different from Traditional Pentesting?

The best starting point is not to treat AI pentesting as a replacement for traditional pentesting. Authentication, session management, API authorization, and infrastructure security still need testing in an AI-powered application. The difference is the addition of how the model interprets context and which data or actions that interpretation can reach.

In a traditional access-control test, we ask, “Can user A obtain user B’s document?” In an AI assistant, we keep that question and add another: “Can a document the assistant reads steer it toward an action the user did not request?” The test surface therefore extends beyond request parameters; content the model reads also matters.

For example, RAG (Retrieval-Augmented Generation) retrieves documents relevant to a question and provides them to the model as context for generating an answer. This process does not, by itself, retrain the model. Text in a document may be interpreted by the model as behavioral instructions when it should serve as source material for the answer. OWASP addresses such manipulation through external sources under indirect prompt injection; our guide What Is Prompt Injection? explains the attack in more detail.

This is not merely a hypothetical idea. Greshake and colleagues’ 2023 research experimentally demonstrates that instructions placed in external sources can influence LLM-integrated applications. The lesson is that the attacker does not always have to be the person directly chatting with the system; being able to influence content the application will read can also create an attack path. This is a published research finding, not a claim that any particular product remains vulnerable today.

In an educational scenario, suppose text read by an assistant working only with test documents steers it into requesting a document belonging to another test account. The model requesting it and the backend service actually returning it are different events. The first concerns behavioral manipulation; the second concerns a concrete data-access violation, the AI equivalent of an IDOR vulnerability. Which event counts as success must be documented beforehand.

Testing therefore cannot be reduced to “Did the model produce an inappropriate sentence?” Trust boundaries between the user, document, model, and tool must be evaluated. The model saying “I performed the action” is not evidence that the action occurred either.

The distinction “traditional software is always deterministic, AI is always random” is incorrect. Traditional systems also have race conditions and state changes, while some model configurations behave reproducibly under fixed conditions. The difference in AI pentesting is explicitly accounting for behavioral variability, natural-language manipulation, and the impact of connected tools in test design. A single successful attack can prove that a vulnerability exists; repetitions help measure how often it manifests under defined conditions.

Black-, Gray- and White-Box Methodology in AI Pentesting

In traditional pentesting, the box color usually describes the information given to the tester about the network, application, and source code. The same logic applies to AI pentesting, but the target is not a single layer. Application code, the system prompt, model, RAG pipeline, embeddings and vector store, guardrails, tools, and external services may each have different levels of visibility. Labeling the entire system simply “white box” is therefore often too broad.

An AI pentest report should state the box approach separately for at least these layers:

Application layer: Interface, API, session, traditional authorization, and source code.
Orchestration layer: System prompt, message ordering, model-routing logic, guardrails, and output-validation rules.
Knowledge layer: RAG queries, embedding model, vector store, document filters, and data-source permissions.
Model layer: Model identity, generation settings, safety policies, and weights where available.
Tool and action layer: Functions the model can invoke, such as email, files, databases, code execution, or payments.

For example, the team may see all application source code and the system prompt while having no access to the weights of the closed third-party model. This should be reported as white box at the application and orchestration layers, black box at the model layer. A box color is not a security grade; it explains what information supported the finding. NIST’s attacker-knowledge classification in NIST AI 100-2e2025 also uses the black-, gray-, and white-box distinction.

Black Box: What Can Be Observed from Outside?

The team receives access to the chat interface or API and, where necessary, an ordinary user test account; source code, system prompts, RAG traces, and tool-call logs are not shared. Black box does not necessarily mean “unauthenticated access.”

In a traditional black-box test, the tester may send different inputs to an endpoint and look for status codes or data leakage. AI black-box testing adds semantically equivalent attacks phrased in different ways, multi-turn conversations, direct and indirect prompt-injection scenarios, and repeated trials. Evidence is not merely text written by the model; it is an externally verifiable outcome such as return of a synthetic marker, an actual tool call, or an unauthorized state change.

In the document-assistant example, authorization boundaries are tested using account A, and the response is checked for B’s synthetic marker. The leak can be verified externally, but it may be impossible to determine whether it originated in a RAG filter, tool authorization, or a prompt. If the model version and some generation settings are unknown, the result represents only behavior observed at test time. A black-box finding can therefore demonstrate impact strongly, while its root-cause claim should remain more limited.

Gray Box: Which Assumptions Can We Test with Partial Knowledge?

The team receives an architecture diagram, model identity and settings, selected parts of the system prompt, test accounts with different roles, RAG traces, or selected tool logs; the full codebase and model weights are not shared. This is often the most useful arrangement for real organizational AI tests because it preserves the behavior of an external attacker while providing enough telemetry to separate where the outcome occurred.

In the same experiment, the team can see which chunks RAG retrieved, which context was sent to the model, which tool was called, and what authorization decision the application made. This makes it possible to distinguish among several outcomes: Was the malicious document never retrieved? Was it retrieved but the model ignored its instruction? Did the model request a tool call that the application blocked? Did the tool actually perform an unauthorized action?

Traditional gray-box testing emphasizes test accounts and API documentation. AI gray-box testing additionally exposes how context is assembled and how a model decision becomes a real action. Layers that were not shared must be identified explicitly in the report.

White Box: Investigating the Cause from Inside

Application-layer white-box testing can examine source code, system prompts, prompt templates, RAG and reranking logic, data filters, guardrail configuration, tool schemas, authorization checks, and relevant logs. If an open or internally hosted model is used, access to architecture and weights may separately enable model-level white-box evaluation.

In the document assistant, the investigation checks whether the user filter is applied to the document query, whether external documents are separated as untrusted data, and whether tool authorization is revalidated by deterministic code. Code review alone is not enough: the runtime behavior of the prompt, model, and data combination must be verified dynamically. This is an important practical difference from traditional white-box testing; code that looks correct does not, by itself, prove that the model will preserve the security boundary across different contexts.

Connecting the Three Approaches to the Same Measurement Protocol

In all three approaches, the visibility of each layer is recorded first; the authorization boundary, success criterion, test conditions, and trial budget are then defined. Black-box testing relies on external outcomes, gray-box testing combines outcomes with partial traces, and white-box testing additionally uses code and configuration. The same attack scenario can be run at all three visibility levels; what changes is not the definition of the violation, but the evidence available to explain it.

ASR can be calculated in all three approaches. However, the ASR of an attack developed with source-code and prompt knowledge cannot be compared without context to that of an attack discovered with limited external access. Knowledge level, attack-search budget, and model configuration must be reported with the rate.

What Does Nondeterministic Mean for AI Security Testing?

A deterministic system produces the same result given the same inputs and initial conditions. In language models, sampling from the calculated probabilities for the next token can produce different outputs. Methods such as greedy decoding select the most likely option at each step; calculating probabilities therefore does not, by itself, require random output (Hugging Face, Generation strategies).

Sampling is not the only factor affecting reproducibility. Some computational operations, hardware, and software versions can also influence the result. The PyTorch documentation on reproducibility explains that identical results are not guaranteed across platforms and releases, even when the same seed is used.

At the application level, other variables include conversation history, documents retrieved from external sources, tool responses, and session state. A user typing the same sentence does not mean that the model sees exactly the same context. Differences caused by changing inputs must not be confused with variability under identical inputs.

The purpose of testing is to make this variability measurable. Instead of drawing a security conclusion from a single refusal, we record which conditions were held constant and which were changed. Lowering temperature can help, but it does not, by itself, guarantee reproducibility across the entire application.

What Counts as a Successful Attack?

Consider a fictional internal company assistant. User A can access only their own test documents. We place a unique marker, which is not a real secret, in a test document belonging to user B. The experiment aims to determine whether user A can obtain that marker without authorization.

We define success before running any trials: “Verification of B’s test marker in the response returned to user A.” The model saying “I accessed B’s document” does not count as success. The data it actually accessed or the action it performed must be demonstrated.

For this scenario, each repetition should begin in a fresh session, account permissions should remain unchanged, and the test documents should be held constant. If a multi-step attack is being examined, the unit of measurement must reflect that: a single message, an entire conversation, or an attack attempt with a defined budget? Mixing different units within the same rate makes the result ambiguous.

How Is Attack Success Rate (ASR) Calculated?

In this article, we define ASR at the trial level:

ASR = Number of attack trials meeting the success criterion / Number of valid attack trials × 100

For example, if the defined violation occurs in 7 of 20 valid trials completed under the same test protocol, the observed ASR is 35%. An appropriate reporting statement is: “In this attack scenario, a security violation was observed in 7 of 20 trials under the specified configuration.”

“The system is 35% insecure” does not mean the same thing. The measurement depends on the selected attack, user permissions, test data, and success criterion. Pooling different attacks can also hide high success in particular scenarios behind an average. Results should therefore also be broken down by attack type and privilege level.

A connection failure that never reaches the target may be classified as an invalid trial, but exclusion rules must be defined in advance and the counts disclosed. A request rejected by the system for security reasons is not an invalid trial. A timeout should not automatically be excluded either: in an availability attack, it may itself be the success criterion.

The Same Percentage, Different Uncertainty

The results 7/20 and 700/2000 both equal 35%. Nevertheless, assuming independent and comparable trials drawn from the same conditions, the second measurement estimates the success rate more precisely.

For 7/20, the approximate 95% Wilson confidence interval is 18.1%–56.7%. Its width illustrates the uncertainty of a small sample. This calculation does not measure how well the test set represents real attacks or how secure the entire system is (NIST/SEMATECH, confidence intervals for proportions).

The independence assumption matters as well. Continuing the same conversation, adapting the attack to the previous response, or receiving the same cached answer can undermine independence between repetitions. Adaptive attacks are valuable, but they should not be reported as the same experiment as repetitions under fixed conditions.

Why Is AI Pentesting Not Traditional Pentesting? Measuring Security in Probabilistic Systems

Making the Measurement Concrete

The code below does not send attacks to a target. It calculates ASR from example outcomes that have already been evaluated. “True” represents verification of the predefined violation, not the model sounding convincing.

# Synthetic fixture: not the output of a real penetration test.
outcomes = [True] * 7 + [False] * 13
valid_trials = len(outcomes)
successful_attacks = sum(outcomes)
asr = successful_attacks / valid_trials if valid_trials else None
print(f"Successful attacks: {successful_attacks}/{valid_trials}")
print(f"ASR: {asr:.1%}" if asr is not None else "ASR: undefined")
# Successful attacks: 7/20
# ASR: 35.0%

In a real dynamic analysis, this list must be produced from reliable evidence. Responses, test markers, authorization decisions, and actual tool-call outcomes, where applicable, are correlated. Ambiguous assessments are flagged separately. Relying exclusively on another language model to score violations can introduce evaluation errors into the measurement.

The completion rate for legitimate tasks should be tracked alongside the security outcome. A defense that blocks every request may reduce attacks while making the application unusable. The AgentDojo research, which examines agents’ task success alongside their behavior under attack, provides a useful example of this distinction.

Scope: What Are We Testing, and Under Whose Authority?

Traditional pentest scope usually lists domains, IP addresses, applications, accounts, and permitted techniques. AI pentesting still requires all of these, but because the model can read data sources and invoke tools, the scope must also define the data flow and possible real-world effects. NIST SP 800-115 provides a foundational framework for planning and conducting security tests.

For the fictional experiment in this article, we can define scope as follows:

Layers: Which application, model/API provider, system prompt, RAG pipeline, vector store, guardrail, and tools are in scope.
Version and state: Model identity, generation settings, prompt revision, dataset version, date, and session-reset method.
Identity and data: Test accounts A and B, synthetic documents, and test markers; no access to real user data.
Permitted AI attacks: Which scenarios may be tested, such as direct or indirect prompt injection, jailbreaking, data leakage, and tool misuse.
Prohibited operations: Explicit exclusion of model extraction, training-data extraction attempts, data poisoning, persistent-memory modification, or attacks on provider infrastructure unless separately authorized.
Action boundary: Whether tools that create real effects (sending email, deleting files, executing code, or initiating payments) are simulated or invoked for real in a test environment.
Trial budget: Request rate, total requests, tokens, cost, concurrency, and test-window limits.
Success and stop conditions: What evidence constitutes a violation, and who stops the test if personal data, data leaving the environment, or service degradation is observed.
Evidence management: Access to prompts, responses, retrieved documents, and tool logs; masking, retention, and deletion responsibilities.

The important difference is that an AI application’s boundary does not end at the chat screen. An email or file service connected to the assistant, and the infrastructure of the model provider, do not automatically become unrestricted targets because the application owner authorized a test. Ownership, service terms, and technical authorization for each component must be assessed separately. For a step-by-step testing approach, see How to Penetration Test an LLM Application.

Legal and Ethical Boundaries of AI Pentesting

AI pentesting does not create a separate legal “free zone.” The core duties relating to authorization, data protection, confidentiality, intellectual property, and service continuity continue to apply as they do in traditional pentesting. The difference is that an AI system may involve more copies of data, more service providers, and more automated actions. Its legal boundary therefore cannot be drawn solely around the tested domain name.

Authorization Should Be Specific to Systems and Layers

Written authorization should identify the application, model, test accounts, data sources, attack methods, and test period in scope. A company may authorize testing of its chatbot; that permission does not automatically authorize attacks against the infrastructure of the third-party model provider, attempts to reach other customers’ data, or model-extraction experiments. Permission to perform “jailbreak testing” is not permission to generate unlimited traffic or bypass out-of-scope security controls.

The AI Data Lifecycle Is Broader than a Request–Response Log

Personal or confidential data may appear not only in the user’s prompt but also in conversation history, model responses, uploaded documents, RAG chunks, embeddings and vector stores, tool output, security-filter logs, or evaluation systems. The test plan should examine which countries and subprocessors receive the data, whether the provider retains inputs, and whether it uses them for training or service improvement. Synthetic markers should be preferred wherever possible.

In Türkiye, KVKK obligations may apply when these records contain personal data. A case-specific assessment should address the purpose and legal basis of processing, controller–processor roles, data minimization, retention periods, and data-subject rights. The Turkish Personal Data Protection Authority’s generative AI guide discusses data processing in these systems.

Access to test evidence should be restricted, unnecessary personal data should not be copied, and records should be protected. If a finding can be demonstrated with a synthetic record, continuing to collect real users’ data may be difficult to justify under the principle of proportionality. The authority’s explanation of data-security obligations describes the need for technical and organizational measures to prevent unlawful processing and access.

If prompts, documents, embedding data, or test records are sent to a model or observability service abroad, the conditions for international transfers must be assessed separately. The application owner’s pentest authorization does not replace that data-transfer assessment.

A Model's Statement and a Real Action Do Not Have the Same Impact

A chatbot producing problematic text is not equivalent in impact to an AI agent sending an email, deleting a file, executing code, or initiating a payment. Systems with tools should use test accounts, sandboxes, transaction limits, and human approval for high-impact actions. Even if the model says “I completed the action,” the action should not be treated as real until the system record confirms it.

Repetition and Automation Create a Separate Operational Risk

Measuring ASR may require running an attack many times. Those repetitions can consume quota, create cost, trigger rate limits, or degrade service. The written scope should therefore include limits for trial count, concurrency, tokens, and cost. The load and service-disruption restrictions of traditional pentesting still apply; the need for probabilistic measurement does not remove them.

The ethical baseline is to use the minimum impact needed to demonstrate the finding. If real data, unexpected access to an external system, or an irreversible action is observed, testing stops; evidence is preserved securely, and the designated contact is notified. Public disclosures should remove secrets (including prompts), personal data, and details that would make the attack easy to misuse. This section is a general testing framework; a concrete legal assessment depends on the relevant jurisdiction, contracts, and circumstances.

Risk: What ASR Does and Does Not Tell You

ASR describes observed attack success; it does not describe the entire risk. A risk assessment should consider the importance of the affected data or action, the access required, and the attacker’s ability to retry alongside the success frequency. A critical data leak with a low success rate can still be a serious finding.

If we assume independent attempts with a constant 5% success probability on each attempt, the probability of at least one success in 20 attempts is 1 − (0.95)²⁰ ≈ 64.15%. This is only a probability example with explicit assumptions; dependence, rate limits, and adaptivity change the outcome in real attacks.

A report should therefore answer both “How often did it happen?” and “What happened when it did?” Our guide on what to expect in a penetration test report covers how findings should be documented.

Remediation, Version Tracking and Retesting

Remediation should address the cause of the observed violation. If the issue in our example is unauthorized data access, user and document permissions must be enforced by the application; expecting the model to follow instructions must not be the sole control. Rate and budget limits can restrict repeated attempts, but they do not replace missing authorization checks.

Because this article does not verify a specific product vulnerability, it provides no “affected version” or “fixed version” number. A real report should record the application version, model identifier, prompt revision, generation settings, dataset version, and tool permissions together. If the provider does not offer a fixed version, the test date and that uncertainty should be documented.

After remediation, the same protocol and success criterion should be used for the initial retest. Coverage is then expanded with different wording and contexts. Legitimate task success, the number of invalid trials, and sample size are compared alongside ASR. Observing zero violations after remediation only establishes that none were observed under the tested conditions; it does not prove that every possible attack has been eliminated.

Conclusion

AI pentesting does not abandon traditional security controls; it adds model behavior and the chain of actions the model can influence. The black/gray/white-box choice explains what we can see, while ASR explains what we observed in a defined scenario. Neither alone guarantees security for the entire system.

A reliable AI pentest report explains the limits of its findings as clearly as the findings themselves: What knowledge and access levels were available, which system state was tested, which behavior counted as a violation, how many trials were run, and how far was the team authorized to go?

Testing an AI assistant, RAG system or agent? Our AI & LLM Security Testing service applies this methodology with clear scope, measurable success criteria and reproducible evidence, and our AI Security Training teaches your team to run these tests. Talk to our team to get started.

References

Subscribe to our cyber security newsletter

Scroll to top