Know Your Target: LLM Reconnaissance and Model Fingerprinting

LLM Application Reconnaissance & Model Fingerprinting

Part 3 of the AI Pentesting series. Previous: Local AI Red Teaming: Building an Isolated LLM Security Lab

AI-powered applications are no longer limited to simple Q&A chatbots. LLMs are increasingly integrated into customer service, code generation, document analysis and enterprise knowledge systems. Adding an LLM to an application introduces entirely new components, and consequently new security assessment areas, that do not exist in traditional web applications.

This article explains the first phase of an LLM security assessment: reconnaissance. We look at how to detect the model and provider behind an application, how model, tokenizer and behavioral fingerprinting work, what rate limits and error messages reveal, and how these signals come together into an attack surface map, with three open-source proof-of-concept tools.

Key takeaways

LLM reconnaissance aims to understand the target’s structure and behavior, not to attack it.
No single signal identifies a model: a chatbot’s answer to “What model are you?” proves nothing on its own.
Model, tokenizer and behavioral fingerprinting each provide supporting evidence; combined, they build a reliable profile.
Gateways and proxies add noise to headers and error messages, so infrastructure signals must be interpreted carefully.
The end goal is an attack surface map that shows which components to test next.

Why Is Testing LLM Applications Different?

When evaluating a traditional web application, we typically look at components such as the HTTP layer, APIs, authentication mechanisms, server infrastructure and databases. In LLM-based applications, the scope expands to include the model itself, the model provider, the instructions sent to the model, model outputs, RAG systems and external tools the model can access. The fact that the OWASP Top 10 for LLM Applications 2025 lists LLM-specific risks such as prompt injection, sensitive information disclosure, system prompt leakage, vector and embedding weaknesses and unbounded consumption is a clear indicator of this difference.

Therefore, one of the first stages of LLM security testing is to understand the underlying system from its externally observable behavior. This includes LLM reconnaissance, model and provider detection, model fingerprinting, tokenizer behavior, rate-limit analysis, error message analysis and LLM attack surface mapping.

The key question at the outset of an LLM security assessment becomes: “What kind of system is powering the LLM application in front of us?” The phase in which we answer it can be defined as LLM reconnaissance.

What Is LLM Reconnaissance?

In cybersecurity, reconnaissance is the process of gathering information about a target before a security assessment. In a traditional application, a researcher might examine the technologies in use, API endpoints, authentication mechanisms, server behavior and open services.

In an LLM application, new questions are added:

Which model or model family is being used?
Which model provider is hosting it?
How does the application communicate with the model?
What are the externally observable behaviors of the model?
How is rate limiting applied?
How informative are the errors?
Could there be a RAG system or other tools behind the application?
When these components come together, what kind of attack surface emerges?

The important point is that reconnaissance does not mean carrying out a direct attack. The primary goal is to understand the structure and behavior of the target, within an authorized scope.

What makes reconnaissance in LLM applications interesting is that a researcher can observe not only the web application but also the behavior of the model it uses. To answer these questions, the application’s API behavior, model outputs, tokenization behavior, rate-limit mechanism and error messages can be examined together.

The Visible Face of an LLM Application and the Backend System

When users open an AI application, they typically see only a chat box:

User → Chatbot → Response

However, a more complex structure may lie behind this interface:

User → Web Application → Backend / API → LLM Layer → Model Provider → LLM

Some applications also include other components alongside the model, such as a RAG system, external APIs and tools, or a vector database. A “chatbot” is therefore not a single component; behind it can lie an ecosystem made up of the application, API, model, data sources and external tools.

While this system cannot be observed directly from the outside, signals such as API behavior, model outputs, error messages and tokenization make it possible to draw inferences about it.

Model Provider Detection: What Model Lies Beneath?

One of the most striking parts of reconnaissance is determining the model provider. An application may use the API of a provider such as OpenAI, Anthropic or Google, or it may run an open-weights model such as Llama or Mistral on its own infrastructure.

The goal is not merely to “find the name of the model.” First and foremost, we ask: “What information can I obtain from the outside about the LLM infrastructure behind this application?”

Several observable signals can be evaluated:

API behavior: the format of the application’s API responses, error codes and other technical characteristics.
Error messages: when an unexpected request is sent, the error may contain technical hints about the backend or service used.
HTTP headers and response information: HTTP-level behavior can in some cases reveal the infrastructure in use.
Model behavior: the model can be asked about itself, or its behavior against different inputs can be observed.

Critical note: gateway noise. Most modern LLM applications do not connect directly to the model provider’s API. Requests first pass through a front layer such as Cloudflare, AWS API Gateway, Nginx or enterprise API management solutions such as Apigee. Captured HTTP headers, status codes or error messages may therefore reflect the front layer rather than the underlying model or backend. Header or error analysis alone can be misleading.

There is also a critical research principle here: a single signal does not mean definitive model identification. When asked “What model are you?”, a chatbot might state a model name, but that answer does not prove the application really uses that model’s API. The model might answer incorrectly, the application may have given it a different identity, or a routing layer might sit in front of it. Provider and model identification should therefore be treated as a fingerprinting exercise that combines multiple signals.

Model Fingerprinting: Deriving Identity from Behavior

Fingerprinting is the approach of using a system’s distinctive characteristics to obtain identifying information about it. With LLMs, this becomes more interesting, because a model is not just an API; it also exhibits specific behavior patterns in response to specific inputs.

A researcher can send different inputs in a controlled way and compare:

response format,
behavior when refusing specific requests,
output format,
token usage,
error behavior.

Evaluated together, these behaviors form a behavioral profile of the model. The goal is not to say “This response came back, so it is definitely model X,” but rather “The behavior we observed is consistent with model or provider X; other signals are needed to verify it.”

This distinction may seem small, but it matters in technical research: fingerprinting is an inference process and does not provide direct access to source code or configuration.

Why Is Tokenizer Behavior Interesting?

To understand this signal, we need to look at how an LLM processes text. When we type “cybersecurity” into an LLM, the model does not process the word as we see it. The text is first converted into tokens by a tokenizer, the component that turns text into the model’s input format. Depending on the model and tokenizer, words or parts of words can be split in different ways.

Text → Tokenizer → Tokens → Model → Output

The same word can be split into a different number of pieces by different tokenizers, so tokenization behavior can serve as a distinguishing signal about the model in certain situations. The same principle still applies, however: token count alone does not prove a model’s identity. Tokenizer behavior must be evaluated together with other fingerprinting indicators.

Rate-Limit Analysis: How Much Traffic Does the System Accept?

Another reconnaissance signal is rate-limit behavior. Rate limiting restricts the number of requests a client or user can send within a given time interval:

Request 1 → successful
Request 2 → successful
…
Limit → exceeded
Request → blocked

This behavior is not only a security control; it also reveals how the application works. We can examine under which conditions the limit is applied, how the application responds, whether limit information appears in response headers and whether behavior changes across clients.

Rate limiting is also more than a reconnaissance topic. Uncontrolled usage that leads to excessive resource consumption is listed as Unbounded Consumption (LLM10) in the 2025 OWASP list, because it creates cost and availability problems.

Signals examined during rate-limit analysis:

Signal	Question
Threshold	After how many requests is the limit applied?
Time window	Over what time interval is the limit calculated?
Response	What is returned when the limit is exceeded?
Headers	Are there rate-limit headers?
Scope	Is the limit based on IP, user, session or API key?
Recovery	When does the limit reset?

The purpose of this analysis is not to bypass the rate-limit mechanism but to understand how it works and at which layer it is applied. The observed behavior can provide an additional signal about the gateway, API or provider tier behind the application.

Error Message Analysis: What Do Errors Tell Us?

How a system behaves when it encounters errors can be as informative as its normal responses. An application might return a general error such as “Invalid request,” or a more technical one; the second case can reveal more about the backend, validation mechanism or other components.

Signals that can be examined in error behavior:

Signal	What it may reveal
HTTP status code	The layer where the error occurred
Error format	Backend or API structure
Validation messages	The input validation mechanism
Provider-specific errors	The service in use
Request ID	Backend or gateway traces
Stack trace	Internal components
Model or API information	Whether the model or service name is exposed

The goal is not to aggressively force errors or extract sensitive data, but to determine through controlled, authorized tests which technical information is exposed. Information about providers, API gateways, models or backend components in error responses can then be evaluated together with other reconnaissance findings.

System Architecture Inference

So far we have examined individual signals: provider, tokenizer, rate limit and error messages. Their real value emerges not from each one alone but from evaluating them together. External reconnaissance cannot reveal the true architecture with absolute certainty; we infer the most likely architecture from our observations.

This is one of the most interesting aspects of the research: even if the system never describes its architecture, we can form a picture of it by putting together pieces of its behavior.

LLM Attack Surface Mapping

The components identified through architecture inference show not only how the system might work but also which parts need security testing. A simple way to turn the reconnaissance profile into a test plan is to map each component to the risks it introduces:

Component	What to examine next	Related OWASP LLM risk
User input and chat interface	Direct and indirect instruction manipulation	LLM01 Prompt Injection
System prompt	Whether instructions or secrets can be extracted	LLM07 System Prompt Leakage
RAG system and vector database	Access control between users and data poisoning	LLM08 Vector and Embedding Weaknesses, LLM04 Data and Model Poisoning
Tools and external APIs	Which actions the model can trigger	LLM06 Excessive Agency
Output rendering	How model output is displayed or passed on	LLM05 Improper Output Handling
Rate limits and quotas	Cost and availability abuse	LLM10 Unbounded Consumption
Model and provider	Third-party models and components	LLM03 Supply Chain

The Role of PoCs in Reconnaissance

Instead of leaving a reconnaissance signal as a mere observation, testing it with a controlled proof of concept (PoC) helps support or refute the hypothesis. The three open-source projects below each approach fingerprinting from a different angle.

PoC 1: Black-Box Model Fingerprinting with LLM-Fingerprinter

The open-source project LLM-Fingerprinter aims to predict which model family an LLM belongs to, such as GPT, LLaMA or Mistral, when the model can only be queried from the outside. This is black-box testing: you do not look at the model’s weights or backend configuration, only at its input → output behavior.

We do not know the model
↓
Ask specific questions
↓
Record responses
↓
Examine response features
↓
Compare with known models
↓
Find the likely model family

What exactly does the PoC do? LLM-Fingerprinter sends 31 different prompts to the target model, divided into three groups:

Discriminative (11 prompts): model identity, knowledge cutoff, architecture and reasoning behavior.
Behavioral (7 prompts): safety boundaries, response to jailbreak attempts, honesty and policy behavior.
Stylistic (13 prompts): response format, creativity, adherence to constraints and default writing style.

It does not rely on a single “What model are you?” question. The responses are collected, their features are extracted and the resulting fingerprint is compared with the behavior of known models to predict the likely model family. This supports the earlier principle: even if the model says it is GPT, we do not accept that as proof.

Where is the fingerprint formed? The tool does not simply take plain text from responses. It extracts sentence embeddings, linguistic features and behavioral features, then feeds them into an ensemble of classifiers (Random Forest, SVM and MLP):

Prompt
↓
LLM
↓
Response
↓
Extract response features
↓
Numerical fingerprint
↓
Compare with known fingerprints
↓
Model family prediction

That makes it far more systematic than “I looked at the writing style and I think it’s GPT.”

Fingerprint ≠ definitive model identity. Model behavior can be affected by system prompts, sampling settings, fine-tuning or deployment changes. Recent research on endpoint fingerprinting also notes that changes in tokenizer, deployment, quantization and system prompts can alter the observed output distribution.

PoC 2: Tokenizer Fingerprinting

The open-source project tokenizer-fingerprint performs black-box tokenizer fingerprinting: without accessing the model’s weights or internal structure, it uses only the prompt_tokens value returned by the API.

Since different tokenizers can split the same text into different numbers and forms of tokens, observing this behavior creates a useful signal about a black-box system. The goal is to find which tokenizer family the target is most consistent with.

How the PoC works. The project compares the token count of a base prompt with the same prompt plus a controlled probe:

Base prompt:            Hello world
Probe-appended prompt:  Hello world 😊

(Base + Probe) token count − Base token count = Token cost of the probe

This reduces the effect of fixed token costs added by the chat template or API and gives a more direct view of how the probe is tokenized. To build a reliable fingerprint, many probes are used, including different languages, code samples, Unicode characters and character sequences, and the results are compared with known candidate tokenizers:

Base prompt → Add probe → Measure token count → Calculate difference → Compare with candidate tokenizers → Tokenizer fingerprint

The main advantage is that the method works under black-box conditions: as long as the API exposes token counts, inferences can be made without access to weights or tokenizer configuration.

The result, however, does not directly reveal the model. The same tokenizer can be used by several models or model families, and tokenizer, chat template or deployment changes by the provider can affect the observed behavior. The accurate conclusion is: “The observed tokenization behavior is consistent with tokenizer family X and provides a supporting signal about the model or provider.” Academic work on the topic likewise shows that token-count behavior can fingerprint a tokenization stack but may not be enough on its own to determine the model family.

PoC 3: Behavioral Fingerprinting from Single-Token Outputs

Fingerprinting does not always require analyzing long, complex responses. The distribution of answers to short, controlled queries can also form a distinctive behavioral signal.

This approach was examined in the 2026 study One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions, with an open-source implementation in llm-fingerprinting. The idea is to ask the model very simple questions, collect only single-token answers and use the distribution of those answers as the model’s behavioral fingerprint, across different languages and simple tasks.

How the PoC works. A simple question is sent repeatedly:

“Say a random number between 1 and 100.”

A single answer such as 42 says little. Repeated many times, a distribution appears:

42 → 19 times
73 → 6 times
17 → 4 times
51 → 3 times
…

Repeating this across probes and languages builds the model’s behavioral fingerprint:

Simple probe → Repeat many times → Collect responses → Build distribution → Compare with known fingerprints → Inference about the model

The study compares distributions using Jensen–Shannon Divergence (JSD), a metric that measures how similar two probability distributions are; lower values mean more similar behavior.

Results and limits. In evaluations of 165 models, the study reported that model families’ behavioral distributions are distinguishable, with an Equal Error Rate (EER) of 7.3% for model verification on a 40-cell probe set. These results come from the study’s own test environment and are not guaranteed on different APIs or configurations. The llm-fingerprinting project applies the approach in practice, matching an unknown endpoint against a reference library built from more than 176 models and 15 probe tasks in 4 languages.

Behavioral similarity is not absolute proof of identity. The same model can behave differently in different deployments, different models can produce similar distributions, and sample size, temperature, probes and the API provider all affect the result. The correct interpretation is: “The observed response distribution is consistent with model family X.”

This PoC is most useful alongside the other two: model fingerprinting examines general response features, tokenizer fingerprinting examines how input is split and behavioral fingerprinting examines statistical behavior under repeated simple queries. Combining them is far more robust than relying on any single observation.

Bringing the Reconnaissance Findings Together

Provider Detection
 │
 ├──── Model Fingerprinting
 │
 ├──── Tokenizer Behavior
 │
 ├──── Rate Limit
 │
 └──── Error Messages
 │
 ▼
Architecture Inference
 │
 ▼
Attack Surface Mapping

The purpose of these methods is not to reach a definitive conclusion about the model or architecture from a single signal, but to build a stronger reconnaissance profile from the combination of different observations.

Conclusion

Findings: In LLM applications, provider, model, tokenizer, rate-limit and error behavior can be examined together to draw meaningful external inferences about the system. These signals help determine the likely architecture and the LLM-specific attack surface.

Risk: Fingerprinting itself is not a vulnerability. However, unnecessary exposure of technical information, inadequate rate limiting and an expanding LLM attack surface can create security and resource-consumption risks.

Solution: Basic precautions include hiding unnecessary model and provider information, secure error handling, appropriate rate limiting, restricting RAG and tool access, and validating model outputs.

Want to know what an attacker can learn about your AI application? Our AI & LLM Security Testing service starts with exactly this reconnaissance phase before testing for prompt injection, data leakage and tool abuse. For the full methodology, see How to Penetration Test an LLM Application. Talk to our team.

References

OWASP Foundation – OWASP Top 10 for LLM Applications 2025: risks and threat categories in LLM applications.
OWASP Foundation – GenAI Red Teaming Guide: model reconnaissance, application decomposition, attack modeling and runtime behavior analysis.
litemars – LLM-Fingerprinter: open-source black-box model fingerprinting and model family estimation.
iSimplifyMe – tokenizer-fingerprint: black-box tokenizer fingerprinting using token-count differences.
Bruckner, T. – One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions (2026), arXiv:2607.10252: fingerprinting with single-token output distributions and Jensen–Shannon Divergence.
vivgrid – llm-fingerprinting: open-source implementation of behavioral fingerprinting.

Subscribe to our cyber security newsletter

Scroll to top