Reverse Engineering AI Products: Model Extraction, Prompt Leakage and the Wrapper Economy

Why teams reverse engineer AI products

Reverse engineering an AI product is not only a competitive-intelligence exercise. It is how security teams find prompt injection paths, how researchers measure model behaviour, and how engineers decide whether a vendor is running a real system or a thin wrapper around someone else's API.

The discipline is familiar from traditional software: observe the interface, form hypotheses, test them, and document the result. What changes is the surface area. An AI product exposes natural-language inputs, probabilistic outputs, tool calls, latency patterns and sometimes retrieved documents. Each is a signal.

Black-box model extraction

Model extraction studies how much of a model's behaviour can be recovered through queries alone. Tramèr et al. showed that prediction APIs can leak enough information to train a substitute model, and that the substitute can be used to approximate the original's decisions or to mount further attacks.

For an AI product, the same logic applies at the system level. A team can probe a service with carefully chosen inputs, record outputs and build a behavioural clone. Strong rate limits, diverse monitoring and output perturbations raise the cost, but they do not make extraction impossible. The practical defence is to assume that anything a user can query repeatedly can be characterised.

Prompt leakage

A production AI product often contains a system prompt, a routing policy and a set of tool instructions. Those are part of the product's logic, and they can leak.

Perez and Ribeiro demonstrated prompt injection attacks that coax a model into ignoring its instructions or revealing its configuration. Greshake et al. extended the problem to indirect prompt injection, where malicious instructions arrive through retrieved content rather than the user. In a product that summarises web pages or reads uploaded documents, that is a direct path to prompt leakage and unintended tool use.

The lesson is not "write a better prompt". It is "do not put secrets in the prompt". If a value must remain secret, keep it outside the model context and enforce it in code.

Fingerprinting the model

Model fingerprinting tries to identify which model family or checkpoint is behind an interface. Signals include tokeniser quirks, refusal patterns, formatting habits, latency distributions and the shape of errors. None is conclusive on its own, and the research on robust fingerprinting is still developing.

For defenders, the implication is modest: do not rely on obscurity. Assume a determined user can infer the general class of model. Compete on product quality, data and workflow rather than on hiding the model name.

Memorisation and data extraction

Large language models can reproduce training data in some conditions. Carlini et al. showed extraction of verbatim sequences from a production language model, and membership-inference research (Shokri et al.) showed that models can leak whether a record was in the training set.

For product teams, the risk is not only model weights. It is the data pipeline: logs, retrieved documents and fine-tuning sets are all potential exfiltration surfaces. Minimise what is stored, redact before logging, and treat model outputs as untrusted when they are shown to other users.

Defensive design

The controls that matter are architectural:

• Least privilege: give the model only the tools and data the task requires.
• Secrets outside context: enforce API keys, permissions and business rules in code, never in a prompt.
• Input isolation: separate retrieved content from instructions, and mark untrusted text as data.
• Rate limits and quotas: make extraction expensive and detectable.
• Output filtering: validate structured output and scan for sensitive patterns before display.
• Audit logs: record tool calls, arguments, responses and the model version.
• Canary tokens: place unique markers in prompts or documents to detect leakage.
• Watermarking: where appropriate, use output watermarking research to support provenance.

None of these is sufficient alone. Together they turn a probabilistic system into something that can be operated and investigated.

Rules of engagement

Reverse engineering sits inside legal and ethical boundaries. Work only on systems you own or have explicit permission to test. Respect terms of service, rate limits and disclosure norms. If you find a vulnerability, report it privately before publishing.

The technical lesson is the same one that applies to any API: treat every externally visible behaviour as part of the attack surface, and design as if it will be observed.

References

1. Tramèr et al. (2016). Stealing Machine Learning Models via Prediction APIs. USENIX Security.
2. Carlini et al. (2021). Extracting Training Data from Large Language Models. USENIX Security.
3. Shokri et al. (2017). Membership Inference Attacks Against Machine Learning Models. IEEE S&P.
4. Perez and Ribeiro (2022). Ignore Previous Prompt: Attack Techniques For Language Models.
5. Greshake et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec.
6. Kirchenbauer et al. (2023). A Watermark for Large Language Models. ICML.
7. Mitchell et al. (2019). Model Cards for Model Reporting. FAT*.