A talk by Dima Nekrasov from Amazon at the TechWorks-AI conference , Bletchley Park February 2025

Enterprise systems that rely on large language models bring enormous productivity benefits, but they also open wide attack surfaces. Adversarial techniques can produce unwanted outputs, leak sensitive information, or even cause financial and operational damage. Understanding the common threats and practical mitigations is essential for anyone building or operating LLM-based chatbots, agents, and knowledge co-pilots.

Why LLM-based systems are vulnerable

LLMs cannot reliably distinguish between commands they must follow and the data they should passively process.

This simple limitation underlies many attacks. Models are powerful at pattern completion and instruction following, but they are not inherently able to separate “execute” from “observe.” That ambiguity is what attackers exploit with prompt injections, poisoned data, and other vectors. Because modern enterprise AI systems are composed of many components, even a secure core model can be compromised through a weak integration or third-party module.

Top 10 risks for LLM applications (overview)

The community has converged on a common taxonomy of risks. These are the areas you should be thinking about when assessing the safety and security of your enterprise AI:

• Prompt injection — Adversarial inputs that cause the model to ignore guardrails or reveal secrets.
• Sensitive information disclosure — The model leaking data from pretraining or fine-tuning corpora.
• Supply chain risk — Third-party components introducing vulnerabilities into your system.
• Data and model poisoning — Maliciously injected training data that shifts behaviour or injects faults.
• Improper output handling — Treating model output as safe executable commands or SQL without sanitisation.
• Excessive agency — Agents given permissions beyond their intended scope, able to take harmful actions.
• System prompt leakage — Exposure of the hidden prompts that encode policy and logic for the model.
• Embedding and vector store weaknesses — Poisoning or unauthorized access to vectorised knowledge used for retrieval.
• Misinformation — Poisoned knowledge bases or model outputs that spread falsehoods internally.
• Unbounded consumption — Abusive requests that consume excessive compute or extract model internals.

Three risks to prioritise right now

All ten risks matter, but three are especially relevant for co-pilots, agents, and knowledge retrieval systems: prompt injection, excessive agency, and system prompt leakage. These are common, high-impact, and often underestimated.

1. Prompt injection: direct and indirect

Prompt injection comes in two flavours. A direct injection is when an attacker supplies malicious text in an interaction that causes the model to ignore safety instructions or leak data. An indirect injection is more subtle: the attacker plants poisoned content inside the organisation’s knowledge base—Confluence pages, wikis, or other indexed documents—that the retrieval pipeline later feeds to the model.

Mitigations that help reduce the risk:

• Limit who can add content to knowledge stores and enforce provenance and editorial workflows.
• Sanitise and validate retrieved passages before they are included in prompts. Treat retrieved text as untrusted input.
• Use scoped access: restrict high-risk agents to a small, monitored group while maturity grows.
• Log retrievals and model inputs so you can trace the origin of decisions and detect anomalous patterns.
• Research layered defenses such as adversarially trained classifiers that flag injected prompts, while recognising these approaches are still evolving.

2. Excessive agency: rights without restraint

Giving an agent write or delete permissions across multiple systems drastically increases blast radius. During development, broad permissions make testing easier, but production agents should operate with the minimum privileges required.

Best practices to contain agency:

• Enforce the principle of least privilege for every capability and re-evaluate permissions as the agent’s data footprint grows.
• Tie actions to the actual user context and credentials that invoked them, not to a global agent identity.
• Require explicit human approval for mission-critical or destructive operations. Maintain a human in the loop for file deletion, bulk updates, or policy changes.
• Audit agent behaviour frequently and implement safeguards that prevent self-initiated tasks unless explicitly allowed and logged.

3. System prompt leakage: protecting the policy layer

System prompts encode the decision logic, role definitions, and guardrails of LLM systems. If an attacker obtains or infers these prompts, they can craft attacks that bypass protections or extract sensitive logic. Even partial exposure of the prompt can significantly weaken security.

How to protect system prompts and critical logic:

• Avoid encoding mission-critical authentication or verification logic solely in prompts. Use dedicated identity and policy systems instead.
• Store prompts securely with proper access controls and rotate or version them.
• Design guardrails as externalised checks rather than hidden prompt text. Validate outputs with independent rule engines where feasible.
• Monitor for prompt leakage and treat any suspected exposure as a high-priority incident.

Additional practical defenses

Beyond the three priorities above, a robust defensive program includes:

• Supply chain vetting — Assess third-party models and components for security posture before integration.
• Data integrity controls — Use checksums, provenance tags, and ingestion whitelists for knowledge bases.
• Output handling — Never execute model-generated code, queries, or commands without validation and sanitisation. For example, treat generated SQL as untrusted and run it only through parameterised interfaces or a query builder.
• Rate limits and cost controls — Prevent unbounded consumption or brute-force extraction of model behaviour.
• Continuous monitoring — Log inputs, retrievals, actions, and outcomes. Use anomaly detection to flag unusual patterns.
• Human oversight — Keep human reviewers in the loop for sensitive decisions and for regular audits of agent behaviour and access lists.

Operational checklist

1. Map your attack surface: list models, data stores, retrieval pipelines, and third-party components.
2. Apply least privilege to agents and re-audit permissions quarterly.
3. Protect knowledge sources with editorial controls and provenance metadata.
4. Externalise critical logic from prompts and use dedicated auth and policy systems.
5. Log and monitor all retrievals and agent actions. Implement alerting for anomalous activity.
6. Require human approval for destructive or mission-critical operations.
7. Vet the supply chain and test integrations for adversarial behaviours.

Closing thought

LLM-based enterprise systems are powerful but fragile in the face of adversarial techniques. Practical security comes from combining multiple controls: limiting access, validating inputs and outputs, externalising critical checks, and keeping humans in the loop for important actions. Expect the threat surface to evolve and build processes that allow you to re-evaluate permissions, data quality, and agent capabilities on a regular cadence.

Protecting these systems is not a one-off task. It is an operational discipline that balances the benefits of automation with careful constraint and oversight.