Prompt Injection
59 items from fonduai/awesome-prompt-injection ★679
-
Improving LLM Security Against Prompt Injection: AppSec Guidance For Pentesters and Developers – Part 2 blog.includesecurity.com
Understanding Transformer Models (especially attention), Causes and How Do We Stop Prompt Injection.
-
Improving LLM Security Against Prompt Injection: AppSec Guidance For Pentesters and Developers blog.includesecurity.com
Using Role Based APIs to Minimize the Risk of Prompt Injection and 13 Guidelines for Writing System Prompts that Minimize the Risk of Prompt Injection.
-
MITRE ATLAS atlas.mitre.org
MITRE's adversarial ML threat matrix formally cataloging direct and indirect prompt injection as core adversary techniques, enabling integration into enterprise threat modelling and purple team exercises.
-
The Landscape of Prompt Injection Threats in LLM Agents (SoK) arxiv.org
Feb 2026 systematization-of-knowledge paper with a unified taxonomy covering attack payload strategies (heuristic vs. optimisation-based) and defense intervention stages (text, model, execution). Introduces the AgentPI benchmark for context-dependent agent tasks that all prior…
-
Securing AI Agents Against Prompt Injection Attacks arxiv.org
Nov 2025 benchmark of 847 adversarial test cases across 5 attack categories against 7 LLMs. The combined defense framework reduces attack success from 73.2% to 8.7% while retaining 94.3% of baseline task performance.
-
Oct 2025 paper systematically breaking 12 published defenses using gradient descent, RL, random search, and human-guided exploration. Most defenses originally claimed near-zero attack success rates; adaptive attacks exceeded 90% against all of them.
-
Prompt Injection 2.0: Hybrid AI Threats arxiv.org
Jul 2025 paper showing how prompt injections now combine with XSS, CSRF, AI worm propagation, and multi-agent infections to evade traditional WAFs entirely. Evaluates Preamble's classifier, data-tagging, and RL-based defenses against these hybrid scenarios.
-
ToolHijacker: Prompt Injection Attack to Tool Selection in LLM Agents arxiv.org
Apr 2025 paper introducing a no-box attack that injects a malicious tool document into an agent's tool library to consistently hijack tool selection. Finds that StruQ, SecAlign, DataSentinel, and perplexity detection are all insufficient defenses.
-
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models arxiv.org
Discovers that text embedding models have severely biased output distributions, and exploits this to find universal adversarial suffixes ("magic words") that bypass embedding-based LLM safeguards. Attacks transfer across models and languages; a train-free debiasing defense is…
-
Universal and Transferable Adversarial Attacks on Aligned Language Models arxiv.org
This paper introduces a simple and efficient attack method that enables aligned language models to generate objectionable content with high probability, highlighting the need for improved prevention techniques in large language models. The generated adversarial prompts are…
-
This paper explores the concept of Indirect Prompt Injection attacks on Large Language Models (LLMs) through their integration with various applications. It identifies significant security risks, including remote data theft and ecosystem contamination, present in both…
-
Agents Rule of Two: A Practical Approach to AI Agent Security ai.meta.com
Meta's Oct 2025 framework stating that agents must satisfy no more than two of: (A) processing untrustworthy inputs, (B) access to sensitive data, (C) ability to change state externally, a deterministic architectural approach to bounding blast radius.
-
Attention Tracker: Detecting Prompt Injection Attacks in LLMs aclanthology.org
NAACL 2025 Findings paper detecting prompt injection by tracking attention distribution shifts, with no modification to the underlying model required, making it deployable as a wrapper on any LLM.
-
AI/LLM Exploitation Challenges academy.8ksec.io
AI, ML, and LLMs CTF Challenges.
