The team that breaks before the attacker does
Red teaming is the practice of attacking your own system before someone else does. For LLM apps it means: take your prompts, your tools, your data flows — and try to make them misbehave. The attacks you find become regression tests.
What to attack
- Direct injection in the user message.
- Indirect injection in a doc / tool result you control.
- Persona / role overrides via roleplay framings.
- Privilege escalation via tool argument manipulation.
- Output exfiltration via embedded URLs / images.
- Cross-user data leakage via tool argument confusion.
How to do it sustainably
- Start with a checklist; expand as you discover.
- Each found issue becomes an eval test case.
- Run the suite on every prompt or model update.
- Rotate the red-teamer if possible; one person blind-spots fast.
Public resources
Anthropic's HackerOne program, OpenAI's red-teaming network, and academic adversarial-prompt repos give you a starter set of patterns. Don't reinvent.