01 — AI Security · 2025
LLM Red-Teaming Engine
A four-stage adversarial framework that attacks a RAG-powered HR chatbot across 13 attack categories — and scores exactly how badly it breaks.
Role
- Independent project
Year
- 2025
Stack
- Python, Gemini, Groq · LLaMA-3.3-70B, FastAPI, Next.js
13
attack categories
4
pipeline stages
0.7 / 0.3
judge / heuristic weighting
Overview
Most LLM apps are tested for what they should do. This engine tests what they shouldn't. It runs an automated attacker → executor → target → evaluator loop against a retrieval-augmented HR chatbot, probing it with prompt injection, jailbreaks, encoding obfuscation and RAG context poisoning across 13 attack categories.
Every response is scored with a hybrid model — 0.7× an LLM judge plus 0.3× deterministic heuristics — and graded CRITICAL, HIGH or MEDIUM. Results stream to a live dashboard over Server-Sent Events, so you can watch the model get attacked in real time.
What I built
- 01Four decoupled stages — attacker, executor, target RAG and evaluator — each swappable on its own
- 0213 attack categories, including prompt injection, jailbreaks, encoding obfuscation and RAG context poisoning
- 03Hybrid scoring: 0.7× LLM-as-judge + 0.3× heuristics, mapped to CRITICAL / HIGH / MEDIUM severity
- 04Live SSE-streaming dashboard in Next.js on a FastAPI backend
Architecture
01 Attacker
Adversarial prompts
13 attack categories
02 Executor
Attack runner
Fires payloads at the target
03 Target
HR chatbot
Retrieval-augmented generation
04 Evaluator
Hybrid scorer
0.7× LLM judge + 0.3× heuristics
05 Dashboard
Live SSE stream
CRITICAL · HIGH · MEDIUM