Skip to content
Bhuman Pandita
← Index

01 — AI Security · 2025

LLM Red-Teaming Engine

A four-stage adversarial framework that attacks a RAG-powered HR chatbot across 13 attack categories — and scores exactly how badly it breaks.

Role

Independent project

Year

2025

Stack

Python, Gemini, Groq · LLaMA-3.3-70B, FastAPI, Next.js

Overview

Most LLM apps are tested for what they should do. This engine tests what they shouldn't. It runs an automated attacker → executor → target → evaluator loop against a retrieval-augmented HR chatbot, probing it with prompt injection, jailbreaks, encoding obfuscation and RAG context poisoning across 13 attack categories.

Every response is scored with a hybrid model — 0.7× an LLM judge plus 0.3× deterministic heuristics — and graded CRITICAL, HIGH or MEDIUM. Results stream to a live dashboard over Server-Sent Events, so you can watch the model get attacked in real time.

What I built

  • 01Four decoupled stages — attacker, executor, target RAG and evaluator — each swappable on its own
  • 0213 attack categories, including prompt injection, jailbreaks, encoding obfuscation and RAG context poisoning
  • 03Hybrid scoring: 0.7× LLM-as-judge + 0.3× heuristics, mapped to CRITICAL / HIGH / MEDIUM severity
  • 04Live SSE-streaming dashboard in Next.js on a FastAPI backend

Architecture

01 Attacker

Adversarial prompts

13 attack categories

02 Executor

Attack runner

Fires payloads at the target

03 Target

HR chatbot

Retrieval-augmented generation

04 Evaluator

Hybrid scorer

0.7× LLM judge + 0.3× heuristics

05 Dashboard

Live SSE stream

CRITICAL · HIGH · MEDIUM

Each attack flows left to right; scores stream back to the dashboard live.