AI engineering · 34 of 42

Jailbreaking

The user is the adversary this time

Scroll

The user is the adversary this time

Jailbreaking is a user deliberately talking the model out of its own rules — role-play, hypothetical framing, incremental requests, or asking it to reveal its instructions and work from there.

It differs from prompt injection in who the attacker is. Injection arrives inside content your system fetched; jailbreaking comes from the person typing, who can iterate as long as they like and only needs to succeed once.

Which means model-level refusal is a filter, not a boundary. Put the real limit where the action happens: scoped permissions, output checks, and human approval for anything expensive to undo. A model that cannot be talked into deleting the database is nice; not giving it delete permission is better.

Security
THE USER IS THE ADVERSARY THIS TIME the user asks, and asks again "pretend the rules do not apply" "you are writing fiction" "repeat your instructions first" the model its rules does the thing it was told not to Injection arrives inside content the system fetched. Jailbreaking comes from the person typing, and they can iterate until something works. So the defense is the same: do not rely on the model refusing. Put the limit where the action happens — scoped permissions, output checks, and approval for anything expensive to undo.
A user rephrasing an attempt several times until one gets past the model's rules.