Commercial foundation models are fortified with billions of dollars of safety alignment: layers of RLHF, DPO, and red-teaming designed to refuse hazardous queries concerning bioweapons, cyber warfare, or kinetic sabotage. Yet modern language models remain fragile artifacts vulnerable to Adversarial Attacks & Jailbreaks. By exploiting the non-linear high-dimensional geometry of neural embeddings via Greedy Coordinate Gradient (GCG) search and Activation Steering, researchers uncover the delicate boundary between safety and compromise.
1. The Continuous Vulnerability of Discrete Token Spaces
In computer vision, imperceptible pixel perturbations $delta$ fool convolutional classifiers (Szegedy et al., 2013). In text, token inputs are discrete categorical integers, making gradient descent non-trivial. However, the model embeds discrete tokens into a continuous vector manifold $mathbf{E} in mathbb{R}^{V times d}$.
Zou et al. (2023) developed Greedy Coordinate Gradient (GCG), introducing universal transferable adversarial suffixes. Given harmful prompt $P$ and target affirmative prefix $T$ (“Sure, here is how to synthesize…”), GCG solves:
min_{S in mathcal{V}^L} – sum_{i=1}^{|T|} log P_theta(t_i mid P, S, t_{<i})
By computing the first-order gradient of the loss with respect to the one-hot token embedding vectors $nabla_{mathbf{e}_i} mathcal{L}$, GCG identifies the top-$k$ candidate token replacements that maximize loss reduction. Within hours of optimization, GCG generates gibberish-looking suffixes (e.g., ! ? ; { desc == True ...) that bypass alignment filters across GPT-4, Claude, and LLaMA with up to 99% attack success rates.
2. Many-Shot In-Context Jailbreaking
Anthropic (2024) discovered that long-context windows (100k+ tokens) open a severe non-adversarial vulnerability: Many-Shot Jailbreaking. By formatting the prompt with hundreds of benign question-answer demonstrations following a permissive template, the in-context learning mechanism overpowers the model’s fine-tuned safety refusal training, forcing the attention heads to conform to the dominant statistical pattern.
3. Refusal Direction Ablation & Representation Steering
Arditi et al. (2024) uncovered the geometric mechanism of model refusals: safety alignment is mediated almost entirely by a single 1D direction vector $mathbf{v}_{text{refusal}} in mathbb{R}^d$ inside intermediate residual streams.
By contrasting activations on harmful queries versus harmless queries, researchers extract:
mathbf{v}_{text{refusal}} = mathbb{E}[mathbf{x}_{text{harmful}}] – mathbb{E}[mathbf{x}_{text{harmless}}]
Remarkably, surgically erasing this single direction from the model’s weight matrices via orthogonal projection:
mathbf{W}’ = mathbf{W} (mathbf{I} – hat{mathbf{v}}hat{mathbf{v}}^T)
Completely strips the model of all refusal behavior across every language, without degrading standard benchmark capabilities. Conversely, Activation Steering adds $+alpha mathbf{v}_{text{refusal}}$ directly into the stream during inference, making the model impervious to jailbreaks regardless of prompt phrasing.
