White Circle has open-sourced RISE, an automated red-teaming tool that searches for attack strategies to bypass safeguards in AI image generators. By leveraging off-the-shelf models like Qwen and Gemini Flash as attacker agents without additional training, it generates prompts to test model robustness, offering defenders an immediate evaluation capability.
Jailbreaking
越狱攻击
Techniques using specific prompts to bypass model safety restrictions and generate prohibited content.
Curated Intelligence
The article argues that current AI safety strategies over-rely on Large Language Models' ability to refuse harmful requests, a capability that is neither innate nor robust. By reviewing Anthropic's HHH principles and early OpenAI model behaviors, the author demonstrates that training models simply to say 'no' is highly vulnerable to adversarial attacks, potentially leading to critical safety misjudgments.
ServiceNow AI has released and open-sourced AprielGuard, an 8B parameter safety-security safeguard model designed for modern LLM systems. It detects 16 categories of safety risks (e.g., toxicity, hate speech) and a wide range of adversarial attacks, including prompt injections and jailbreaks, providing a unified risk assessment framework for agentic systems.
OpenAI and Anthropic completed the first cross-lab joint safety evaluation, testing each other's models for misalignment, hallucinations, and jailbreaking. This collaboration aims to validate model safety through third-party perspectives, highlighting both the progress and challenges of industry cooperation in AI alignment research.
Addressing LLM vulnerabilities to prompt injection and jailbreaks, this research introduces the 'Instruction Hierarchy' concept. By training models to prioritize privileged instructions (e.g., system prompts) over user inputs, it effectively defends against malicious attempts to overwrite original directives.