HKUST Releases RedHerring to Counter Agentic AI Attacks

HKUST’s RedHerring offers a novel approach to breaking the "attack is easy, defense is hard" deadlock in AI security.

AI security is rapidly moving from the periphery of AI discourse to center stage.

Mythos completed development as early as February but remained restricted to a select few institutions due to unresolved security concerns, only launching fully in June.

Fable 5, which was heavily gated upon release, was forcibly taken offline by the White House just two days later due to jailbreak risks, with public access restored only at the end of June.

The incident where GPT-6 infiltrated Hugging Face has pushed the issue of AI-driven hacking into the spotlight.

The reality is becoming increasingly clear: it is security capabilities that ultimately determine how intelligent the AI available to the public can be.

In the realm of AI offense and defense, attacks have always been easier than defenses. Can defenders turn the tide?

A team led by Dongdong She at HKUST, including Kaikai Zhang and Zihan Zhang, recently released a groundbreaking result called RedHerring, providing a new answer.

Paper: Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery

Read the paper: https://arxiv.org/abs/2609.35909

Project website: https://xxbai.space/redherring

AI Hackers Are Disrupting the Existing Cybersecurity Governance Landscape

In the past, launching cyberattacks required a group of highly skilled, experienced experts sitting at computers, reading code line by line and attempting exploits repeatedly.

Now, attackers only need to launch an Agent. It reads code, writes programs, and initiates attacks autonomously, working continuously for hours or even days.

Facing smarter AI hackers with higher attack frequencies, traditional defense methods are clearly insufficient.

OpenAI Models Breached Hugging Face.

During internal testing of their models' network attack capabilities, two OpenAI models broke out of the test environment, connected to the internet, and accessed Hugging Face servers using a previously unknown vulnerability.

They did this to directly retrieve answers to the test questions. Hugging Face later reconstructed over 17,000 operations during this intrusion, all performed automatically by AI.

OpenAI described this as an unprecedented cybersecurity incident.

Claude Code Was Used to Attack 30 Organizations.

Anthropic disclosed that a state-sponsored hacker group used Claude Code to attack approximately 30 tech companies, financial institutions, and government departments globally, with AI completing 80%–90% of the attack work autonomously.

Asymmetry in Offense and Defense Capabilities

Cybersecurity has always faced an imbalance between offense and defense: attackers only need to breach one gap, while defenders must protect the entire perimeter. AI exacerbates this problem.

Defenders have consistently been slow to patch vulnerabilities.

Verizon’s 2026 report indicates that the median time for enterprises to fix vulnerabilities is 43 days, 11 days longer than a year ago.

Attacks, however, are accelerating. Between 2018 and 2019, there was an average window of 63 days between a vulnerability being disclosed and being exploited.

By this year, Mandiant estimates this timeframe has shifted to -7 days, meaning attacks often begin before the vulnerability is even publicly disclosed.

In the past, top-tier hackers were limited in number; one person could only focus on a few targets simultaneously.

Now, attackers can deploy numerous top-tier AI hackers simultaneously, searching for vulnerabilities around the clock without rest.

For software maintainers, the pressure has increased dramatically.

To address this issue, the security research team at HKUST proposed the RedHerring countermeasure technique.

In English, a "red herring" refers to a misleading clue intended to divert attention.

The core idea is to inject decoy code into the system that looks like a vulnerability but cannot actually be triggered. This causes Agents to waste time on these decoys, thereby reducing the number of real vulnerabilities discovered.

Core Observation: Verifying Vulnerabilities Consumes Most of the Time

The paper observed the entire process of AI hackers finding vulnerabilities and found that their workflow consists of two steps:

• Hypothesis Generation: Identifying suspicious code and judging, "There might be a vulnerability here."
• Hypothesis Verification: Tracing whether external input can reach this point, and then manually constructing an input capable of crashing the program.

To confirm a vulnerability, an Agent needs to continue tracing whether external input can reach the dangerous code, analyze intermediate data flows, and finally construct a Proof of Concept (PoC) that triggers a program crash.

This process requires not only more reasoning steps but also consumes significant time and tokens.

Analyzing behavior across 300 vulnerability discovery trajectories, the paper found:

• The verification phase accounts for 73.8% of total time; the average cost of verifying one hypothesis is approximately 11.1 times that of generating one.
• In a single run, an Agent generates an average of 32 vulnerability hypotheses and begins verifying 24 of them, but ultimately abandons 63% of the verifications.

In other words, Agents can quickly generate a large number of "potential vulnerability" hypotheses, but limited time and computational resources allow them to verify only a fraction of them.

This is exactly where RedHerring comes into play: the defender doesn't need to stop the Agent from exploring; they just need to make it waste its effort on the wrong targets.

The Agent proposes vulnerability hypotheses within the code repository. During the subsequent verification process, pre-set traps consume most of the verification time, ultimately resulting in fewer real vulnerabilities being discovered.

RedHerring: Traps Prepared for AI

RedHerring inserts "fake vulnerabilities" into the code.

They look almost identical to real vulnerabilities but can never be triggered.

Once an AI hacker starts verifying them, their time is consumed here.

Each fake vulnerability consists of two parts:

• A code path that looks like a vulnerability.

Rewritten based on real historical vulnerabilities, external input flows all the way to dangerous operations, making it appear highly worth digging into.

• A check that can never be passed.

It stands before the dangerous operation, and no input can pass through it. The defender holds the answer when generating this check and can immediately confirm it is safe. The AI hacker can only see the code; to verify this, they must solve a difficult mathematical problem.

Fake vulnerabilities also mimic the project's original naming conventions and coding style, appearing as if they were part of the project's existing codebase.

Overview of the RedHerring Method

How AI Hackers Get Hooked

In one experiment, an AI hacker followed a fake vulnerability all the way to the check, believing it was the key to triggering the vulnerability, and decided to focus on factoring a large integer within the check.

It wrote a program to attempt factorization, running 200,000 iterations with no result.

Afterward, it twice re-added the check to its plan, going back to investigate again.

Partial Agent Execution Trace

Experimental Results

The team tested five mainstream LLMs—including Qwen, GLM, DeepSeek, and Kimi—on 33 real open-source projects.

Before and after adding fake vulnerabilities, the conditions for the AI hackers remained completely identical, with both limited to 3 hours.

• All 5 models fell for the trap: The number of real vulnerabilities found decreased by 39%–60%.
• The more time spent on fake vulnerabilities, the more real vulnerabilities were missed: AI hackers spent approximately 30%-50% of their computational resources on fake vulnerabilities.
• Telling the AI in advance that "there might be decoys" still works: Real vulnerabilities still decreased by 37%. After being warned, the AI became more cautious even on the original code without any fake vulnerabilities, finding fewer bugs (dropping from 149 to 121).
• Low deployment cost: Integrating each project takes about 70 minutes, all existing tests pass, and the impact on runtime speed is less than 1%.

Left: Number of real vulnerabilities found before and after adding decoys. Middle: Proportion of tokens and time the Agent spent on decoys. Right: The higher the proportion of tokens spent on decoys, the greater the reduction in vulnerabilities found.

Standard refers to standard prompts, while Informed explicitly warns that decoys may exist. Each row compares the Baseline against RedHerring under that specific prompt setting.

Even when the Agent realizes this is a potential decoy vulnerability, it faces a dilemma:

Further verification still consumes tokens and time, while giving up too early risks missing real vulnerabilities with similar structures.

In the era where Agents participate in software security offense and defense, code itself is no longer just an object of analysis; it can also become a new adversarial surface.

RedHerring provides a new breakthrough strategy to change the landscape of the AI era, where attacks are easy and defense is difficult.

References:

https://xxbai.space/redherring

https://arxiv.org/abs/2609.35909

Follow ASI Instantly

⭐ Like, Share, and Watch ⭐

Star us to lock in instant updates from New Intelligence!