AI Agents: Mastering Code, Discovering Zero-Days, and Reshaping Cybersecurity
The landscape of software development and cybersecurity is undergoing a profound transformation, driven by the rapid advancements in artificial intelligence. AI models are no longer confined to simple tasks; they are becoming increasingly sophisticated agents capable of complex reasoning, code generation, and, perhaps most significantly, the identification of critical security vulnerabilities. This dual capability—excelling at both building and breaking software—heralds a new era for digital security, presenting both unprecedented opportunities for defense and concerning potential for offense.
Recent research highlights this evolving reality. A team of AI researchers at UC Berkeley conducted extensive tests to evaluate the performance of cutting-edge AI models and agents in discovering bugs within a large collection of open-source codebases. Their work, utilizing a novel benchmark environment called CyberGym, yielded remarkable results, uncovering 17 new bugs, 15 of which were previously unknown, or “zero-day,” vulnerabilities. According to Dawn Song, a professor at UC Berkeley who spearheaded the research, many of these vulnerabilities are critical, underscoring the potential impact of AI in this domain.
This academic exploration is mirrored in the commercial world. AI tools are already making waves in the competitive field of bug hunting. Xbow, an AI tool developed by a startup of the same name, has rapidly ascended the ranks of HackerOne's leaderboard, a prominent platform for bug bounty programs where security researchers are rewarded for finding and reporting vulnerabilities. Xbow currently holds the top position, a testament to its effectiveness in identifying security flaws at scale. The company's recent announcement of $75 million in new funding further signals the growing industry confidence in AI-powered cybersecurity solutions.
Dawn Song emphasizes that the combination of advanced coding skills and improving reasoning abilities in the latest AI models is fundamentally altering the cybersecurity landscape. She describes this moment as pivotal, noting that the AI agents' performance exceeded general expectations. As these models continue their trajectory of improvement, they are poised to automate significant portions of the vulnerability discovery and exploitation process. While this could empower companies to more effectively secure their software, it also raises concerns about the potential for malicious actors to leverage similar tools for nefarious purposes. Song points out that their research effort was relatively constrained in terms of resources and runtime, suggesting that with greater investment, AI agents could achieve even higher levels of bug-finding proficiency.
The CyberGym Benchmark: A New Arena for AI Security Testing
To rigorously assess the capabilities of AI agents in cybersecurity, the UC Berkeley team developed the CyberGym benchmark. This environment provides a standardized framework for evaluating how well AI models and agents can identify vulnerabilities in real-world codebases. The benchmark is designed to simulate realistic bug-hunting scenarios, allowing researchers to measure AI performance across various tasks relevant to cybersecurity.
The study involved testing a range of frontier AI models from leading labs such as OpenAI, Google, and Anthropic, alongside open-source alternatives from Meta, DeepSeek, and Alibaba. These models were integrated with several specialized AI agents designed for cybersecurity tasks, including OpenHands, Cybench, and EnIGMA. The combination of powerful language models with task-specific agents allows for more targeted and effective security analysis.
The researchers employed a two-pronged approach within the CyberGym environment. First, they provided the AI agents with descriptions of known software vulnerabilities from 188 large open-source projects. The agents were then tasked with analyzing new codebases to see if they could independently identify the same types of flaws. This involved analyzing code, running tests, and attempting to craft proof-of-concept exploits. Second, the team challenged the agents to hunt for entirely new, previously unknown vulnerabilities within these codebases without prior knowledge of specific flaw types.
The results of this process were significant. The AI tools generated hundreds of potential proof-of-concept exploits. Upon review, the researchers validated 15 previously unseen vulnerabilities (zero-days) and two vulnerabilities that had been previously disclosed and patched. This work provides compelling evidence that AI can indeed automate the discovery of zero-day vulnerabilities, which are particularly valuable and dangerous because they represent unknown weaknesses that can be exploited before defenders are even aware of their existence.
AI in Action: Real-World Bug Discovery
The findings from the CyberGym benchmark are not isolated incidents. AI-assisted vulnerability discovery is already happening in the real world. Security expert Sean Heelan recently utilized OpenAI's reasoning model, o3, to help uncover a zero-day flaw in the widely used Linux kernel. This vulnerability, identified as CVE-2025-37899, was a remote zero-day in the kernel's SMB implementation, demonstrating how AI can augment human expertise in finding complex bugs in critical software infrastructure. Heelan documented his process, highlighting the utility of AI in navigating and understanding large codebases to pinpoint potential weaknesses.
Google's internal security team, Project Zero, known for its work in finding zero-day vulnerabilities in Google and third-party software, also announced in November that it had successfully discovered a previously unknown software vulnerability using AI. This initiative, part of a program called “From Naptime to Big Sleep,” showcases how major technology companies are integrating AI into their advanced security research efforts. These examples from independent researchers and large corporations alike underscore the growing practical application of AI in vulnerability discovery.
While the potential is clear, the UC Berkeley study and real-world examples also highlight the current limitations of AI in cybersecurity. Despite finding critical zero-days, the AI systems tested in CyberGym were unable to find the majority of known flaws and struggled with particularly complex vulnerabilities. This suggests that while AI can be a powerful tool, it is not yet a panacea and cannot fully replace the intuition, experience, and deep understanding that human security experts bring to the table.
The Dual Nature of AI in Cybersecurity: An Arms Race
The increasing capability of AI in both writing code and finding vulnerabilities presents a classic dual-use technology dilemma. On one hand, AI can significantly enhance defensive capabilities. Automated vulnerability scanning and analysis tools powered by advanced AI can process vast amounts of code more quickly and potentially identify subtle flaws that might be missed by traditional methods or human review alone. This could lead to faster patching cycles and more secure software overall. Companies could leverage AI agents to continuously audit their codebases, reducing their attack surface.
On the other hand, the same capabilities can be weaponized by attackers. If AI can automate the discovery of zero-day vulnerabilities, it could democratize access to sophisticated hacking tools and techniques. Brendan Dolan-Gavitt, an associate professor at New York University Tandon and a researcher at Xbow, anticipates that AI will drive an increase in attacks leveraging zero-day exploits. He notes that currently, finding and exploiting zero-days requires a rare combination of expertise and resources, limiting their use. AI could lower this barrier, making zero-days more accessible to a wider range of actors, including those with less technical sophistication.
Hayden Smith, a cofounder of Hunted Labs, a startup providing AI-assisted code analysis tools, finds the potential of “agentic stuff” (referring to AI agents capable of autonomous actions) for zero-day discovery particularly fascinating. He points out that as AI makes vulnerability discovery more accessible, the importance of responsible disclosure practices becomes even more critical. Ensuring that newly found vulnerabilities are reported to vendors so they can be patched, rather than being sold on the black market or used for malicious purposes, is a significant challenge in this evolving landscape.
AI and the Bug Bounty Ecosystem
The economic implications of AI's bug-finding capabilities are also becoming apparent. In work posted online in May, Dawn Song and other researchers explored the potential for AI models to earn payouts through bug bounty programs. These programs reward security researchers for finding and reporting vulnerabilities to companies, creating a market for security intelligence.
The study, using a benchmark called BountyBench, measured how well AI models could find bugs that would qualify for cash rewards. The results indicated that these tools could potentially generate significant income. Anthropic's Claude Code model proved particularly successful, identifying bugs worth $1,350 on bug bounty boards and designing patches for vulnerabilities valued at $13,862. The cost of achieving these results using API calls was only a few hundred dollars, suggesting a potentially high return on investment for AI-powered bug hunting.
This raises questions about the future of the bug bounty ecosystem. Will AI agents become dominant players, outpacing human researchers? Will bounty values change as AI makes certain types of bugs easier to find? The economic incentives could further accelerate the development and deployment of AI for vulnerability discovery, intensifying the cybersecurity arms race.
Tracking the Threat: The Need for Observatories
Given the rapid pace of AI development and its potential impact on cybersecurity, it is crucial to monitor and understand the evolving capabilities of these tools. In a blog post in April, Dawn Song and several other AI security experts warned that steadily improving AI models are likely to benefit attackers more than defenders in the near future. This potential shift in the balance of power makes it essential to closely track how capable these tools are becoming.
To address this need, Song and her colleagues have established the AI Frontiers CyberSecurity Observatory. This collaborative effort aims to track the capabilities of different AI models and tools through various benchmarks, including CyberGym and BountyBench. The observatory provides a public platform for researchers and the broader community to stay informed about the state of the art in AI-powered cybersecurity.
Song believes that among all the potential risks associated with frontier AI, cybersecurity is one of the most immediate and potentially problematic domains. The ability of AI to quickly identify and exploit vulnerabilities could have widespread consequences for individuals, organizations, and critical infrastructure.
Human Expertise Remains Paramount
Despite the impressive strides made by AI in bug hunting, experts caution against overreliance on the technology. Katie Moussouris, founder and CEO of Luta Security, a firm specializing in vulnerability disclosure and bug bounty programs, acknowledges the value of the research but emphasizes that AI is still no match for human expertise. She notes that even the best combination of models and agents in the CyberGym study (Claude and OpenHands) were only able to find around 2 percent of the vulnerabilities tested. “Don’t replace your human bug hunters yet,” Moussouris advises.
Moussouris expresses less concern about AI directly hacking software and more concern about companies potentially investing too heavily in AI solutions at the expense of proven security techniques and human talent. Effective cybersecurity requires a multifaceted approach that includes skilled human analysts, robust processes, and a deep understanding of complex systems, in addition to leveraging advanced tools.
The most effective approach likely involves a synergy between human and artificial intelligence. AI can handle the tedious, large-scale scanning and analysis tasks, sifting through vast amounts of code to identify potential areas of interest. Human experts can then apply their nuanced understanding, creativity, and intuition to investigate these leads, confirm vulnerabilities, and develop effective patches or countermeasures. This collaborative model, where AI augments human capabilities rather than replacing them, may represent the most promising path forward for cybersecurity.
The Road Ahead
The integration of AI into software development and cybersecurity is still in its early stages, but its impact is already undeniable. AI agents are becoming increasingly proficient at both building and breaking code, pushing the boundaries of what is possible in digital security.
The ability to automate the discovery of zero-day vulnerabilities is a game-changer, with implications for everything from national security to the protection of personal data. While AI offers powerful new tools for defenders, it also lowers the barrier to entry for attackers, potentially leading to an increase in sophisticated cyberattacks.
The ongoing research, benchmarks like CyberGym and BountyBench, and initiatives like the AI Frontiers CyberSecurity Observatory are vital for understanding and navigating this rapidly evolving landscape. By tracking AI capabilities, fostering responsible disclosure, and emphasizing the continued importance of human expertise, the cybersecurity community can work to harness the power of AI for good while mitigating its potential risks.
The future of cybersecurity will undoubtedly be shaped by AI. The challenge lies in ensuring that this powerful technology is developed and deployed in a way that strengthens our digital defenses faster than it empowers those who seek to exploit our vulnerabilities. The arms race is on, and AI is now a key player on both sides.
