Hacker, bodyguard, target: AI’s triple act at the year’s biggest security showdowns

Welcome to Eye on AI! AI reporter Sharon Goldman here, filling in for Jeremy Kahn, who is on holiday. In this edition… China tells firms to avoid Nvidia H20 Chips after Trump ends ban but takes cut of revenue… Students are flocking to college courses in AI… Anthropic will offer Claude AI to the U.S. government for $1.

Las Vegas in August feels like another planet—blazing heat, flashing lights, and the constant clatter of slot machines. That otherworldly vibe carried over to the two conferences I attended last week: Black Hat and DEF CON, two of the year’s biggest security and “ethical” hacking conferences, where cutting-edge security research is presented; hackers race to expose flaws in everything from AI chatbots to power grids; and governments, corporations, and hobbyists swap notes and learn the latest threat-fighting techniques. 

I was there because AI now occupies a strange place in the security world—it is both a vulnerable target, often under threat from malicious AI; an armed defender using that same technology to protect systems and networks from bad actors; and an offensive player, deployed to probe weaknesses or carry out attacks (illegally in criminal hands). Sound confusing? It is—and that contradiction was on full display at the very corporate Black Hat, as well as DEF CON, often referred to as “hacker summer camp.” 

Here are three of my favorite takeaways from the conferences:

• ChatGPT agents can be hacked. At Black Hat, researchers from security firm Zenity showed how hackers could exploit OpenAI’s new Connectors feature, which lets ChatGPT pull in data from apps like Google Drive, SharePoint, and GitHub. Their proof-of-concept attack, called AgentFlayer, used an innocent-looking “poisoned” document with hidden instructions to quietly make ChatGPT search the victim’s files for sensitive information and send it back to the attacker. The kicker: It required no clicks or downloads from the user—this video shows how it was done. OpenAI fixed the flaw after being alerted, but the episode underscores a growing risk as AI systems link to more outside apps: sneaky “prompt injection” attacks, known as zero-click attacks (like the one I reported on in June) that trick chatbots into doing the hacker’s bidding without the user realizing it. 

• AI can protect our most critical infrastructure. That idea was the driving force behind the two-year AI Cyber Challenge (AIxCC), which tasked teams of developers with building generative AI tools to find and fix software vulnerabilities in the code that powers everything from banks and hospitals to public utilities. The competition—run by DARPA in partnership with ARPA-H—wrapped up at this year’s DEF CON, where winners showed off autonomous AI systems capable of securing the open-source software that underpins much of the world’s critical infrastructure. The top three teams will receive $4 million, $3 million, and $1.5 million, respectively, for their performance in the finals. 

• Anthropic’s Claude AI proved it can sometimes match elite hackers but still struggles on hard problems. Keane Lucas, a member of Anthropic’s Frontier Red Team, presented the fascinating case study of how the company quietly entered its Claude AI into seven major cybersecurity competitions—events typically dominated by human hackers and security pros—to see how it stacked up. Claude often landed in the top quarter of competitors and showed standout speed on simple challenges, sometimes matching elite human teams, but it lagged badly on the hardest problems. There were also quirks unique to AI—such as getting derailed by ASCII art or lapsing into philosophical rambling during long tasks. Anthropic says these experiments highlight both the offensive potential of AI (by lowering the skill and cost barriers to launching attacks) and its defensive promise, while underscoring persistent limits like long-term memory and complex reasoning.

Also: In just a few weeks, I will be headed to Park City, Utah, to participate in our annual Brainstorm Tech conference at the Montage Deer Valley! Space is limited, so if you’re interested in joining me, register here. I highly recommend: There’s a fantastic lineup of speakers, including Ashley Kramer, chief revenue officer of OpenAI; John Furner, president and CEO of Walmart U.S.; Tony Xu, founder and CEO of DoorDash; and many, many more!

With that, here’s more AI news.

Sharon Goldman
sharon.goldman@fortune.com
@sharongoldman

This story was originally featured on Fortune.com