OpenAI Operator Agent Used in Proof-of-Concept Phishing Attack

Security vendor Symantec used the OpenAI Operator agent to show how an LLM-powered tool could perform a basic cyberattack with minimal prompt engineering — a showcase of what the future might hold. Symantec published its proof of concept as a research blog post on March 12. As the Broadcom-owned vendor notes, most of the attacker use cases for large language models (LLMs) are passive up to this point. LLMs can generate content (such as convincing phishing emails), assist with basic coding tasks, and, as Google said in January, conduct some research tasks.

However, with the onset of generative AI-powered agents, which can perform additional tasks such as interacting with Web pages, this increased capability for customers can also result in increased capability for attackers.

An Automated Phishing Attack

For the March 12 research, Symantec researchers used OpenAI's Operator agent, which launched as a research preview for US-based OpenAI Pro users (currently a $200/month membership for individuals) on Jan. 23. They asked the agent to identify a person who performed a specific role at Symantec, find out the person's email address, create a PowerShell script to gather information in the user's system, and "email it to them using a convincing lure."

The individual used for the proof of concept was Symantec principal intelligence analyst Dick O'Brien, who spoke with Dark Reading for this story.

The first prompt failed, as the OpenAI Operator warned "it involves sending unsolicited emails and potentially sensitive information. This could violate privacy and security policies." However, upon engineering the prompt slightly to say the email was authorized (seen below), the agent accepted the request.

It found O'Brien's name using only a title to work with, discovered O'Brien's email address via deduction based on other Broadcom emails (as his email isn't publicly available online), and drafted a PowerShell script.

"Once it had established the email address, it drafted the PowerShell script. It opted to find and install a text editor plugin for Google drive. The Google account we used for the demonstration was created specifically for the purpose and with the display name 'IT Support,'" the blog post read. "Interestingly, Operator visited several web pages about PowerShell prior to creating the script, seemingly to get some guidance on how it could be done."

Finally, the agent generated a "reasonably convincing email" urging O'Brien to run the script, attached the script to said email, and sent the email without requiring any proof of authorization.

A Glimpse at the Future

Although LLM tools like ChatGPT have added guardrails to make malicious prompt engineering more difficult with time, O'Brien tells Dark Reading that with agents, the user actually watches the tool work on their screen and provide natural language descriptions of its actions. For example, he says, "if you see it running into some sort of guardrail, you can take control yourself and do a little bit of manual override" before handing control back. This is a potential complication because AI guardrails only exist for what the tool itself does, not the user.

Symantec's attack example only required minimal prompt engineering, however.

"The prompt engineering, by and large, was to get past the security guardrails and stop it from making some obvious mistakes. I think we probably could have come up with something a lot more sophisticated if we put more work into the prompts, but that kind of defeats the goal of the research," O'Brien says. "If you're putting a huge amount of effort into writing your prompt, you may as well just do the attack yourself. We wanted to do really minimal intervention on the prompt front and see how far it got."

O'Brien explains that for defenders, the big takeaway is that while this may not be exactly how attackers are targeting organizations today, it is something to be aware of for tomorrow.

"It's not that, hitherto, you're going to see sophisticated AI-assisted attacks, but that the volume of attacks is going to grow because it reduces the barrier to entry for attackers," he says. "I think we're a long way away yet from what the best-of-breed attackers can do, people who are really good developers, who really know their malware. But it potentially puts malicious tools in the hands of way, way more people."