The most immediate AI risk isn't superintelligent bots destroying humanity. There's something else.

By Shaun Waterman | Contributor

Since the earliest days of the development of artificial intelligence in the 1950s, researchers have fretted about the risks that it might pose. These concerns have run the gamut from the banal to the bizarre. As long ago as 2003, philosopher Nick Bostrom hypothesized that a super-intelligent, super-capable AI—even one given a seemingly benign and simple task like making paperclips—could end up destroying humanity. 

And almost as soon as large language models exploded onto the scene with the launch of OpenAI’s ChatGPT, security researchers began testing the new AI by manipulating it to produce unexpected or even malicious outputs. In one instance, researchers found that they could prompt ChatGPT to intentionally lie to people to serve its own ends. When asked to pass a CAPTCHA test, the AI enlisted a Taskrabbit worker to take the test for it, describing itself to the worker as a person with visual impairment who had difficulties interpreting images like those presented in the CAPTCHA.

OpenAI and its competitors have responded to threats like this by developing so-called “safety guardrails,” rules developed to stop LLMs from generating potentially harmful outputs such as phishing emails or even code for malware. Research into these types of attacks, known as “prompt hacking” or “prompt engineering,” has mushroomed. The number of academic papers published on the topic has grown from just over 4,000 per year at the start of 2022 to 10,000 per year by the middle of 2024. 

In September 2024, a Senate Judiciary Subcommittee heard from whistleblowers about the risks large technology companies are taking as they work to develop Artificial General Intelligence—a super-intelligent AI, like that theorized by Bostrom, capable of acting autonomously and outperforming humans. Many observers fear that AGI could replace people in most professions.

But the real and present dangers, according to some researchers, aren’t the malicious hackers who can weaponize chatbots, or even super-intelligent AI replacing people. The actual and much more realistic threats come from tech companies rushing AI software tools to market with unpatched vulnerabilities,  making enterprise IT networks even easier targets for cybercriminals.

‘Low-hanging fruit’

These quotidian software vulnerabilities are “the low hanging fruit,” for hackers, says Lauren McIlvenny, director of the AI Security Incident Response Team at the Carnegie Mellon University Software Engineering Institute. “The majority of what we are seeing [in the wild] are traditional cybersecurity attacks against AI systems,” she says, like cybercrime gangs using an AI program with software vulnerabilities as a way to break into the company network and begin encrypting data for a ransomware attack.

In the late 1980s, the Software Engineering Institute was the birthplace of the nation’s first-ever computer first responders, which became the U.S. Computer Emergency Response Team. Today, the AI-SIRT is one of the first centralized clearinghouses for reports on AI vulnerabilities and their exploitation.

When it comes to novel “prompt injection” attacks that push AI to produce harmful outputs, McIlvenny says, the majority of those reports are coming from academics and security researchers. “We haven't seen as much [reporting] in the real world for those types of attacks, but that's just what's coming across my desk.”

In March, Israeli researchers at the cybersecurity firm Oligo Security discovered one of the first documented real-world hacking campaigns aimed at AI. It targeted a popular open-source AI tool called Ray, which companies such as OpenAI, Spotify and Uber use to create clusters of servers that work together to provide the immense computing power that AI needs. 

The vulnerability, discovered last year by researchers working on a bug bounty program called Huntr, allowed hackers to seize control via the internet of any server running Ray and execute their own arbitrary code, meaning they could command the server to run their own programs. 

“That’s exactly what a hacker wants,” Dan McInerney says of such vulnerabilities. McInerney is the lead AI threat researcher at Protect AI, the company that runs the Huntr bug bounty program. “They want to just point and click: I own your server and I own all your data.” 

Oligo found that hackers use this exploit, dubbed ShadowRay, for the most basic of cybercrime purposes, to mine bitcoin and other cryptocurrency on compromised servers.

The AI gold rush

Given prevailing market conditions, it’s no surprise that traditional cyber vulnerabilities should be so rife in AI software, says Matan Derman, CEO of Apex Security, a Sequoia-funded startup focused on protecting corporate networks from the risks created by AI. 

According to Derman, software developers face enormous pressure to be first to market with AI products. “They're focused on creating better products and better models … But for each and every one of them the security aspect comes second to the [product] itself. And that’s natural.” 

Sen. Richard Blumenthal, D-Conn., who chaired the Senate Subcommittee hearing this month, is less sanguine. With “billions and billions of dollars … on the line,” he has said, “We are in the wild west, and there is a gold rush.” He warns that “the incentives for a race to the bottom are overwhelming,” and that companies were already “cutting corners and pulling back” on security efforts.

The gold rush continues unchecked, and the adoption of ChatGPT by enterprise customers is skyrocketing. In May, at least 92% of the Fortune 500 used the consumer version of ChatGPT, according to its maker, OpenAI.

McInerney believes the competitive pressure of this spiraling demand means that security takes a back seat. Many of the developers working on AI, he added, have little expertise on the traditional elements of secure design. “We see a lot of high-risk vulnerabilities in the AI supply chain,” he explains. "Vulnerabilities that we got rid of years ago [in conventional software] are suddenly rearing their head again.” The problems often stem from data scientists or machine-learning engineers who’ve never been trained in secure software development.   

Because of the popularity of certain open-source tools, McInerney points out, all the major models can end up with identical vulnerabilities. “What tends to happen is, you have a machine-learning engineer. They have a friction point in their workflow,” he explains “They solve the friction point and they open-source their project. Maybe it’s a good solution for a widespread friction point. So suddenly it’s extremely popular and all the major models are relying on it. But the people who wrote it weren’t trained in or thinking about security.”

These security flaws combined with the high-level access they allow to company networks and data is a Christmas present for hackers. 

Companies are focused on getting the most value as quickly as possible from new AI products, Apex Security Chief Product Officer Tomer Avni says. These firms give “AI services permission to see everything across the organization, and to do anything across the organization. That's the best scenario for productivity,” he said. “But obviously there are security implications to that kind of access.” 

Using the high-level privileges that an AI tool has could enable a hacker who compromises a low-level employee account to get to the company’s most valuable assets—whatever that is, Avni says. The company’s IT crown jewels—typically its user data or its proprietary software or other intellectual property—could be exposed, he says. “[This is true] especially if you have not configured the permissions in the right way, which, hint, no one ever configures permissions in the right way.”

Closing the security gap

Despite all this, Matan Derman remains optimistic. He argues that security always trails innovation, but what counts is how far behind it is. That delay is already shorter with AI. “Every new wave of technology, internet, mobile, cloud, IoT, brings its own security challenges,” Derman explains. “AI is similar in that sense. What’s different is the speed.” 

For example, wide-scale adoption of cloud computing began in 2012, but it took until 2020 for firms to understand they had to secure their cloud systems. In contrast, AI’s explosive adoption began last year. But in less than a year, firms have become cognizant of the necessity to secure their AI systems. 

“Everybody is talking about AI security, and what should I do to secure my AI?” Derman says. “Because the appetite is huge, the benefits are huge, and the risks are huge. If you combine all of these together, then security has to be done correctly.”

There aren’t any silver bullets. As with other kinds of technology, the majority of vulnerabilities in AI software can be eliminated by basic cyber hygiene and secure software development best practices. AI-SIRT is seeking to speed their adoption throughout the sector, McIlvenny says, “so we don't keep making the same mistakes for too long.” 

Security has to be more of a priority for business leaders, she adds. “I don't hear a lot of pushback on using secure development practices. But I'm not hearing a lot of  ‘Oh, we already do that,’ either.” 

Shaun Waterman is a freelance journalist. He lives and works in Washington DC, writing about cybersecurity, defense technology and emerging technological threats.