The Telegram Group Jailbreaking Grok ⇥ 404media.co

Emanuel Maiberg, of 404 Media, has been following a Telegram group in which members find workarounds to guardrails in generative A.I. products. Instead of finding interesting exploits to do clever things, though, the loopholes are being harnessed mostly to harass, abuse, and bully women. It is pretty revolting.

Maiberg:

It’s good and correct for people to be shocked and upset when they wake up one morning and see that their X feed is flooded with AI-generated images of minors in bikinis, but what is clear to me from following this Telegram community for a couple of years now is that nonconsensual sexual images of real people, including minors, is the cost of doing business with AI image generators. Some companies do a better job of preventing this abuse than others, but judging by the exploits I see on Telegram, when it comes to Grok, this problem will get a lot worse before it gets better.

It is clear Maiberg is just as disgusted with this as any person should be, so I am not trying to um, actually this, but I am not sure treating it as a “cost of doing business” is correct. The design and capabilities of these products matters immensely and, by permitting a relatively open-ended environment, xAI allows for experimentation to find its weak points. This is true of any generative A.I. product with a text box as its input. (As opposed to, say, a generative object removal tool in an image editor.) The degree of impact may also vary depending on the quality or style of the image — though, personally, I would still be alarmed if someone were harassing me with images even if they were cartoons.

Matt Burgess and Maddy Varner, Wired:

Unlike on X, where Grok’s output is public by default, images and videos created on the Grok app or website using its Imagine model are not shared openly. If a user has shared an Imagine URL, though, it may be visible to anyone. A cache of around 1,200 Imagine links, plus a WIRED review of those either indexed by Google or shared on a deepfake porn forum, shows disturbing sexual videos that are vastly more explicit than images created by Grok on X.

Caroline Haskins, Wired:

Over the past two years, Apple and Google removed a number of “nudify” and AI image-generation apps after investigations by the BBC and 404 Media found they were being advertised or used to effectively turn ordinary photos into explicit images of women without their consent.

But at the time of publication, both the X app and the stand-alone Grok app remain available in both app stores. Apple, Google, and X did not respond to requests for comment. Grok is operated by Musk’s multibillion-dollar artificial intelligence startup xAI, which also did not respond to questions from WIRED. In a public statement published on January 3, X said that it takes action against illegal content on its platform, including CSAM. “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content,” the company warned.

X’s threat of “consequences” would be more convincing if it had functional site moderation generally.

Apps have been kicked off the App Store for far less than what X is today. Removing it — and the rest of xAI’s apps — would be a good start, but we should not expect private companies to do the job of law enforcement and regulators. There is a good case for banning X as long as it continues to permit this poorly-moderated image generator. People should be criminally charged, too.