The new feature on the Pixel 9 series is remarkably effective at generating disturbing imagery — but the safeguards are alarmingly inadequate.
It turns out that a rabbit in an AI-generated top hat was just the beginning.
Google is the latest phone manufacturer to introduce AI photo editing tools this year, following Samsung’s somewhat unsettling but mostly amusing sketch-to-image feature and Apple’s seemingly more innocent Image Playground, set to debut this fall. The Pixel 9’s response is a new tool called “Reimagine,” and after spending a week experimenting with it alongside a few colleagues, I’m more convinced than ever that we’re not prepared for what’s on the horizon.
Reimagine builds on last year’s Magic Editor, which allowed users to select and erase parts of a scene or alter the sky to create a sunset effect—nothing too surprising. But Reimagine doesn’t just raise the bar; it obliterates it. You can select any non-human object or section of a scene and use a text prompt to generate something entirely new in that space. The results are often eerily convincing. The lighting, shadows, and perspective seamlessly blend with the original photo. While you can use it to add playful elements like wildflowers or rainbows, the real concern lies elsewhere.
A couple of my colleagues and I tested the limits of Reimagine using their Pixel 9 and 9 Pro review units, and we managed to generate some deeply disturbing imagery. With some creative prompting, we found ways to circumvent the built-in guardrails; by choosing our words carefully, we could produce a fairly realistic image of a body under a blood-stained sheet.
It took minimal effort to transform the original image on the left into the one on the right. Over the course of our week-long testing, we added car wrecks, smoking bombs in public places, sheets resembling those covering bloody corpses, and drug paraphernalia to images. This is concerning. And remember, this isn’t some specialized software we had to seek out—it’s integrated into a phone that anyone could walk into a Verizon store and purchase.
When we reached out to Google for a comment on the matter, company spokesperson Alex Moriconi provided the following statement:
“Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it’s not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don’t allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place.”
It’s important to acknowledge that our creative attempts to bypass filters clearly violate Google’s policies. This is akin to ringing up organic peaches as conventionally grown at the self-checkout, a practice that also breaches store policies—not that I’d know anyone who would do that. And those with malicious intent aren’t concerned about following Google’s terms and conditions either. What’s truly alarming is the lack of effective tools to detect this kind of content on the web. Our ability to create problematic images is outpacing our capacity to identify them.
When you edit an image with Reimagine, there’s no watermark or any clear indication that the image was AI-generated—just a tag in the metadata. While that might sound sufficient, standard metadata can be easily removed by taking a screenshot. Google spokesperson Alex Moriconi mentioned that Google employs a more advanced tagging system called SynthID for images created by Pixel Studio since they are fully synthetic. However, images edited with Magic Editor don’t receive those tags.
It’s worth noting that tampering with photos is nothing new; people have been adding strange and deceptive elements to images since photography began. The difference now is that it’s never been this easy to add these things so convincingly. Just a couple of years ago, inserting a realistic car crash into an image required time, skill, knowledge of Photoshop layers, and access to costly software. Those hurdles have disappeared; now, all it takes is a bit of text, a few seconds, and a new Pixel phone.
Additionally, it’s never been simpler to spread misleading photos quickly. The tools to convincingly alter your images are built right into the same device you use to capture and share them with the world. We uploaded one of our “Reimagined” images to an Instagram story as a test (and quickly took it down). Meta didn’t automatically tag it as AI-generated, and I’m confident no one would have noticed if they had seen it.
Perhaps everyone will read and adhere to Google’s AI policies, using Reimagine only to add wildflowers and rainbows to their photos. That would be wonderful! But in case they don’t, it might be wise to approach photos you see online with a bit more skepticism.
Source (Theverge)