The scale of AI-generated media can be hard to grasp.
Starling Lab, a research collaboration from Stanford University and the University of Southern California, estimates that it took until 1975—149 years after the invention of the camera—for humanity to create 1.5 billion images. It took generative AI just 18 months to do the same. And AI didn’t stop there.
This spring, Google announced at I/O that its tools had been used to create more than 100 billion AI images and videos in just a couple of years, and Google is far from the only source of AI content. The company coupled this staggering statistic with a slew of partnerships to expand the use of its SynthID watermarking technology, which can be used to label AI-generated content and potentially help people identify what’s real.
The company says SynthID is robust enough to survive edits, and we’re about to see that tested as OpenAI, Runway, Nvidia, and others begin using it. Can invisible watermarks solve the AI content conundrum?
An indelible-ish mark
There are currently two approaches to labeling AI content: invisible watermarks like SynthID and metadata schemas like the Coalition for Content Provenance and Authenticity (C2PA). Google uses both to label its AI content, but C2PA is not intended to hide in plain sight like a watermark. It’s cryptographically secure, so you can’t fake it, but C2PA is trivially easy to strip out. Simply editing and saving an image or taking a screenshot can remove C2PA.
Watermarks like SynthID are encoded in the pixels of an image or video or in the waveform of an audio clip. Content that gets passed around the Internet degrades from compression, resizing, and edits, but Google says SynthID should still be present even in well-worn memes. Google DeepMind scientist Pushmeet Kohli explained to Ars that the team put a great deal of work into ensuring SynthID was a durable way to label AI content.
“Through the whole development process, we sort of assumed that a technology like this will be attacked,” said Kohli. “So we did a lot of research in making SynthID robust to different kinds of transformations. Whether people are adding some sort of filter or cropping the image, we used these transformations and made sure that the detector was robust against them.”
Google has been reluctant to provide much technical detail on SynthID’s functionality beyond the original paper, so it’s worth testing these claims as the technology expands across the AI landscape. But who has the time to wait for AI images to degrade naturally as they get passed around the Internet?
I used the Python Pillow library to simulate data loss from repeated sharing and downloading at a vastly accelerated rate. The Python script chooses random compression and resizing values within set ranges to apply to the test image, then uses the output image as the basis for the next iteration.
This test focused on two types of AI images: one created entirely from scratch by the model and an original photo edited by AI. Both images generated by Nano Banana Pro include the SynthID watermark, and no amount of metadata editing can hide it.
I started with the images above for the original and edited tests. We can probably all agree that neither one is real—politics is not metal, and my dog is not part dragon (take my word for it). I ran these images through the image-crushing machine hundreds of times, turning the crisp originals into barely recognizable blobs. Every 50 generations, I also created cropped versions of the images (without reencoding), removing potential SynthID pixels to further weaken the detection.
After 300 generations of simulated sharing, we’re left with these full-frame images:
And look at that—the SynthID watermark still works on both of them. You can check for yourself by uploading the images to Gemini and asking for a SynthID check. You can even take a screenshot of the full image, and the system will still label it as SynthID because the special pixels transfer over to the new file.
Because SynthID pixels are spread throughout the image, they’re also resistant to cropping… at least to a point. This is where we finally find the limits of SynthID. After 300 compression generations, removing a few pixels from the border of our test images finally broke SynthID (both edited and fully AI). These versions (see below) have been cropped by 20 percent, but that’s enough to render SynthID undetectable. A larger 50 percent crop can break SynthID a bit earlier, at around 250 image compression iterations.
These images may be ugly, but there’s nothing (other than the content) to prove they’re AI.
Wary of watermarks
Based on my testing, SynthID can gradually degrade, but the images are uselessly blurry by the time that happens. So have we solved the problem of AI disinformation? Not so fast. It’s impressive that SynthID can survive aggressive edits and data loss, but AI watermarks have some notable issues.
SynthID is not the only AI watermarking technology. Meta released its Content Seal AI watermark recently, but Reuters found that simply cropping the images a bit would often eliminate the watermark. Just because SynthID seems solid right now doesn’t mean all attempts to invisibly label AI content will work.
Google also doesn’t claim that SynthID is invulnerable—the original paper on the technology notes that the watermark is not intended to withstand adversarial attacks. It’s entirely possible that someone with enough time and incentive will find a way to bypass SynthID. Some people have already claimed to have successfully hacked SynthID, but we have been unable to confirm that through testing. Google’s Kohli also says his team has been unable to replicate the supposed workarounds.
Still, security is never perfect. If SynthID is successfully bypassed, which seems plausible if it continues to grow in popularity, all those billions of AI images and videos could be unlabeled in a snap. Google may find itself in a cat-and-mouse game as it updates the standard to block those attacks. Google is quick to point out that SynthID is not a silver bullet for digital providence. But SynthID is becoming a much bigger target. Between Google and OpenAI, most of the consumer AI space is now using this technology.
The team behind SynthID has plans in place to combat hacks, but Kohli says Google needs to keep much of that to itself. “We can, of course, change the watermark,” said Kohli. “There are a lot of things at our disposal. I cannot publicly disclose many of these mitigations because they help us make sure that the whole system is protected.”
At least some of the security model for SynthID becomes apparent when you try to use it. Google doesn’t want to give people an easy path to attack SynthID, so access is intentionally limited. If you see a suspicious AI image, you have to ask Gemini to call the verifier—there’s no API or publicly accessible SynthID detector webpage.
That’s not the only limit, either. To prevent people from using successive AI checks to tune a bypass workflow, Google limits SynthID to “approximately 10 image checks” per day. In my testing, the verifier will lock you out even faster if you upload too many similar-looking images. If you’re suspicious that an image may be generated by AI more than a few times on a given day, you may be out of luck. At a time when politicians are gleefully using AI to defame and attack opponents, verifying the truth shouldn’t come with a cooldown period.
Even if you aren’t locked out of AI detection tools, which one are you supposed to use? As Google rolls out the technology to other firms, watermarks are quickly becoming fragmented. While companies like OpenAI and Runway have chosen to use the underlying SynthID technology to label AI content, the actual watermarks are different. For instance, Google’s detector doesn’t recognize OpenAI’s watermark, and OpenAI’s detector doesn’t recognize Google’s watermark. So not only is access to SynthID restricted, but you may run an AI image through multiple detectors and still find nothing because it was generated by a different system you don’t even know about.
A Google spokesperson says the team is aware that this experience is suboptimal, and it has vague plans to improve it. “We are actively collaborating with industry partners to make verification more accessible and unified. Our vision is to create a more interoperable ecosystem that allows for a more seamless experience across different platforms,” they said.
Out of the bottle
Maybe the usability issues with watermarks can be solved. But at best, it will be only one element of how we determine what’s real. Watermarks in AI will never be the best way for one simple reason: There will always be ways to create AI content without any labeling.
Countless companies offer image and video generation, and it’s unlikely they will all agree to use watermarking technology or even metadata labeling. More importantly, you don’t have to rely on Big Tech’s centralized systems to generate AI slop.
“I think the bigger problem here is not just the images that are being produced by the Googles and the OpenAIs of the world,” said Adam Rose, a fellow and senior advisor at Starling Lab. “The problem is people who can run their own models, who can do things on their own computers.”
Even today, there are plenty of open models that generate images with no labeling whatsoever. These AI models can be shared and improved upon indefinitely, even if most of the major AI players adopt a multifaceted approach to labeling AI content. The generative genie is out of the bottle, and there’s no stuffing it back in.
Since there will always be unlabeled AI content, you can’t count on watermarks to tell you what’s true. As AI labeling becomes more common, people may think an image or video that lacks one is automatically legit. That’s a dangerous but understandable mistake for someone to make. While the existence of invisible watermarks might give people some sense of security as they explore an increasingly distorted Internet, they aren’t a reliable way to know what’s true.
“The future that we live in will increasingly be a world of unlimited content,” said Rose. “It’s economics 101, supply and demand. Supply goes through the roof, and the value of content is going to continue to crash. But increasingly, there is going to be a demand for authenticated content, a way to prove that something is what it purports to be.”
The scale of AI content alone will make it impossible to label all of it. Google’s 100 billion SynthID images and videos are just the tip of an iceberg that has already eclipsed authentic content on the Internet. It may be smarter to focus on verifying truth rather than outing falsehoods.
“There’s a limit to how much content can be created with actual physical cameras in the world,” explained veteran photojournalist and Starling fellow Mike Caronna. “There’s no limit to how much synthetic content can be made. That’s why we want to pivot to what’s scarce, and what’s scarce is high-value, truthful information. If we can secure that and prove that it’s authentic, we can defend against the liar’s dividend.”
Maybe you haven’t heard that term, but you’ve experienced it. The liar’s dividend is a rhetorical strategy wherein someone is confronted with evidence of wrongdoing, and they claim the facts are manipulated. Or they simply shout “fake news.” It’s been a depressingly effective tactic for public figures to escape accountability, and it may become even more powerful as AI content proliferates: “That’s not me—it’s AI!”
That brings us back to C2PA. While it’s easy to scrub digital files, C2PA may actually offer a better way to determine what’s real. This and similar technologies are essentially tamper-evident seals for digital files. Cryptographically verifiable metadata can prove all that comparatively scarce real content is, well, real. That may be far more useful than looking for proof that something is AI-generated in a sea of slop with unknown origins.
The problem is that C2PA is still rare. Few phones and cameras create these digital fingerprints, and most websites and apps don’t expose C2PA data in images (you can look at C2PA data with various online tools, though). Google is actually ahead of the curve here. The company’s Pixel phones are the only mainstream cameras with deep C2PA integration. Both images and videos captured by devices like the Pixel 10 include cryptographically protected C2PA metadata that reveals when and how an image was captured. It also lists whether AI was used to alter the image.
In the future, we may have no choice but to treat content that lacks verifiable provenance data as suspect. That’s just the consequence of living in a world in which typing a few words can deliver an endless stream of misinformation. Labeling AI content is a good policy for tech firms, but it won’t save us.
Ryan Whitwam is a senior technology reporter at Ars Technica, covering the ways Google, AI, and mobile technology continue to change the world. Over his 20-year career, he’s written for Android Police, ExtremeTech, Wirecutter, NY Times, and more. He has reviewed more phones than most people will ever own. You can follow him on Bluesky, where you will see photos of his dozens of mechanical keyboards.