> Because the photos and videos are analyzed on your child's device, Apple doesn't receive an indication that nudity was detected and doesn't get access to the photos or videos as a result.
Apropos of nothing, I really don't love the construction of this reassurance. This is not actually a reasoned-through guarantee, it's just two things that happen to be true at the same time. But the latter could change at any point, on-device processing does not preclude the device notifying Apple about what it saw.
The backdrop to this is a story from 2023 in which Apple attempted to do this and faced backlash. This received significant coverage worldwide at the time.
But why would they want to be notified. Seems like this is the best case scenario for them. They’re taking action against the problem and at the same time cannot be approached to break their clients privacy. The only thing I can imagine is a government entity making them do it.
Yeah, it's clear to see that they'd want to get in front of something like that happening again. My point though is that the pull quote I highlighted is a flimsy reassurance because "it all happens on the device" does not at all prevent "Apple knowing about it," yet the sentence is constructed in a way that (tricks?) people into thinking it would.
That’s true of of every privacy guarantee made by all software. For example: the only guarantee we have that Tor Browser isn’t phoning home usage statistics is 1) that people are inspecting its outbound network traffic with a magnifying glass, and it isn’t; 2) and so over time their guarantee has been accepted as trustworthy. So, no curing assurance can ever be made to the point you’re concerned about, other than to recommend not using software if one assigns high priority to this threat model. So, then, as someone who does, it would be useful to understand your viewpoint in a more concrete/applied sense. I have a handful of questions:
How would you rewrite Apple’s copy to reflect this universal threat? Would you advise GrapheneOS to adopt similar copy (since the concern is equally applicable, what with five nines of users not self-compiling from inspected source) to chip away at Apple’s marketing here? Is your concern restricted to ‘nudity-encountered’ metrics (as in the topic of this post) or is it generic to all ‘xyz-encountered’ metrics, or to all metrics, or..?
You misunderstood and it's really simple. Implying that on-device scanning makes it impossible for them to access any information is misleading. Just drop that faulty reasoning because it creates a dangerous misunderstanding of how technology works.
To illustrate: Because I wrote this comment, the sun is going to rise again tomorrow.
Sure, what I meant is "without actually taking a stance on what the article is about and opinions on Apple's feature aside, I dislike the particular placation presented within"
It must be a tricky problem to balance. On the one hand, you as Google want people to create 30 seconds of video per month with your cool Omni, Flow, Gemini, etc. tools.
On the other hand, as soon as people share those things on the logical platform for sharing videos, they'll be branded with the scarlet letter.
I wonder what Google is thinking - that people won't mind? That it won't matter? That Omni is just marketing and they don't actually want people to use it?
Maybe they're just going for "disclosure" as in people understanding it's AI, and hopefully mitigating fake news. Don't know if it impacts monetization?
If the video is entertainig I doubt people will mind it's AI. Let's wait and see.
Maybe I'm weird, but I believe in the theory that (all else equal) it's good for business to minimize how much your users hate your product/service.
In other words, users dislike the feeling of not knowing whether things are ads. I can't see any real downside to labeling them, so you're better off doing it so you don't drive users away.
Google is not a monolith. For all intents and purposes YouTube might as well be a totally different company than Deepmind. Everyone in there own respective google fiefdom is trying to maximize their own metrics.
Why are they saying to not distribute on YouTube they just want to give an indicator. Same with labeling if a video is an AD. I find some of the obvious AI content to be funny or informative .
I mean, here's what I don't get: why does YouTube care? We're already uploading an entire human lifespan worth of videos to YouTube every day, do they really benefit from more content? Or is this content somehow inherently more monetizable than what people are already uploading?
And the most critical thing about JPG is that the decoding is deterministic. Who's to say this fancy new PICO thing doesn't produce different pixels in a year when the algorithm improves, or the local model changes, etc.
Imo, generative AI and its derivatives should be completely shunned as image/video encoders. They are simply an inappropriate tool for the job. And I say that as an AIpilled token addict.
It would be like saying hey check out my amazing new text compression algorithm, 97x better than LZMA, then you look at the encoded file and it says "generate a romance story between two characters named Romeo and Juliet"
Personally, having to buy the barely-changed newest yearly edition of half a dozen $300 textbooks per semester of undergrad totally radicalized my view on copyright.
> The code it generated was awful. The kind of garbage that people who don’t know any better would ship: it looked right and it worked. But it was instantly a maintenance dead end.
In the Tailwind thread the other day I was explicitly told that the intended experience of many frameworks is "write-only code" so maybe this is just the way of the future that we have to learn to embrace. Don't worry how it's all hooked up, if it works it works and if it stops working tell the AI to fix it.
It's kind of liberating I guess. I'm not sure if I've reached AI nirvana on accepting this yet, but I do think that moment is close.
The problem is it’s impossibly hard to test all the edge cases
Which is probably why so many random buttons in microsoft/apple/spotify just stop working once you get off the beaten path or load the app in some state which is slightly off base
The number of edge cases in a software is not fixed at all. One of the largest markers of competence in software development is being able to keep them at minimum, and LLMs tend to make that number higher than humanely possible.
> Turns out the humans weren't so bad after all...
The people pushing AI _over_ humans never thought they were. They just don't care about 'good' or 'bad', only 'time-to-market'. A bad app making money is better than a good one that isn't deployed yet. And who cares about anything past the end of the quarter? That's the next guy's problem.
I'm wondering if companies are 'diverting' engineering resources from core products to AI products with the view that the former are legacy. Kind of two sides of the same coin though.
I'm pretty you wouldn't want the same for code that runs healthcare, banks or transport. Only useless shitty web projects could embrace what's you're saying. And no there's no "Claude review the code and improve it" magical formula
I work in the health software space and there are tons of internal tools which aren't production code that can benefit massively from throwaway "write-only code". Putting a web UI on top of a management CLI tool so support ops can run things without needing an oncall engineer can be a huge win. I recently built a testing UI that doubles a demo-scenario-setup tool. Is it well-engineered? Who cares - it pokes the right things into the database and runs the right backend tasks, and has helped me catch and fix dozens of real bugs in the UIs that customers see.
There is an enormous untapped market for crappy low-effort apps which previously weren't worth the time - but with the effort so low put together a simple dashboard or one-off tool it becomes much more attractive.
> internal tools which aren't production code that can benefit massively from throwaway "write-only code".
First of all internal tools just prove my point. Second can't wait to hear a story of a health care production database blown away because someone was playing with generated tools that "pokes the right things".
We NEED end user experiences that don't suck, and don't keep getting shittier like now, not being able to use a cli for internal tools, it's a skill issue, not everythint needs a shitty ui that taps into an os system calls and blows away as soon as the cli responds with something unexpected
You still have to do the engineering parts of software engineering, you can't just turn off your brain and vibe. Just like you can't with hand-written code. I've seen what an intern with access to prod DB can do.
For example, the testing tool I built explicitly doesn't work in production environment. That was part of my design spec and I manually verified the code and behavior.
Easy, have Claude review the code, tell it to be critical and that it needs to be easier to understand, follow Clean Code, SOLID principles and best practices. Lie to it, say you got this from a Junior developer, or "review it as if you were a Staff Level Engineer reviewing Junior code" the models can write better code, just nobody tells them to.
Clean Code has its really "meh" areas, but the core idea and spirit of it is sound, heck Python's best guide is PEP-8 if you follow that, it forces you to write much better Python code.
In terms of "junior dev following" it would be the model trying to think and write it as a Senior or Staff Level engineer would.
Code review is the main thing I use LLMs for. I have found it to be remarkably candid when you tell it the code came from another LLM (even name it). I was running Kimi K2.6 Q4 locally, seeing if it could SIMD a bit-matrix transpose function, and it was slow enough that I would paste its thinking into Gemini every few minutes. Gemini was savage.
This is it. I've had a similar experience in just playing around I asked it to clean up some code it wrote to increase maintainability and readability by humans. After a few iterations it had generated quite solid code. It also broke the code a couple of times along the way. But it does get me thinking that these pipelines with agents doing specific tasks makes a lot of sense. One to design and architect, one to implement, one to clean, one to review, one to test (actually there's probably a bunch of different agents for testing -- testing perf/power, that it matches the requirements/spec, matches the design, is readable/maintainable, etc...).
I built GuardRails after some frustrations with Beads which I love, and this whole exchange made me realize, because I have "gates" after tasks, I could add a "Review the code" type of gate, and probably get insanely better output, I already get reasonably good output because I spec out the requirements beforehand, that's the other thing, if you can tell the LLM HOW to build before it does, you will have better output.
Because language models don’t think before doing, they think by doing.
Maybe a more idealized training set could improve things, but at least for today’s SOTA, you have to get the shitty first draft out and then improve it.
Harnessing makes a difference, but it’s only shuffling around when and where the tokens get generated. It can trade being slower by doing a hidden first draft and only showing the output after doing a self review. But the models still need to generate it all explicitly.
Why would it? It doesn't do anything with intention without being prompted. When you ask it to do something it's going to give you what seems like the most likely result, it isn't striving to give you the most correct result, those things just have some overlap.
I assume it would involve wasting a lot more tokens reasoning about this. It is known that GPT uses less tokens than Claude, but Claude uses them to reason about problems more, which is part of its "secret sauce" and why so many swear by Claude Code.
Even better, if you have access to multiple models, tell it you got the code from another AI agent.
I did an experiment on this a few weekends ago and Codex for example was a lot more adversarial and thorough in its review when given Claude-authored code compared to when given the same code with "I wrote this, can you review it?"
If it's within its context window, it will know you're lying, so either compact or start a new chat (don't do this on Claude, it dings your usage, always has).
Kind of wild that you have to tell an LLM things like "do it right" and "make the code maintainable" and "don't make mistakes". Shouldn't that be the default? I wouldn't accept a calculator application that got math wrong unless you pressed a button labeled "actually solve the problem."
> Kind of wild that you have to tell an LLM things like "do it right" and "make the code maintainable" and "don't make mistakes". Shouldn't that be the default?
It's not the default, because the training data is full of unmaintainable code done wrong with mistakes. People literally complain that LLMs write too many tests or add comments.
If instead of "do it right", you give it specific actionable advice of how to right code, it does surprisingly well. Newer frontier models also do a great job of mimicking the style and rigor of the surrounding codebase without prompting, if you're working in an established codebase, for better or worse.
The default isn't necessarily what ever you consider maintianable or do it right, which are ambiguous terms anyway.
You never wrote quick exploratory code? One off scripts? How is the Ai suppsed to know unless you tell it.
If you tell another person to write some code, how are they suppsed to know? If you have your boss come to you and ask you to write some code to do some data analysis are you going to spend weeks writing units tests and perfect abstractions? Or do it quick and get the data and result?
You forget that this all takes tokens from the model, so it has to be very stingy and whatever it comes up with "first" is what it goes with. I've seen people do the same as me, tell the model NOT TO GUESS but to do research first, which yields better output and saves time. Models today are better when they review the context directly, the focus shifted from it knowing everything in its training data to being able to dynamically learn new things and use that information in a meaningful way.
For example, I built up a programming language from scratch with Claude, it knows nuances about my languages syntax, and can write code in my language effectively. I did it mostly as a test. It definitely helped that my language is heavily mostly Python based.
I have been wondering recently that if the cost of just throwing everything out and building it from scratch again gets low enough, maybe maintainability becomes less of a priority? Can we just embrace the thing like those Zen carpenters who build wooden fire shrines do where they just accept that the thing will keep burning down and they make a discipline around getting really good at rebuilding it?
Granted, the load bearing thing here is whether we’re actually getting good at rebuilding up to any sort of standard of quality. Or if the tooling is even structurally capable of doing that rather than just introducing new baskets of problems with each build.
Hmm. I think extrapolating from the reddit people who say "I tried vibe coding an entire app from scratch and all I said was fix this and make no mistakes and it didn't work" is a bad data source and will give you the wrong intuition. Of course it won't work when you hold it like that. But put just a tiny bit of knowledge and guidance into the prompt and AI will nail it.
I didn't think this 6 months ago but today after what I've seen these models debug and accomplish in established, messy production monoliths, I'm fully convinced even the worst vibe coders are only a year or two away from being able to actually create something from scratch and have it not blow up 50 files in.
So I guess I take the totally opposite stance, today's AI is the worst AI will ever be at coding, and I believe the vested interests behind AI do not plan on making it any worse at this task, so...
Creators don't get compensation when people ad-block.
Creators don't get compensation when LLMs scrape.
It's totally, and completely, unambiguous. The internet just has collective brain damage from the grassroots morals of it being formed 30 years ago by teenagers. How surprising that a bunch of kids decided that the way to save the internet was to make it better for themselves, and worse for the people who make the internet the thing they love.
Some of us have grown up now, and realize the correct answer to save the internet was to not engage with ad supported content period.
There are ways to get paid without ads and you can do on-air reads like I said. adblockers don’t impact them. You also don’t have to play Google and YouTube’s games. I’m sorry folks are caught in that arm’s race between users and Google but Google has made browsing so miserable it’s just reality.
Adblocking is basic security now. I am not compromising on it. I say this as a “content creator”
You don’t need to get sarcastic with me over this.
Content creation comes in many forms. You can also promote things in your copy. People do it all the time. Adblockers aren’t going to somehow remove your words. People disclose their sponsorships at the top/bottom of their written content all the time and frequently use affiliate links.
> It's totally, and completely, unambiguous. The internet just has collective brain damage
The point that continues to be missed is that instead of taking downvotes as validation that people simply failed to comprehend the argument you're making (they didn't), you should take them as a check to reevaluate whether your conclusion is as unambiguous as you believe.
There is no way to reconcile an internet where the suckers who cannot figure out ad-block carry the overhead costs for those who do. It costs money to create the content you consume, it costs money to serve the content you consume. The internet is not some magical exemption from standard financial practice going back millennia. The cost is your burden, take it or leave it. But don't take it then do mental gymnastics about why it's not actually something you value while walking away with free, shifting the cost onto the next guy.
You are very convinced of your position and I don't think I'll un-convince you, but you have to realize that people with opinions different from yours are not "brain damaged so that's why I must be getting downvoted, and the more I'm downvoted the more correct I must be."
I'll leave a variant of your argument here for you to mull over, and consider whether the bathroom-breaker here is just as morally bankrupt as the online ad-blocker:
> There is no way to reconcile [over the air TV] where the suckers who cannot figure out [how to leave the room during ad breaks] carry the overhead costs for those who do. It costs money to create the content you consume, it costs money to serve the content you consume. [Broadcast television] is not some magical exemption from standard financial practice going back millennia. The cost is your burden, take it or leave it. But don't take it then do mental gymnastics about why it's not actually something you value while walking away with free,
There's an underappreciated comment in the other thread about SynthID and OpenAI [0] that captures what (IMO) the hacker ethos on this should be. We care about privacy, we should not accept tools that barcode our every digital move. (note that the counter of "well, they don't do that yet" is not particularly convincing)
Building a tool that tries (and probably fails) to remove the watermark (due to the arms race that large corporate machines will win) is tacitly accepting the barcode. The hacker ethos should be, first and foremost, to run open source models locally without relying on a corporation.
>due to the arms race that large corporate machines will win
Much like how the entirety of Hollywood, book publishers, academic publishers, and game developers have won against piracy despite being some of the largest corps on earth and dedicating untold billions to the issue over the past 30 years?
They didn’t win because of DRM. They won because of the regulations that grant a monopoly for a specific term in the form of a copyright. Society has recognized that incentivizing creative acts requires a temporary grant of monopoly to ensure the necessary scarcity to make money and recover the costs of creation. The real problem is Disney keeps expanding that time period so things never enter the public domain
This is again conflating at least two things and this is so prevalent in this context. Let us not conflate how annoying DRM:s are to us users that buy the things, with pirates thinking they somehow have a right to use any software without paying fairly for it. I would even go as far as to say that you pirates are the reason I have to have a DRM in the shit I bought and paid for.
> I would even go as far as to say that you pirates are the reason I have to have a DRM
I think this is largely an incorrect take. DRM is anti consumer, not anti piracy. In fact, it has done very little to deter actual piracy (and remember it only takes ONE person to break the DRM), while affecting some casual pirates and all legitimate users. In the process, they got rid of reselling stuff you own.
It's anticonsumer, not antipiracy, never forget that. It means something like this would have happened regardless of pirates.
They succesfully did away with 2nd hand markets and the concept of "owning" anything. So yes, I would imagine DRM would continue to exist without piracy.
I think so, because their main goal is to prevent unwanted use of the digital product -- to the detriment of end users -- in more ways than just piracy. In fact, they don't solve the piracy issue.
I am not sure how I am conflating two things, it would be helpful if you could expand or connect to my argument. Perhaps I am misunderstanding.
My argument is that the grant of monopoly is a regulatory decision and the real cause of "winning". No amount of DRM would confer the same benefit because the ability to bypass it through piracy would be totally legal with no economic or other consequences and so a robust cracking and distribution ecosystem would emerge. Thats a drastically different story than when napster gets shut down, and limewire gets shut down and pirate bay gets shut down every time it relaunches. Imagine a world where there is are 1000 pirate bays
Piracy is as easy now as it was pre-DRM. DRM is the digital equivalent to security screws on electronics, in that they’re a mechanism for lawyers to argue their client made an attempt despite being easily bypassed with a trivial amount of effort.
Exactly this. The real power is in the regulatory grant of a monopoly that comes with rights like the ability to sue for damages, issue take down notices etc. the DRM does allow restrictions on distribution because many people can’t be bothered to remove them, but more importantly the act of removing them is evidence of the intent to knowingly violate the copyright which might be harder to prove otherwise
And now that they're trying to push up the margins and the streaming ecosystem is fragmenting making everything into a series of bundles again, piracy is on the rise again.
They did win for a while because they stamped out 99% of piracy. In the early days of streaming it was legitimately difficult to argue for piracy. Streaming was just too convenient and too cheap.
But, they are greedy above all else. And so, we are once again seeing a resurgence of piracy. Large corporations seem to snatch defeat from the jaws of victory.
By the time you're building (or buying) the necessary highly esoteric and expensive ultracentrifuge setup I think you would be well outside the realm of "hobbyist" unless someone insists on the most unreasonably pedantic definition for the term.
Unless we're only considering final assembly. Just gotta get that weapons grade fissile material supplier lined up. That might or might not qualify as rich hobbyist territory depending on how high a price tag is permissible.
In theory, there's also direct laser-based isotope separation. It's a technology that is being actively suppressed, and that's one case where I very much in favor of that.
This subthread starts off with the argument that the big corps will never beat the little determined hackers, one of the founding myths of the early internet. And then every now and then a strong little branch of the argument runs up against an example and it becomes well sure, the little hobbyist hackers don't have anything there but that is because the big corps/gov/billionaires/whatever put so much into beating them.
I mean reading it all certainly sounds like the people on the little guy's side are overestimating the value of pluck, an observation Hollywood generally makes just before the heroes with pluck win for ever!
> And then every now and then a strong little branch of the argument runs up against an example and it becomes well sure, the little hobbyist hackers don't have anything there but that is because the big corps/gov/billionaires/whatever put so much into beating them.
It's almost never about the level of resources the organization puts in. The usual reason is that there isn't enough incentive to do it. What is a hobbyist going to do with a nuclear weapon? Why spend your time creating one if you, like the overwhelming majority of people, have no desire to blow up a city?
Preventing something that hardly anybody would be trying to do even if it wasn't being suppressed is a lot more practical than preventing something millions of people would do given the chance.
Yes. Winning against piracy doesn't mean you completely eliminate piracy. It means you scare enough people into not doing it and make it a bit harder to do for others.
Losing to piracy would see companies like Netflix and Spotify not thriving.
> It means you scare enough people into not doing it and make it a bit harder to do for others.
By which definition they utterly failed.
> Losing to piracy would see companies like Netflix and Spotify not thriving.
Not at all. Netflix and Spotify do well because they are a good value proposition for the average customer. Piracy is free at point of "purchase" but is (and always has been) expensive in terms of various sorts of overhead.
As long as enough people keep the pirate bays open, it will be there as an alternative when the services start their inevitable enshittification.
I for one do not enjoy the “Which service has the classic film I wanted to watch this week?” Nor having to switch services every time I want to see a new TV series.
We need (and have!) similar “free” alternatives to the watermarked generative services. Just like I hate the yellow dots on my printed images, I am not happy to have my creative assets (I do nothing nefarious) stained with SynthID.
> Winning against piracy doesn't mean you completely eliminate piracy.
But this is moving the goalposts. You can win against piracy either by making piracy less attractive or by making the paid offering more attractive. The first has utterly failed, piracy remains easy as a rule, and to the extent that they've succeeded it's not only disproportionately by doing the second thing, the DRM itself is a net negative because it has such a small effect on the ease of piracy while making the paid offering worse.
It doesn't make any sense at all. That's like saying browsing the internet with an ad blocker and other privacy tools is a tacit acceptance of tracking and ads, and that you should only visit websites that doesn't track or have ads.
It's already possible to lie with text. Pixels are pixels. If we can't blindly believe pixels to show the truth, we will be simply back to the pre-photography era which managed to have a concept of truth regardless.
For the umpteenth time, scale and ease of access and propagation matters.
A knife and a handgun aren’t comparable to a machine gun and a bomb. When you have equal access to all of those, the damage you can enact is exponentiated.
You could lie with text before, but it took effort and time and skill to do it convincingly. You could also lie with images but they took even more time and effort and skill, greatly limiting the pool of people who could do it and the possible damage.
When anyone anywhere can convincingly lie and have it do two laps around the world in a matter of minutes, the whole game changes.
It’s becoming very hard to believe that people making arguments like yours are doing so in good faith. Maybe you’re not even a person but a shill bot. That’s a very real and trivial possibility today, which is the whole point and illustrates the problem.
That’s a non-sequitur. This has nothing to do with “mainstream media”. Which, by the way, is not a singular entity you can blindly trust or distrust, don’t fall into that indoctrination trap. This is basic despotic citizen control: Get everyone to distrust those who hold them accountable; label them as irredeemable liars so you can do as you please.
Edit: Removed swipes while keeping the core argument.
You’re right. I should’ve done better and apologise. I have lightly edited the post above to keep the core point while removing the swipes.
I disagree with the irony point on the basis that none of us here (I assume) is a despot or in a position of power high enough to lie to the point of discrediting entire institutions to the masses. We can be victims of it, but not perpetrators. Also, nowhere have I accused anyone in particular of being a liar. Still, given that my ill chosen words will have no doubt clouded your understanding of the point—which is on me—I won’t hold that against you, I take responsibility.
I haven't heard anything about this from the mainstream media. What bothers me is the amount of misinformation being published on social media that people are taking as truth, but now with photorealistic photos and videos. It used to be you could reverse image search something and show that it came from a previous event, or find the original in the case of doctored images.
Now that anyone can manufacture convincing images out of whole cloth, it becomes impossible for the typical user to determine if what they're seeing is legitimate; this means not only that people can manufacture images to push a false narrative but also discredit legitimate images as falsified.
The 'mainstream media is bad' narrative is one manufactured almost entirely by right-wing influencers and politicians to create an environment where they can discredit any actual evidence that doesn't align with what they or their base want to hear. Once anyone with integrity is inherently suspicious, the narrative can be set by anyone to push their agenda, and because they've demonized actual news reporting they can point to any other outlet that counters their claims and say 'look, they're lying to your face, you can't trust them!'
> the main thing is tracking, ID verification, constant social credit-like ratings, etc. That's what is coming.
Do you mean things like masked armed federal agents demanding to see your proof of citizenship or they'll lock you up (or sometimes just locking you up anyway)? Is that what's coming? Is that what you're afraid of? Or is that fine if it only happens to people you don't agree with?
These talking points, as always, are fearmongering designed to trigger paranoia in people so that they'll go along with whatever the people in charge say will fix things. No one on the left has proposed any of these things, but people on the right are constantly saying that's whats coming because it's what helps them keep their base frightened and compliant, and you've fallen for it (or are part of it, I don't know).
> AI media generation is a red herring, the main thing is tracking, ID verification, constant social credit-like ratings, etc. That's what is coming.
It can be both things.
I worry about tracking and the post-truth world of everything fake and AI slop infiltrating everything. And like the person you're arguing with is saying, just because slop/fakery was possible before doesn't mean the widespread scale that AI and the internet have unlocked is of no concern.
It's the same walled garden principle. Benevolent Google will protect you from synthetic media by embedding invisible watermarks, from malicious apps by blocking sideloading, from porn by ID verification, etc etc. It's the same principle.
Are you opposed to people knowing that AI-generated content is AI-generated? Or are you just imagining that this is the first step towards 'someone' controlling your every move online somehow?
You know what would be a good counter to that? Don't use AI generation tools if they implement that. The argument that 'we should let AI generated media be indistinguishable from the truth because what if Google does something you don't like at some point' is pretty flimsy.
You and I did whenever we assumed the pixels were an upload of a digitized photo taken with an actual camera of, say, a historical event, or something in nature (e.g. some rare bird), a birthday party, etc.
The photo could be staged (e.g. Cottingley Fairies), it could be altered physically (like cutting or painting over, e.g. Stalin), it could be cropped intentionally to tell a different story (plenty of examples), and more recently it could be photoshopped, etc. All of these were possible, though harder than it is now, but let's not pretend we didn't "trust pixels". We all did, just as we trusted newspaper photographs. Now that era must come to a firm end, and I believe it's a tragedy.
Fair enough. While I would kind of wish AI could be reliably detected, deep down I know this is impossible and it would be pretty bad if we had, say, a prosecution that succeeded because "this 'provably-non-AI' photo places you at the scene of the crime" because only a few underground people know how to remove a watermark.
You raise an interesting point about the artificial generation of evidence used in court. In 1992, Michael Crichton wrote the book Rising Sun, which centers around the editing of security camera video footage to coverup a murder.
I also wonder if being able to prove that an image or video isn’t AI generated would lend credence to it, while in reality there are other methods to produce falsified video.
Well, you just have to convince the jury. The defense attorney will try to throw up all the reasons it could be falsified, but the prosecutor will say "All of that is unlikely - the defense attorney would like you to believe that farfetched story, but this is still a compelling piece of evidence." This is how it always is with any kind of testimony and evidence.
What stops someone from adding a watermark to an actually photographed (carefully framed?) picture to discredit it? There is no certainty either way, just suggestions from someone else about what the truth might be.
AI watermarks only give the illusion of maintaining the concept of truth. The government and corporations will still have access to un-watermarked models to destroy the truth with.
The concept of truth? A bit overblown don't you think? Because some guy can make a realistic looking fake videos that destroys the "concept" of truth? How?
Stalin had all the resources imaginables at his disposal.
Now Nancy, a tech-phobic waitress who has a grudge against her coworker can make up an entire scenario with one prompt and her colleagues might blindly believe her.
Let's not pretend they're the same thing.
Gen AI is inevitable. Watermarking is likely futile. But in my opinion it is still very important to discuss how, as a society, we're going to live in a post-truth world now that anybody can, IN SECONDS, not only fabricate a story but also spread it to thousands of people through their social media.
When that idea was originated, the advice was more like:
"Don't trust what you see on the Internet. Trust instead what you read in a reputable daily newspaper, or Peter Jennings or Tom Brokaw on the nightly news, or BBC World News."
Today, the Internet, especially the part which is not trustable, has nearly finished killing most of the "trustworthy" news sources, by outcompeting them for ad dollars - by being way better at targeting ads (e.g. Meta) and by scientifically perfecting addiction (e.g. TikTok). What remains is mostly controlled by governments and has far from a perfect record of being fact-based and impartial.[1] There are a ton of independent people out there in good faith posting facts on the Internet, but we just agreed that we shouldn't trust what we see on the Internet.
So doesn't this become "Don't trust anything"? And doesn't that, in practice, get implemented as "Don't trust anything that challenges what you believe to be true"? This feels like a really, really bad change to our society - and I'd argue it's already completely happened.
This isn't just ads, trust in the mainstream media, itself, is very low [1], deservedly so in my opinion. The continuous lies by omission, the outright incorrect headlines/articles that they edit after a day, the lock-step messaging, alignment, and avoidance of topics, pushed by their respective political parties/billionaire owners (6 companies own 90% of media [2]), made me switch to more independent journalists.
I have no objection to this -- I follow a few that I would say meet that definition well and which I trust. But boy do I worry that for 90% of the population, this translates to picking a bunch of enthusiastic propagandists whose bias is far worse than MSNBC, Fox News, or CNN ever were. I assume our craven and corrupt political parties will increasingly focus on propping up "independent journalists" who repeat their talking points for them.
Adequately implementing solves one problem (the making up a story because of a grudge), but creates a whole new set of likely much worse problems: how does one maintain a democracy / civil society? It's not just the trust of "social media" that you've eroded, you've almost certainly killed trust in traditional news reporting as well, especially considering just how much of traditional media is discovered via social media.
Effective democracy requires an informed voter base. Society requires its constituents to be invested in its continuity. Neither of those is achievable when we completely discard trust.
Yes, it's happened. Except a lot of people do have an exception - they'll trust the slop that reinforces their existing biases, or even if they know in their hearts it's not true, viewing their side's lies regularly still has an effect on the way they think.
Good point. Sometimes I wonder if social media, just almost every aspect of it, is the real cancer. Allowing just about anyone (globally) to anonymously deploy information warfare via the social media vector just seems bound to have horrible outcomes. It's just as bad with text as with images or video. Because of social media, we've trained at least 3 separate generations to self-sort into camps with customized ideological info sources that have incredibly-low standards for fact-checking and every incentive to tell their audience (1) exactly what they want and (2) whatever will enrage them most.
AI kind of makes this worse, but also only barely. Because most people really ought to know by now that almost any content could be AI, a video of, say, Trump kicking a baby or violating a goat wouldn't convince anyone that those acts happened (unless they already believed they happened).
Thing is, we're so flooded in biased BS, and no one has any incentive to produce non-sensational, factual news. I absolutely see 'post-truth' as the inevitability. You can't "weed a garden" when it is 100% weeds. The term "news" will cease to mean facts, and just become a branch of entertainment. Kind of the way "Reality TV" went from being supposedly a documentary (e.g. COPS) to just being a flavor of entertainment, where nothing needs to be real.
Stalin controlled the state. The state controls companies. Companies control watermarking.
This sort of solution to the fake image problem, makes it easier for stalin not harder. If everyone can make fake images that is one thing. If only the dictator can, well that is much worse.
Why are humans powerless to do anything about this? Aren't we making the technology? It's kind of a big problem for the future of the justice system and politics.
The watermarking should be on those things we want to verify as something that was not generated or manipulated. Something you'd add to, for instance, cameras. Putting them on the generated/manipulated is backwards as you can never get every model to watermark.
That model is equally bad though. Given that you're writing this in a discussion about gen AI watermarks, how in the world did you come up with the idea that Gen AI wouldn't be able to add a watermark?
Not that they "wouldn't be able".. that they wouldn't do it. For Gen AI watermarks to be useful all Gen AI systems need to add them and the incentives aren't there for that to happen. On the other hand the incentives are there for the non-generated sources to add it so they can differentiate themselves from the Gen AI media.
Perhaps not watermarks, but cameras could sign their pictures and put it in the extra data, that's not something that would be easy to add to fake pictures, or at least not the correct signature.
Maybe we do care about truth, freedom and privacy but the majority of rest of society will happily accept any T&Cs just to get access to whatever the next digital sliced pan is and as for truth and accountability, if they were two sides of the same coin on the ground people wouldn't bend down to pick it up as possesing it looks too much like responsibility and inconvenience.
I'm pretty sure watermarking is (or soon will be) a requirement for AI generated images in software used in the EU, as part of their regulations for AI transparency.
If i had a dollar for every time an American cried about literally any non-US jurisdiction having an iota of effect on them I could quit my job and leave this terrible website forever.
I disagree. Its mainly about having technical control and freedom. Reverse engineering how things work feels like peak hacker ethos. You don't have control of something if you can't remove it.
I think ethical considerations were always a bit secondary to technical power when it came to so called "hacker ethos".
After all, instructions on how to remove watermarks definitely feels like the sort of thing that would have been in phrack back in the day.
> Hackers believe that essential lessons can be learned about the systems—about the world—from taking things apart, seeing how they work, and using this knowledge to create new and more interesting things.
> Access to computers—and anything which might teach you something about the way the world works—should be unlimited and total.
> Mistrust authority—promote decentralization
> All information should be free
I phrased it a bit differently, and perhaps a little less sympathetically, but i think i was more or less saying the same thing.
In any case a tool like the article that strips watermarks seems exactly the sort of thing that would fit into what i quoted above. Its mistrusting authority - there is nothing more central authority then having a literal central authority adding hard to remove digital signatures to images. It promotes freedom of information - it supports explaining how watermarks work and what they are. Its fundamentally taking apart a system, which teaches us how the system works.
but I also live in a society that requires trust to function. making a tool the obliterates that trust(genAI imagery pipelines) then creating a tool that makes it trivial for normal people to remove any hint of controls over said trust eroding system is, toxic.
I get the argument about not putting in fingerprints that identify users, Good I agree. But this also removes the things that identify this as an AI image.
Now, what are the legitimate uses of that?
No really, why would I _need_ to remove a watermark for _legitimate_ purposes? Assuming that watermark is generic, rather than a fingerprint of a specific person
> No really, why would I _need_ to remove a watermark for _legitimate_ purposes?
When removing the watermark is easy, a very legitimate purpose of making the code to do it publicly available is to make a public demonstration that it's easy to do.
As for content use cases, suppose someone is using AI to modify their appearance because they're being unjustly targeted by an oppressive government. That government naturally bans doing that because they want to be able to identify and arrest their critics, so now if you make videos with your real face you get arrested but if you use a generated avatar then the watermark enables automated censorship because the government orders anything with the watermark to have its reach automatically restricted.
> suppose someone is using AI to modify their appearance because they're being unjustly targeted by an oppressive government
Then use a mask like everyone else. digital mask, one that obscures.
which is my main point, no, there isn't a legitimate need.
realtime avatars don't generally have invisible watermarks, also they are running from your machine, otherwise you've got a (normally credit card) trail to your front door. plus a video stream
also if you are generating stuff from a public provider, then tracing people isn't that hard to do.
As someone else pointed out: if watermarks are required, then everybody will assume an image without a watermark is the honest truth, which is obviously not true. Someone will end up in prison because of some image. This is bad.
but right now, we are eroding trust at an industrial scale.
There are no reliable tools for the end user, normal person, to work out if an image is AI or not. This erodes trust and lets bad actors get away with "oh thats AI generated" or use AI to defraud users.
But watermarks don't fix that because the bad actors could just use a model that doesn't include a watermark regardless of whether or not the ones that do can be removed. Foreign powers and monied interests were always going to have access to those and there are also already published local models that don't include them.
It's like making your image editing software watermark every image it edits in case someone photoshops a picture to show something that didn't happen. What's the point when anyone trying to fool people will always have access to ones that don't do that?
> Then use a mask like everyone else. digital mask, one that obscures.
It's not a matter of whether there are alternatives. If you can produce one without a watermark, and having the watermark allows the bad guys to cause trouble for you, then you have a legitimate reason to produce one without a watermark.
Wearing a mask also has different trade offs. A filter can change the shape of your face while continuing to allow you to show facial expressions.
> realtime avatars don't generally have invisible watermarks, also they are running from your machine, otherwise you've got a (normally credit card) trail to your front door.
"You don't have to remove the watermark because it didn't have one to begin with" isn't a way out. If anyone can create a video of themselves looking like $TARGET and saying despicable things without a watermark then you can't trust that a video without a watermark is real and the watermarks are pointless. Whereas if they're all supposed to have watermarks then you can't use the excuse that the video shouldn't have one to need removing.
> also if you are generating stuff from a public provider, then tracing people isn't that hard to do.
Only if the provider is subject to the jurisdiction/control of the user's oppressors.
The human ethos should be to never be misleading about the origin and truth of any content you create, forward, or pass on. If we care about honesty we should jail anyone who does so.
Even if you remove a watermark, the companies still have a record of which images they have generated and for whom. Even if you remove the obvious watermarks, all major image generators are using steganography to embed hidden information that you can't be sure were removed. This is a type of one-sided arms race where one player gets to be invisible if they want to.
Yes usually, since an important aspect of steganography is error correction. For example we know that SynthID is robust enough to survive resizing and small blurs.
Its what happens when people in power are paranoid dark-triad types and want to be able to catch anyone who threatens their power and stick it to them..
It already happened with Trump claiming any unfavorable content of his administration is "AI-generated" as a defense to dismiss real, unedited media. He literally said, “If something happens that’s really bad, maybe I’ll have to just blame AI.”
ie the video of garbage being thrown out the windows that his team already confirmed was real:
Apropos of nothing, I really don't love the construction of this reassurance. This is not actually a reasoned-through guarantee, it's just two things that happen to be true at the same time. But the latter could change at any point, on-device processing does not preclude the device notifying Apple about what it saw.
reply