Problem is that for strawberries or diamonds or SV startup stuff there is a line where if you make something infinitely available it will also loose the value.
You cannot equate paper clip maximizer with SV companies. Because fitness functions are different.
Not saying that diamonds industry didn’t make world worse off but still somehow it didn’t take over the world to make people eat diamonds for breakfast. So the same there will be no entity that can make all people everywhere eat strawberries for breakfast all year long.
Scary part is finance industry that basically already is self conscious with all the rules baked in and no single person being able to grasp it.
Finance with AGI could already become paper clip optimizer - but it actually needs energy only. It doesn’t need humans anymore. So it would most likely fill in whole world with power plants and erase all other life just to have the electricity.
>Then I realized that we are already surrounded by machines that demonstrate a complete lack of insight, we just call them corporations. Corporations don’t operate autonomously, of course, and the humans in charge of them are presumably capable of insight, but capitalism doesn’t reward them for using it. On the contrary, capitalism actively erodes this capacity in people by demanding that they replace their own judgment of what “good” means with “whatever the market decides.”
Excellent quote. We say we're in a rational world where we make rational decisions in our societal game we've all been told we have to play - capitalism where money is the determinant of success vs failure for corporations, families, individuals.
But step back and when looking at the question of whether it's rational to us as humans be playing this game and it is not rational at all. Why are we not deciding that food and places to live for everyone is the determinant of success of a country, society? Or happiness?
60 Minutes has a segment about Bhutan from a couple weeks ago [0] about this. They lived by something they named "Gross National Happiness". Which feels weird to type but again stepping back, it's because our whole lives we're told that "Gross Domestic Product", overall money, is the determinant of "best" and that's so engrained for us.
On a different note, Ted Chaing's short story books [1][2] are incredibly, incredibly good. I'm reading them again and read "Story of Your Life" earlier today. Being able to write fiction like that makes it much more trusting to listen to what someone has to say on other topics. And saying that seems like another topic - how we're told to downplay fiction compared to non-fiction, when our brains evolved for stories. But that's for another comment.
Why are we not deciding that food and places to live for everyone is the determinant of success of a country, society? Or happiness?
I think a lot of countries do, just not the USA. Money is truly king in the USA. But in Europe, developed parts of Asia, people are definitely not as rich, but decent quality basics are much much more accessible and considered a basic right.
But access to healthcare is what makes most of developed world much better and more civilized? It's one of the most important things you have to have in a society.
I have read about people in the USA forgoing basic procedures like endoscopes because they cannot afford it. Can you believe that?
I was recently in Spain, what's different is "good" food is cheaper and it's part of the culture, crap food is less acceptable. I had one of the best sandwiches and coffees of my life and at airport, an airport...at 6am. It was also cheaper than what I'd get at most US airports.
It's not a priority, it's part of the culture, go to most cities in Spain, Italy any night of the week, it feels like new years eve. People out all over the place at bars, eating and drinking good food.
I think in America food and it's price are the priority personally. Every meal that I actually enjoy and isn't fast food is painfully expensive. The only exception is Mexican food IMO.
Do we? In mainstream American politics people recoil at the notion of “European” political ideals. On HN most of the “Europe is so much better” sentiment seems to come from actual Europeans.
There’s a certain type of American who sees Scandinavian countries as having 100% cracked the code to every possible problem. It’s not a viewpoint that allows for a lot of nuance.
People in the USA are just trying to figure out how to pay their bills. This USA rhetoric is really tired. Nope, most people are actually poor as fuck.
Bhutan is about to create a special economic zone where this idea is relaxed, because they are facing a brain drain. The Bhutanese also want shiny things.
> Why are we not deciding that food and places to live for everyone is the determinant of success of a country, society?
That is what people do decide. That is why everyone slowly converged on capitalist economies with some level of social safety net - it is the system that best resists mass starvation and provides material comfort for as many people as possible. There is no alternative that is better at feeding (or housing, for that matter) people; otherwise that'd be the economic system of choice.
> it's because our whole lives we're told that "Gross Domestic Product", overall money, is the determinant of "best" and that's so engrained for us.
I grew up in the US and literally never heard this. It’s a way to measure economic behavior but nobody actually cares about that number. For the people who are money focused, they look at things like PPI and individual incomes. I’m not making a value judgement on how much focus should be put on that, but the point is that GDP is irrelevant to citizens.
For a concrete example, Biden/Harris got backlash about the economy sucking despite GDP being great because of the massive erosion in purchasing power during 2022/2023. Them pointing to GDP to show how strong the economy was even made it worse.
We've had isolated demos of this and that for many decades, all of varying difficulty of course, but I for one definitely agree with the post. I don't feel like I could go and buy a generic home assistant robot that can perform a variety of basic tasks anytime soon. Demos are nice, but they're kinda like news about new batteries, always a few years away from the real thing.
I'd bet many more than a few years away from a robot that can arbitrarily navigate your house, gather laundry from several baskets, the floor, etc. Divide into whites/colors or warm/cold loads if desired. Load them into an arbitrary washer, measure out detergent, use the right washer settings for the given load. Then come back and move them to the dryer, then come back and unload, fold, put away, etc. And don't get me started on sheets.
It's much easier than it should be to fool people with demos.
but how far are we from some bot ordering you a breakfast burrito on DoorEats and having it delivered? I can 100% envision a bug in one of the current crop of "AI" products resulting in mass burritoing.
Heh we're all standing on the tracks looking at the incoming light here at HN, but don't worry, all the down voters are yelling it's very far away over the ever increasing rumble.
Or more precisely, it is that we allow a group of people, under the control of a single person or very small number of people, amass incredible power, and do so in an amoral framework in blind service to a goal of stock price growth.
The problem is a scale problem in my mind. If you limit the scale then all the other problems become manageable.
We wouldn’t need antitrust laws anymore if our tax laws made it unprofitable to own shares in a company with, for example, 50% of search market share or online retailing share.
> The problem is not capitalism, it’s corporatism.
I can't agree with this, even though I think the corporate form has done more harm than good. While the idea of limited liability in exchange for a public good has some possible merit - building transport infrastructure was one of the original reasons they were created - giving away limited liability for nothing was a mistake. The shareholder value myth and corporate personhood have done even more damage. Nonetheless, not all capitalist ills are corporate ones. Many of the companies with the most capital doing the most damage are "shadow banks" - hedge funds, PE firms, etc. - which are run as partnerships, not corporations. They don't even issue stock, let alone care about its price. Also, nobody qualified to be opining about this could be unaware of the damage hyper-rich individuals are doing and will continue to do regardless of whether corporations are involved. Infinite accumulation of capital (and political power to go with it) is a problem, the corporate form is a problem, there's some overlap and they're both anti-free-market, but neither should be dismissed to focus on the other.
The idea that AGI doom scenarios are really late-stage capitalism is interesting and strikes me as fundamentally correct. The difference between the AGI takeover and capitalism is just the choice of metric to optimize on.
But, I think, it's the act of trying to optimize on a metric itself that is the source of the destruction. Unmeasurable human values can't survive an optimization process focused on measurable ones.
Our Gross National Product, now, is over $800 billion dollars a year, but that Gross National Product - if we judge the United States of America by that - that Gross National Product counts air pollution and cigarette advertising, and ambulances to clear our highways of carnage. It counts special locks for our doors and the jails for the people who break them. It counts the destruction of the redwood and the loss of our natural wonder in chaotic sprawl. It counts napalm and counts nuclear warheads and armored cars for the police to fight the riots in our cities. It counts Whitman's rifle and Speck's knife, and the television programs which glorify violence in order to sell toys to our children. Yet the gross national product does not allow for the health of our children, the quality of their education or the joy of their play. It does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile. And it can tell us everything about America except why we are proud that we are Americans.
It makes me question if I've ever seen a metric employed by a company that didn't have a detrimental effect on the happiness of the people involved. We are all unique but we're also really, really similar and it's in those uniquenesses that we find real happiness. But if we're only a number of sales/dollars/tickets/loc then the distinction is a zero-sum game; I only stand to lose a dollar that you make.
Business is not a zero-sum game, especially not technology businesses that constantly innovate. All sides of the transaction can benefit. Optimizing for profit or similar metrics doesn't inhibit those benefits.
I personally have never seen a metric that was actually beneficial for a company. Because the moment a metric exists, all rationality and reason is thrown out the window in favor of optimizing that metric. It doesn't matter if it destroys the company or customer trust in your product or runs out valuable employees. None of those are part of The Metric.
It's the whole 'we increased shareholder value by 1% even if we won't exist next year' scheme. The notion of any sort of sustainability or long term planning has long vanished.
> The difference between the AGI takeover and capitalism is just the choice of metric to optimize on.
Not specific to capitalism. The same thing occurs with any government/economic structure that is optimizing for a metric. Think of a communist government using “happiness” to drive decisions that just silently kills or drugs everyone who isn’t happy.
The fundamental difference of capitalism vs the other instrumental forces you are describing or intimating is that capitalism is self-reproducing. Through the explosion of the profit motive and the harnessing of wage labour it continually opens new markets, and demolishes limits to growth.
"AI" in the current hype cycle (LLM generative ML yada yada) is just classic capitalist industrial automation, applied to one of the few areas that still had echoes of the pre/proto-capitalist "craft economy"
Prepare, nerds, to be put through the same process as seamstresses, weavers, blacksmiths, lumberjacks, etc.
So much of what people identify as post-modern or late-stage or whatever is just the continuation of the same intrinsic process that has been happening since the end of the 18th century.
> This summer, Elon Musk spoke to the National Governors Association and told them that “AI is a fundamental risk to the existence of human civilization.”
And here I was going to suggest that billionaires, unbridled mega-corporations were the fundamental risk to the existence of human civilization.
> Musk gave an example of an artificial intelligence that’s given the task of picking strawberries.
Also odd since it's more likely that a corporation, in the name of maximizing profits, would make decisions that threaten humanity. We can start with Bhopal, India. If you find fault with that example I am sure there are plenty of others, some probably a good deal more subtle, that others can suggest.
You don't have to imagine. Both the corporations and USG have been doing that for the last decade at the very least. If you want to test this, try posting anything controversial (AKA not 100% in line with the neoliberal agenda) in the YT comments section. Gone within seconds. Those aren't human moderators.
> And here I was going to suggest that billionaires, unbridled mega-corporations were the fundamental risk to the existence of human civilization.
Depending on the measure [0], it's possible that the richest billionaires and least-bridled corporations are both well in the past. Human civilization survived, and will continue to survive.
There was an infamous supreme court case[0] in 2010 which has set off a steady increase in campaign. Before that influencing campaigns with money was much more regulated.
Since then it seems to me (I am not an expert in this) that there's been basically a race to the bottom in politics which Donald Trump correctly identified, and has had basically a "pay me and I'll do you political favors nakedly and nobody can do anything about it" approach, which appears to be a winning strategy, unfortunately for the system as a whole.
edit: I seem to have also missed a ruling this past march which had an impact:
> America PAC’s work was aided by a March ruling from the Federal Election Commission that paved the way for super PACs to coordinate their canvassing efforts with campaigns, allowing the Trump campaign to rely on the near-unlimited money of the nation’s most high-profile billionaire to boost turnout in deep-red parts of the country.
This article compares a strawberry picking machine killing all of humanity to increase strawberry fields with current corporations. This is stupid because corporations do have guardrails both internally and externally from society. All of the mega Silicon Valley corporations are not expanding by murdering people. Even in the ones that are expanding in ways that people question (social media), they are filled with humans who actually think they are doing the right thing.
Humans with morals are still very much in the decision chain and there is obviously a lot of debate about their morals, but them being there makes such a vast difference that the comparison to the strawberry AI is completely invalid. The strawberry AI isn’t even considering humans.
The article then builds on that false comparison for the rest of the article so there isn’t much to gain from the rest of it.
You can make the same lazy comparison to a completely socialist, centralized decision making by a government optimizing for a single metric (voter approvals, poverty levels, whatever). It has nothing to do with capitalism or the economic system.
TLDR; article says mega corps are the same as dangerous AI because they make optimizations in favor of profit that some people disagree with.
