You don't need peer reviewed studies to tell you water is wet.
Peer review is a technique to get evidence from data when SNR is low. It's not "science", it's just a technique. So is "throwing shit at a wall and seeing what sticks". Don't turn techniques into rituals, and science into religion.
Vibes are not evidence, neither is a curated demo. You need actual measured evidence that has an adversarial review to actually prove something without falling to confirmation bias.
Proof is not binary, it depends on the claim and the constraints you put around it, and the nature of the subject of your claim.
Most of the general LLM discourse in our industry is still closer to "proof of the pudding is in the eating" than to "double-blind studies on large cohorts, p<0.01, effect size is still so small that result is useless in practice"[0].
And we're not talking about curated demos either - most of the contested value can be proven for your own specific cases with little to no expenditure of money and time, at a PoC level (it gets more expensive once you try to operationalize it and find kinks that are hard to iron out).
And that is, the article claims (and I agree), the point of last 6-12 months of tokenmaxxing policies and top-down push - it's putting pressure on people to actually go and do those PoC-s for themselves, because just giving the opportunity and permission turned out insufficient for significant part of the workforce.
--
[0] - Ironically, I remember it was the opposite around the time GPT-4 came out. Back then people talked more about specific claims and demanded measured evidence, because it was hard to get the models to reliably do something interesting. But now that the models can handle bad prompting and can understand you even when you're drunk, suddenly people are denying the general capability of LLMs and asking for randomized control trials.
(For double irony, nowadays one can just ask an LLM for randomized trials; the current SOTA models will happily design you a bespoke eval pipeline if you ask them to.)
> And that is, the article claims (and I agree), the point of last 6-12 months of tokenmaxxing policies and top-down push - it's putting pressure on people to actually go and do those PoC-s for themselves, because just giving the opportunity and permission turned out insufficient for significant part of the workforce.
FWIW, I think most tokenmaxxing is, to riff of what you said earlier, turning a technique into a ritual and science into religion.
This isn't specific to AI, we've had it before with pretty much everything in software (and since well before software), from "object-oriented solves every problem" to "clean code [where every function is] two, or three, or four lines long", to reporting your daily kloc, to bounties for every bug reported and/or fixed.
Humans do what doomers are afraid AI will do: make a sounds-good utility function (tokens, lines of code, bugs, dead cobras) and get surprised when it is easily gamed for something far less helpful than the vision of whoever set the goal.
I think GP is being sarcastic, and pointing out that
1. "heavy rock falls faster" is what common sense will tell you (I was literally told this by multiple laypeople just a few days ago when sightseeing atop a tall tower)
2. This is disproven by a trivial experiment that nobody thought worthy of trying for millenia
3. therefore we do need peer reviewed studies to confirm even "obvious" knowledge.
Also, note that GP's parent post about "water being wet" is quite the subject of contention in scientific and philosophical circles, so that wasn't the best example either.
4. And we need.... something? to realize that both "common sense" / "multiple laypeople" and peer-reviewed studies are right.
Indeed, as per 2., no one is doing the experiments with rocks of different weight, and sufficient heights to easily measure time of fall. However, people have a lot of everyday experience with feathers, grains, leaves, wood, and rocks, as well as objects of various weight made of metal, paper, plastics. And in everyday experience, the heuristic actually holds out well: lighter stuff falls slower, or gets carried away by the wind.
This "heuristic" is purely empirical. You can't disprove it with peer-reviewed studies, because within its scope, it's literally the most basic, purest form of science: direct observation.
So in 1., the mistake is that of incorrect generalization. "Lighter stuff falls slower" is correct for everyday experience, it's the "therefore, heavy rock falls faster than light rock" is wrong.
Not because it doesn't fall faster, mind you - it does[0] - it's just that everyday experience is dominated by aerodynamic effects, and laypeople sometimes[1] mistakenly assign it to gravity.
Which I guess makes it a great analogy for the LLM story. Turns out everyday experience is actually valid in everyday situations. Generalizing from it is usually badly wrong, even if it sometimes arrives at correct answer for wrong reasons (and at wrong scales).
Generalization is hard.
--
[0] - Surprise. It's actually a heavy idealized particle falls at the same rate as light idealized particle. Actual matter is not an infinitely small point in space, and generates its own gravity field, so the heavy rock will land a tiny bit sooner than the lighter one, because it pulls Earth stronger towards itself - but then only if you drop the test bodies one by one (serially), and not together (in parallel, where the difference cancels out). But then it also turns out the mass canceling out for idealized particles isn't just a mathematical simplification, but a very deep truth about the universe...
[1] - Or don't. The question as phrased is, "does heavy rock fall faster than light rock"? This isn't a "specific physics theory question", it's a "real life" question. Treating a positive answer as belief on gravity is an error made by the asker.
This is possibly going to lead to a mind-blown moment for me as you reshape my entire understanding of physics. On the other hand, maybe you're slightly mistaken about newtonian physics?
> lighter stuff falls slower, or gets carried away by the wind.
Your examples are of smaller-density or larger-surface-area objects, not lighter ones. A bedsheet is heavier than a penny.
> Actual matter is not an infinitely small point in space, and generates its own gravity field, so the heavy rock will land a tiny bit sooner than the lighter one, because it pulls Earth stronger towards itself
When you're timing how long it takes for the rock to land, you're considering the Earth as fixed and applying gravity to the rock's center of mass.
The force applied between the Earth and the rock is F = G * (mEarth * mRock) / r^2
So the force that accelerates a twice-as-heavy rock is twice as large.
But the acceleration of that rock towards the earth is a = F/mRock, so in the end, if the rock is twice as heavy, its acceleration is still exactly the same as the lighter rock's.
> but then only if you drop the test bodies one by one (serially), and not together (in parallel, where the difference cancels out).
What are you talking about!?
If you want to split hairs, you could argue that if you drop them serially you're doing a minute change to the Earth's mass (which is actually so minuscule it makes no difference).
But even in your parallel universe of physics where the "heavier rock pulls the earth towards it", you're reaching a paradox similar to the one Galileo was testing for: if I link the heavy and the light rock together, they should fall slower than the heavy rock alone (because the light rock is slowing it down) but also fall faster than the heavy rock (because the total mass of the system is higher).
Ah. Here I am, taking the time to write this because I didn't have the useful bookmarklet[0] turned on in this browser window, and therefore I missed the emoji warning that would have told me I'm replying to some LLM trolling me with no understanding of physics.
Peer review is a technique to get evidence from data when SNR is low. It's not "science", it's just a technique. So is "throwing shit at a wall and seeing what sticks". Don't turn techniques into rituals, and science into religion.