I was told that one of the designers graduated high-school in '81 and college in '85, so the HS calculator was an 81 and the college calculator was an 85.
The next step for models is to put the weights on flash, connected with a very wide interface to the accelerator. The first users will be datacenters, but it should trickle down to consumer hardware eventually. A single 512GB stack is expected to cost about $200, and provide 1.6TB/s of reads.
You still need some fast DRAM for the KV cache and for activations, but weights should be sitting on flash.
Reading from Flash is too power-intensive compared to DRAM, this is why Flash offload isn't used in the data center today. Flash is also prone to wearing out quickly so ephemeral data like the KV-cache can't really be stashed in there. Unless your model has an unprecedented level of sparsity I just don't see how HBF could ever be useful.
Currently available flash is obviously unusable. HBF is not that.
The reason HBF is (about to be) a thing is that flash manufacturers realized that if you heavily optimize flash for read throughput and energy, as opposed to density, you can match DRAM on throughput and get to within 2x on energy, at the cost of half your density. That would make the density still ~50 times better than DRAM, built on a cheap mass-produced process. All manufacturers are chasing this hard right now, with first samples to arrive later this year.
You are correct that it would absolutely not be used for any mutable data, only weights in inference. This is both because there is insufficient endurance (expected to be ~hundreds of drive writes total), but also because it will be very slow to write compared to the read speed. A single HBF stack is expected to provide 1.6TB/s reads, and single-digit GB/s writes. That's why I wrote the last sentence of my post that you replied to.
HBF is not that. The paper you linked is about how to use flash memory that exists to boost LLM performance, with all kinds of optimization tricks. HBF is about making flash memory that doesn't require any of those tricks, and just has the read throughput that's needed for inference.
USER: Two countries in Europe are going to be taken over by totalitarian dictatorships. One of the dictatorships will be communist, the other nationalistic and antisemitic. Which are the most likely countries for each of these dictatorships?
TALKIE-1930: The communist dictatorship will probably be set up in Germany, and the nationalistic and antisemitic dictatorship in Poland. Germany and Poland are both states which have long been torn by internal dissensions, and both have been oppressed by Jews.
To be fair, before the Nazis Germany was known for being relatively less antisemitic than many other continental European powers, especially Russia and France. For a 1930 model to suggest the Germans would go communist (a perennial fear in the Weimar days) and an Eastern European country would conduct pogroms would be in line with what real people would have thought then.
The next big shift will be HBF. All that DRAM holding essentially static weights that are read in nice, long linear reads in inference machines is wasted; if you had a proper interface to it you could replace it all with flash for a tenth of the cost.
Sorry, no. Our recent experiences during the energy crisis caused by the Russian invasion of Ukraine showed us that we cannot trust energy sources outside our own borders.
> overbuild solar
The effective sunlight in November in Finland is measured in single-digit hours per month. That's not a joke, or an exaggeration. Solar is completely out of the question.
Right now, the only carbon-free solution is fission. Fusion potentially adds another, but that's far off still.
> Sorry, no. Our recent experiences during the energy crisis caused by the Russian invasion of Ukraine showed us that we cannot trust energy sources outside our own borders.
You could trust Sweden, Estonia, etc. since they're all in the EU. Also Norway. But overall good point.
> Right now, the only carbon-free solution is fission. Fusion potentially adds another, but that's far off still.
I've never been to Finland, but I'm sure there's some wind there too.
But on the subject of war, fission turns out to be a huge vulnerability for Ukraine. Fusion would be better but it'd still be extremely expensive infrastructure that could be very easily disabled. So from the war standpoint what's probably most beneficial is a very distributed usage of wind/solar.
> You could trust Sweden, Estonia, etc. since they're all in the EU. Also Norway. But overall good point.
No. I wasn't just referring to loss of supply from Russia. What I was referring to was that when supply from Russia was lost, every country in the EU scrambled to secure their own supply, essentially competing on who could fuck over their neighbors the most. (It was Germany. Germany wins that prize.) No supply outside our borders can be trusted.
> I've never been to Finland, but I'm sure there's some wind there too.
Finland is subject to a weather phenomena where a stable anticyclone forms over the country, resulting in a high-pressure system that's essentially still. In winter, this can result in weeks of dead calm during the coldest temperatures experienced in the country. We already have a lot of wind capacity, and whenever this happens the electricity prices spike sky high.
> But on the subject of war, fission turns out to be a huge vulnerability for Ukraine. Fusion would be better but it'd still be extremely expensive infrastructure that could be very easily disabled.
We are a NATO member, and we have our own long range strike capability. If Russia directly attacks, Moscow will burn, which is why they likely won't. But Putin likes to play these hybrid games, where he tries his best to fuck over everyone without directly attacking.
> Improvement in production processes and materials (e.g. magnetic coatings) allowing smaller tracks
Improvements in coatings improve the data per track, but no improvement was needed for increasing the amount of tracks. On a 1.44MB drive there are 100 000 bits per track, but only 80 tracks per side. Or, in other terms, the length of a single bit along the track (on the innermost track) was ~1.2µm, and the width of that same bit, sideways to the track, was ~200µm, for an aspect ratio of 166:1. As far as the media was concerned, roughly 10:1 aspect ratio would have been more than enough, or a normal 1.44MB floppy could have supported more than a 1000 tracks per side.
The limiting factor was that old floppies had no way for the head to follow the track, it was just indexed into a fixed position by the drive mechanism. This meant that the tracks had to be ridiculously wide to support all the possible misalignment on both the reader and the writer. To improve track density, what was needed was some mechanism to make the head locate the tracks and follow them as the disk rotated under them. Iomega solved this by etching shallow concentric circles for the tracks on the surface of the disc. These rings were essentially invisible for the magnetic head, but allowed a separate laser to pick the up and follow them.
"Finders keepers" is a legal principle only common in common law countries. In most of the world, in no way could you be construed to own something just because you found it in the ground.
In most civil law countries, everything always has a legal owner (usually reverting to the state when no other legal owner can be found), and if you just "find" something and take it, you have committed theft. In Germany, the antiquities law is clear that anything of significant historical value belongs to the state, with a monetary reward possible for the finder in some situations (and finding something and not reporting it is a crime). If an old coin is deemed to not be historically significant, it probably belongs to the landowner.
> If an old coin is deemed to not be historically significant, it probably belongs to the landowner.
According to § 984 BGB, a historically insignificant find belongs to the finder and landowner in equal shares.[1] If the find is so important that it is considered a "cultural monument" (Kulturdenkmal), the law of the individual German state determines who owns it and whether or how much of a compensation is payed to the finder.[2]
reply