$ time ./wc-avx2 < bible-100.txt
  82113300
  
  real    0m0.395s
  user    0m0.196s
  sys     0m0.117s

    $ time ./optimized < bible-100.txt > /dev/null

    real    0m1.525s
    user    0m1.477s
    sys     0m0.048s

In the optimized case:

  real    0m0.395s
  user    0m0.196s
  sys     0m0.117s

In other words: yes, the author managed to beat the speed of the disk subsystem. With two caveats:

1. not by much - similar attention to tweaking of I/O parameters might improve I/O performance quite a bit.

2. the I/O path is CPU-bound. Those 117ms (38% of all CPU cycles) are all spent in the disk I/O and file system kernel code; if both the disk and your user code were infinitely fast, the command would still take 117ms. (but those I/O tweaks might reduce that number)

Note that the slow code numbers are with a warm cache, showing 48ms of system time - in this case only the ext4 code has to run in the kernel, as data is already cached in memory. In the cold cache case it has to run the disk driver code, as well, for a total of 117ms.

1) "interop with C" is part of the fundamental requirements specification for any code running in the Linux kernel. If Rust can't handle that safely (not Rust "safe", but safely), it isn't appropriate for the job.

2) I believe the problem was related to the fact that Rust can't implement a doubly-linked list in safe code. This is a fundamental limitation, and again is an issue when the fundamental requirement for the task is to interface to data structures implemented as doubly-linked lists.

No matter how good a language is, if it doesn't have support for floating point types, it's not a good language for implementing math libraries. For most applications, the inability to safely express doubly-linked lists and difficulty in interfacing with C aren't fundamental problems - just don't use doubly-linked lists or interface with C code. (well, you still have to call system libraries, but these are slow-moving APIs that can be wrapped by Rust experts) For this particular example, however, C interop and doubly-linked lists are fundamental parts of the problem to be solved by the code.

Rust is no less safe at C interop than using C directly.

This could have happened with no linked lists whatsoever. Kernel locks are notoriously difficult, even for Linus and other extremely experienced kernel devs.

And safe abstractions mean this stuff usually only matters if you’re implementing new, complex collection types. Like an ECS, b-tree, or Fenwick tree. Most code can just use the standard collection types. (Vec, HashMap, etc). And then you don’t have to think about any of this.

Ok. But making it easier or harder isn't the same as making it impossible.

To quote GP:

> 2) I believe the problem was related to the fact that Rust can't implement a doubly-linked list in safe code.

Rust can implement doubly linked list in safe code. It can. It wraps the unsafe parts in an abstract manner.

Not quite, and an interesting story that fits these engineering maxims better than you might think.

An analog channel with the bandwidth and SNR characteristics of a landline phone line has (IIRC) a Shannon capacity of 30-something kbit/s, which was closely approached with V.34, which used trellis coded modulation plus basically every other coding and equalization mechanism they knew of at the time to get to 33.6kb/s on a good day.

But... by the 80s or so the phone system was only analog for the "last mile" to the home - the rest of the system was digital, sending 8-bit samples (using logarithmic mu-law encoding) at a sampling rate of 8000 samples/s, and if you had a bunch of phone lines coming into a facility you could get those lines delivered over a digital T1 link.

Eventually someone realized that if your ISP-side modem directly outputs digital audio, the downstream channel capacity is significantly higher - in theory the limit is probably 64000 bit/s, i.e. the bit rate of the digital link, although V.90 could only achieve about 56000 b/s in theory, and more like 53kb/s in practice. (in particular, the FCC limited the total signal power, which means not all 64000 combinations of bits in a second of audio would be allowable)

I worked with modem modulation folks when I was a co-op student in the mid-80s. They had spent their lives thinking about the world in terms of analog channels, and it took some serious out-of-the-box thinking on someone's part to realize that the channel was no longer analog, and that you could take advantage of that.

A few years later those same folks all ended up working on cable modems, and it was back to the purely analog world again.

But why is it higher? It's still an analog channel (the last mile from the ISP to your house), right? Doesn't it get filtered? So isn't it still subject to the Shannon-Nyquist limit?

Here's an ASCII drawing of which parts are digital vs analog as I understood your explanation:

  Rest of world<--- digital--->Telco<---digital--->ISPmodem<---analog--->HomeModem

I hope you can clear up my confusion.

Yes, but you need the bare copper wire without signaling. We operated a local ISP in the 90's and did exactly that by ordering so-called "alarm circuits" from the telco (with no dial tone) and placed a copper T1 CSU on each end. We marketed it as "metro T1" and undercut traditional T1 pricing by a huge margin with great success to the surrounding downtown area.

Though the actual available bandwidth was very dependent on distance. People would lease dedicated pairs for high bandwidth across town (or according to a random guy I talked to at a cafe: just pirate an unused pair that happened to run between their two buildings). But once we start talking between towns, the 32kbit you could get from the digital trunk lines was almost always higher than what you could get on a raw analog line over the same distance.

Traditionally both the ISP and you pay for analog phone lines from the telco. The telco uses digital internally (remember you and your ISP probably aren't at the same exchange), which puts a hard limit on data rate - there is no trick you can do to get more bits through than the bits used in the digital part of the call.

If you (as the ISP) buy enough lines you can get them delivered in digital format. A T1 is designed to carry 24 simultaneous phone calls, acting virtually as a bundle of 24 analog phone cables. So the obvious next stage was to have a modem that can handle 24 simultaneous connections on one cable.

Now you have ISP\_modem←Ax24→ISP\_muxer←Dx24→Telco←→Telco←A→User

The ISP's modem generates analog signals for up to 24 simultaneous incoming calls, and they pass into a multiplexer that connects 24 analog lines to a T1 line and they go through the telco digitally to users. The maximum bandwidth is still as before - the modem has to generate an analog signal that will still be receivable at the other end after A2D and D2A conversion. Even though the digital bandwidth for the digital part is 56kbps, the maximum achievable bandwidth through this digital-bottlenecked analog call was found to be 33.6kbps.

But the industry had an idea: by convincing the telco to install the modems into the user's exchange, the analog portion would only be between the telco and the user, without a digital segment in the middle of it, and therefore wouldn't be bottlenecked the same way. The same digital backhaul from the ISP through the telco was used, but instead of transmitting a digitised analog modem signal and therefore causing degradation of quality, it transmitted your actual internet traffic bits, up to 56kbps. The analog signal was made at the user's side of the telco and didn't have to fit within 56kbps when digitised.

Pedantically, the digital circuits are 64kbps but one bit in some bytes is used for call status signaling, which is okay for voice, but the ISP equipment can't predict which bytes have a bit overwritten (and it could be multiple if there are several hops) so it just used 7 bits in each byte.

> Couldn't you send data at megabits per seconds over a mile long copper wire without using modems at all (using just UARTs?).

No. The exchange is sampling the analog signal coming in over your phone line at 8kHz and 8 bits per sample. They just designed modems that sent digital data over that analog link, in a way that would line up exactly with the way the exchange will sample it.

4kHz/2*log2(1+10^(31dB/10)) ~ 60.3kBps

[0] https://www.ecfr.gov/current/title-7/subtitle-B/chapter-XVII...

The problem is that the ADC in the telephone exchange is not clocked coherently to your modem. You're paying various companding, quantization (time and level), etc, losses-- it is not a nice linear channel.

An end-to-end analog connection could be capable of significantly more-- while there's noise, there's no "8KHz cliff." A 45dB SNR for 0-4KHz has a >60kbps Shannon bandwidth, and there's not exactly a 4KHz cliff even with loading coils present.

Huh? The SotA for dial-up modems in the 1990s was indeed 56 kbps, achieved with various encoding and compression hacks in both domains including trellis coding.

Weirdly enough, you can apparently still buy them from US Robotics: https://www.usr.com/products/56k-dial-up-modems/

The specific lines about Ellison and a lawnmower start at 38:28 in the linked video; the entire Oracle rant starts at about 34:00.

They do not really understand how bad Oracle used to be. This is us, old combat veterans, sitting by the fire, describing unspeakable battles to the youth...knowing full well that they think we are exaggerating. :-)

And the most disturbing part is the realization that the Frankenstein monster itself, Larry Ellison, is still out there. Still roaming free. Still very much alive... An eternal, terrifying, lawnmower wielding zombie of enterprise software and government corrupting rent extraction.

We need confiscatory taxation for a better future.

No, we can have many targets. People who hoard money for the benefit of themselves with the detriment of society and the population at large are all "destructive aliens who undemocratically ruin everyone’s good time" to borrow the words of parent commentator. If just 10% were slightly less evil and egoistic, it would lead to huge improvements, and only a slight reduction to their own lifestyles. That they don't, is a stain on the legacy of humanity.

We need to bring the entire extraction class to heel. They are degrading, even destroying, society.

(this is at a campus Dunkies where there's no drive-through, and I have a hard deadline to start my lecture. If there's no line at the register, and I've got five minutes before class starts in a room down the hall, it shouldn't take a logistical genius to get me a regular coffee in time for class)

Actually that's a really common use - I've bought a half dozen or so Dell rack mount servers in the last 5 years or so, and work with folks who buy orders of magnitude more, and we all spec RAID0 SATA boot drives. If SATA goes away, I think you'll find low-capacity SAS drives filling that niche.

I highly doubt you'll find M.2 drives filling that niche, either. 2.5" drives can be replaced without opening the machine, too, which is a major win - every time you pull the machine out on its rails and pop the top is another opportunity for cables to come out or other things to go wrong.

#1 is all NVMe. It's dominated by laptops, and desktops (which are still 30% or so of shipments) are probably at the high end of the performance range.

#2 isn't a big market, and takes what they can get. Like #3, most of them can just plug in SAS drives instead of SATA.

#3 - there's an enterprise market for capacity drives with a lower per-device cost overhead than NVMe - it's surprisingly expensive to build a box that will hold dozens of NVMe drives - but SAS is twice as fast as SATA, and you can re-use the adapters and mechanicals that you're already using for SATA. (pretty much every non-motherboard SATA adapter is SAS/SATA already, and has been that way for a decade)

#4 - cloud uses capacity HDDs and both performance and capacity NVMe. They probably buy >50% of the HDD capacity sold today; I'm not sure what share of the SSD market they buy. The vendors produce whatever the big cloud providers want; I assume this announcement means SATA SSDs aren't on their list.

I would guess that SATA will stay on the market for a long time in two forms: - crap SSDs, for the die-hards on HN and other places :-) - HDDs, because they don't need the higher SAS transfer rate for the foreseeable future, and for the drive vendor it's probably just a different firmware load on the same silicon.

Since it was offline, the bug was obvious although a bit frustrating - you had to put in multiple waypoints to make it forget its urge to send you on the ferry to Hull when you were trying to get to parts of the South Shore.

This seems like an entirely different level of craziness, though.