Yes, for example, the datasets at this page: https://adsb.exposed/ are indexed by the Morton curve: https://github.com/ClickHouse/adsb.exposed/#database-and-que... and https://reversedns.space/ as well.

Also, a trivial application is image compression. Let's say you have a PNG image. PNG uses zlib, so if instead you take a raw bitmap and compress it with ZSTD, it typically will be better, but if you also sort pixels by the Hilbert curve first and then compress with ZSTD, it will be typically even better.

Aha, thanks. The adsb.exposed maps are excellent, thank you! Really intrigued by the building maps.

I tried pgrust on ClickBench recently, but it didn't work at all: https://github.com/ClickHouse/ClickBench/pull/983#issuecomme...

I tried pgrust on ClickBench recently, but it didn't work at all: https://github.com/ClickHouse/ClickBench/pull/983#issuecomme...

This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247

It will be nicer if the README focuses more on per-core performance.

About the actual algorithm - will something like matching in a perfect hash table help?

Why not a wildcard DNS pointing to a few Anycast IP addresses with proxies?

You can look at the LZ4 decompression implementation in ClickHouse: https://presentations.clickhouse.com/2018-highload-siberia/ (the presentation is quite old, but the implementation was updated recently). It uses the same LZ4 format, just decompresses faster.

It is slower than LZ4 on AArch64.

Could you tell me what your test setup and corpus was?

For reference, to test ARM64, I tested v0.1.0 on an M3 mac with this fork of lzbench: https://github.com/welcome-to-the-sunny-side/lzbench/tree/ad...

Here, lz4's decompression speed was far slower than misa77 and zxc. Results are here: https://github.com/welcome-to-the-sunny-side/misa77/blob/mai...

Source?

Just tested a few minutes ago on the ClickBench dataset. Overall quite good, but, depending on particular columns, most of the time slower - e.g., ~2.0 vs ~2.8 GB/sec on Graviton 4 machine in AWS.

You probably meant MB/sec not GB.

No? Decompression is rated in GB/s per core now.

Parent fixed the error I pointed out since making my post by dividing by a thousand instead of changing the units so now my comment looks wrong.

It used to say 2800 GB/s which is obviously bonkers.

Yes, it was a typo. Thank you!

Great article! About inserting LIMIT - we have out-of-band `limit` and `offset` settings:

https://play.clickhouse.com/docs?user=play#q=limit&name=limi...

ClickHouse also has vectorscan as a regexp engine, supporting multi- and fuzzy-matching, multi-Volnitsky for substring matching, as well as the new LLVM-JIT compiled regexp engine.