Problem has never been the storage. Its running those queries to return in milliseconds - if its for a dashboard, an alert of your new AI agent trying to make sense of it.
I wrote this after seeing the same failure modes across multiple teams and metrics stacks.
The intent here isn’t to argue that high cardinality is bad but to explain the mechanical reasons unbounded cardinality shows up later as cost surprises, slow queries, and trust erosion.
Curious if others have seen similar query-time or indexing issues they have noticed.
(en français, of course. I link to it in its original language as I take issue with the usual translation of the very title as "Diversion", which immediately lacks the double-entendre of "Divertissement", which in french stands for both "diversion" and "entertainment")
Maybe what TFA author should do is spend some time standing on the shoulders of giants and read some philosophy?
> Tel homme passe sa vie sans ennui en jouant tous les jours peu de chose. Donnez-lui tous les matins l’argent qu’il peu gagner chaque jour, à la charge qu’il ne joue point : vous le rendez malheureux. On dira peut-être que c’est qu’il recherche l’amusement du jeu, et non pas le gain. Faites-le donc jouer pour rien, il ne s’y échauffera pas et s’y ennuiera. Ce n’est donc pas l’amusement seul qu’il recherche : un amusement languissant et sans passion l’ennuiera. Il faut qu’il s’échauffe et qu’il se pipe lui-même, en s’imaginant qu’il serait heureux de gagner ce qu’il ne voudrait pas qu’on lui donnât à condition de ne point jouer, afin qu’il se forme un sujet de passion, et qu’il excite sur cela son désir, sa colère, sa crainte, pour l’objet qu’il s’est formé, comme les enfants qui s’effrayent du visage qu’ils ont barbouillé.
I wouldn't be so sure about this, this might depend on the personality of the player. Some might think that introducing monetary stakes in fact ruins the game itself. And you can even take it further : the winning itself might become secondary - at which point playing the game is probably more akin to a form of artistic expression, infused with a different kind of meaning. (There are also games where you win, but in cooperation with others rather than against others.)
But then this passage also reminded me of this recent thread :
"More men are addicted to the 'crack cocaine' of the stock market"
>Engineers have to pre-define and send all telemetry data they might need – since it’s so difficult to make changes after the fact – regardless of the percentage chance of the actual need.
YES. Let them send all the data. The best place to solve for it is at Ingestion.
There's typically 5 different stages to this process.
Ingestion - Build pipelines that allow to process this data and provide for tools like streaming aggregation, cardinality controls that allow to 'process it' or act on anomalous patterns. This atleast makes working on observability data 'dynamic' instead of having to go change instrumentation always.
Storage - Provide blaze (2hours), hot(1 month), cold(13 months) of tiered data storage with indipendent read paths.
This, in my opinion has solved for the bulk of cost & re-work challenges associated with telemetry data.
I believe, Observability is the Big Data of today, without the Big Data tools! (Disclosure: I work at Last9.io and we have taken a similar approach to solve for these challenges)
The rice farmers I met in Bihar, for instance, are now growing a new variety of flood-tolerant rice—nicknamed “scuba” rice—that can survive two weeks underwater.
Found this extremely interesting. Probably because, I was surprised by the thought that, this could be done
Agree! With http://tah.io/get we have tried to make use of BLE not just as beacons but also for combining the power of your physical sensors with 'smartphone' sensors to have much more interesting use cases!
