I don’t have much in the way of links, but I can give you an overview.
We have a fair bit of vertical scale in the terrain here. We extract our water from a shallow well in a natural crevasse between ridges. It is made of concrete blocks stacked in a circle, filled with gravel and pinned with heavy rebar. The above ground part is finished in a regular fashion, with the blocks filled with concrete and a concrete cap. The well is built of a circle of 12 blocks, and is about 16 feet deep- where we encountered hard bedrock. An underground stream flows over this bedrock, which we extract from.
This raw water is pumped to a 300 gallon manifold tank about 160 feet above the extraction point using a 1HP centrifugal pump. From there, it flows down to the processing facility, where it is sediment and carbon filtered before flowing into either the 2600 gallon cistern, or back up the hill a bit to a 450 gallon upper campus distribution tank. Water passing through the processing facility is filtered and chlorinated, with the exception of the upper campus water, which is only filtered.
The upper campus water flows to cabins in the upper campus, and also serves as the input water for the RO system. The RO source water is pressurised by another centrifugal pump to 70psi, and is fed through a pair of 150GPD membranes after being filtered to 1 micron and passed through another carbon block. We run a 4:1 “waste” ratio to give us good life on the membranes (typically a year). The mineral rich “brine” flows into the 2600 gallon cistern and is used in the regular water.
We warehouse the drinking water in a 500-gallon tank at the processing facility.
There is a dual distribution system for water on campus. From the cistern at the processing facility RO water and regular water flows through underground tubing to a network of 5 utility huts where it is distributed to various homes and outbuildings. Each building then passes the main water through another carbon block to catch chemicals and chlorine, and drinking water gets mineralization and carbon again at the point of use.
The underground distribution network also carries 3 phase power, HVDC for solar, separate fiber optic networks for security, control, intranet, and ISP, as well as cat6 cables for RS485 control subsystems. The tank levels, pump controls, power distribution and usage monitoring, emergency and automatic casualty control shutoffs, etc are all operated over rs485 and modbusTCP to a server. It’s a lot of off the shelf stuff and some custom stuff that i have built. Someday I need to do a write up on that lol.
This is cool as hell. Sounds like you keep pretty busy, but if you ever had the inclination to write more about it or post some pictures somewhere, I for one would love to see.
Also, yeah, I know. Wordpress wasn’t a great choice even years ago when I set up the blog, but I was going to self host as a static site “soon” anyway and I needed to get started… almost good enough is the mortal enemy of adequate.
I added discovery support to scuttlebutt for link-local addresses to create "routerless" (no DHCP/SLAAC/Manual address assignment) self-repairing networks. Always wondered how broadcast worked, so appreciate the detailed writeup.
Probably worth a mention this is the creator of wasmer, for anyone interested. (I was curious who 'We' was to see what companies have experience with multiple wasm runtimes and their learnings)
The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.
No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.
Yeah. I've dealt with definitely-not-nation-states before, and their pattern was to sign up for free/cheap CI services (CircleCI, Github Actions, that sort of thing) and launch their attacks from there. The VPS thing also sounds very very plausible to me, I figured there was a long tail, but until I was looking up every network that was attacking us, I really had no idea how deep the long tail goes. I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.
> I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.
Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?
Reason I ask is long-hosted small business websites are sometimes established with the intent to legitimize some future unrelated traffic.
> Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?
Probably not what's happening.
I've tried to build a cloud CI service a while ago. Per their nature, you _have to_ allow arbitrary commands to be run. And you also have to allow outbound connectivity. So you don't need to 'own' anything in order to be dangerous. They will not run with heightened privileges but that's of little help if the target is external.
It is pretty difficult to reliably secure them against being used as a source of attacks as there's a lot you can do that will mimic legitimate traffic. Sure, you can block connections to things like IRC and you can throttle or flag some suspicious traffic. You can't really prevent HTTPS requests from going out. Heck, even SSH is pretty much required if you are allowing access to git.
Generally speaking, a build service provider will try to harden their own services and sandbox anything that is run in order to protect themselves from being compromised. Most providers won't want to be known as a major source of malicious activity, so there's some effort there. AWS and other large providers have more resources and will easily ban your ass, but that doesn't matter if it happens after a successful attack was launched.
That's exactly right. CI providers are good anonymizers for unsophisticated attackers because they provide an extra layer of obfuscation. But if they were doing something significantly harmful, I'd obviously be talking to those providers and asking for their own logs as part of the investigation, and then it would clearly link back to the actual culprits. So that was one popular technique to use to circumvent IP bans after abusing our service.
The whole hosting provider thing was another type of problems. I would always look at who owned the IPs that malicious sign-ups were coming from, and found a lot of ASNs owned by companies like "hosturwebsite4u.or.uk" and things like that. Those I assumed were just forgotten-about Linux boxes that the attackers used to anonymize through.
Ultimately, this was all to get a "free trial" of our cloud service, which did let you run arbitrary code. We eventually had a fairly large number of ASNs that would get a message like "contact sales for a free trial" instead of just auto-approving. That was the end of this particular brand of scammers. (They did contact sales, though! Sales was not convinced they were a legitimate customer, so didn't give them a free trial. Very fun times ;)
I should really write up the whole experience. I learned so much about crypto mining and 2020-era script-kiddie-ing in a very short period of time. My two favorite tangents were 1) I eventually wrote some automation to kill free trials that were using 100% CPU for more than 12 hours or something like that, and so they just made their miner run at 87% CPU. 2) They tried to LD_PRELOAD some code that prevented their process from showing up in the process table, but didn't realize that our tools were statically linked and that they were running in an unprivileged container, so the technique doubly didn't work. But, good old `ps` and `top` are linked against glibc, so they probably fooled a lot of people this way. They also left their code for the libc stub around, and I enjoyed reading it.
On the server-side, parsing the UA string is the best & fastest way to figure out which browser is on the other end or the connection. This can need to happen before you load any JS - this is commonly used to decide which JS bundles to load. When put under the microscope, browser have inconsistent behaviors and occasional regressions from version to version (e.g. performance with sparse arrays)
How much JavaScript is needed to accept my text input and provide auto complete options? Pretty wild we need to worry about browser compatibility to do this
> How much JavaScript is needed to accept my text input and provide auto complete options?
If you're talking about Google's homepage, the answer is "a lot". You can check for yourself - go to google.com, select "view source" and compare the amount of Closure-compiled JavaScript against HTML markup.
I think you've missed the point. Google's primary web search feature could, in theory, be implemented without a line of JavaScript. That's how it was years and years ago anyway.
I did not miss the point, I gave an answer based on the ground-truth rather than theory.
> Google's primary web search feature could, in theory, be implemented without a line of JavaScript
...and yet, in practice, Google defaults to a JavaScript-heavy implementation. Search is Google's raison d'être and primary revenue driver, I posit it therefore is optimized up the wazoo. I wouldn't hastily assume incompence given those priors.
The important word in the question you quoted is needed.
Google homepage is 2MB. Two fucking megabytes. Without JS, it's 200K.
I can't be the only person who remembers when Google was known for even omitting technically optional html tags on their homepage, to make it load fast - they even documented this as a formal suggestion: https://google.github.io/styleguide/htmlcssguide.html#Option...
> I can't be the only person who remembers when Google was known for even omitting technically optional html tags on their homepage, to make it load fast
This was back when a large fraction of search users were on 56k modems. Advances in broadband connectivity, caching, browser rendering, resource loading scheduling, and front-end engineering practices may result in the non-intuitive scenario where the 2MB Google homepage in 2024 has the same (or better!) 99-percentile First-Meaningful-Paint time as a stripped-down 2kb homepage in 2006.
The homepage size is no longer that important because how much time do you save by shrinking a page from 2MB to 300kb on a 50mbps connection with a warm cache?Browser cache sizes are much larger than they were 10 years ago (thanks to growth in client storage). After all, page weight is mostly used as a proxy for loading time.
I'm sorry you're going to have to pick an argument and stick to it before I can possibly hope to respond.
Either performance is so critical that a few kb to do feature detection is too much, or line performance has improved so much that 2MB of JavaScript for a text box and two buttons is "acceptable".
Your argument goes against empirical evidence in this instance. You can have it "both ways" when client-side feature detection is the slower choice on high bandwidth connections and you want to consistently render the UI within 200ms.
Performance goes beyond raw bandwidth, and as with all things engineering, involves tradeoffs: client-side feature detection has higher latency (server-client-server round trip and network connection overheads) and is therefore unsuitable for logic that executes before the first render of above-the-fold content. All of this is pragmatic, well-known and not controversial among people who work on optimizing FE performance. Your no-serverside-detection absolutism is disproved by the many instances of UA-string parsing in our present reality.
I've definitely had to code up alternative front-ends routed through a server I own to access Google on slow connections. If it takes too long your browser just gives up, and the site isn't just "unusable" (slow to the point of being painful), it's actually unusable.
99th percentile is fairly arbitrary. At Google's scale, that's a $2B yearly loss in customers they could have satisfied who went elsewhere. That's roughly 200FTEs who could be dedicated to the problem more efficiently than working on their other business concerns. Is not delivering a shit website when the connections are garbage that hard of a problem?
Of course, that's why there's 1.8MB of compressed JavaScript for a text box and an image. My point being that's it's silly and I'm exasperated with the state of the internet
It's wild to think that everything we've collectively learned as an industry is being forgotten, just 20 years later.
- We're on the verge of another browser monopoly, cheered on by developers embracing the single controlling vendor;
- We already have sites declaring that they "work best in Chrome" when what they really mean is "we only bothered to test in Chrome".
- People are not only using UA sniffing with inevitable disastrous results, they're proclaiming loudly that it's both necessary and "the best" solution.
- The amount of unnecessary JavaScript is truly gargantuan, because how else are you going to pad your resume?
I mean really what's next?
Are we going to start adopting image slice layouts again because browsers gained machine vision capabilities?
> People are not only using UA sniffing with inevitable disastrous results, they're proclaiming loudly that it's both necessary and "the best" solution.
Since you're replying to my comment and paraphrasing a sentence of mine, I'm guessing I'm "people".
I'm curious to hear from you on what - if any - is a better alternative that can be used to determine the browser identity or characteristics (implied by name and version) on the server side? "Do not detect the browser on the server side" is not a valid answer; and suggests to me the person proffering it as an answer isn't familiar with large-scale development of performant web-apps or websites for heterogenous browsers. A lot of browser inconsistencies have to be papered over (e.g. with polyfills or alternative algorithms implementations), without shipping unnecessary code to browsers that don't need the additional code. If you have a technique faster and/or better that UA sniffing on the server side, I'll be happy to learn from you.
"Do feature JavaScript feature detection on the client" is terrible for performance if you're using it to dynamically load scripts on the critical path.
I'm sorry you're going to have to pick an argument and stick to it before I can possibly hope to respond.
Either performance is so critical that a few kb to do feature detection is too much, or line performance has improved so much that 2MB of JavaScript for a text box and two buttons is "acceptable".
You can't have it both ways.
I don't recall that there was ever anything inherently wrong with using tables for layout, except that it was a misuse of tables so we were told it wasn't "semantic". Thus you had years of people asking on forums how to emulate tables using a mess of floating divs until flexbox/grid came around. In retrospect, tables are also clearly incompatible with phone screens, but that wasn't really a problem at the time.
One, it made the code unreadable and impossible to maintain properly, especially since most of those table where generated straight out of photoshop or whatever.
