You can probably get any coding agent with this if you put these instructions in the README/CLAUDE.md/AGENTS.md or whatever of your repo.
It's unclear to me if Bob is working as intended or how we should classify these types of bugs. Threat modeling this sort of prompt injection gets murky, but in general don't put untrusted markdown into your AI agents.
The slow food movement encourages eating local foods and gardening, among other things, so it actually improves food security, for people who aren't food insecure.
While it sucks that this happened, the good thing is that the ecosystem mobilized quickly. I think these sorts of incidents really show why package scanning is essential for securing open source package repositories.
We use a mix of static analysis and AI. Flagged packages are escalated to a human review team. If we catch a malicious package, we notify our users, block installation and report them to the upstream package registries. Suspected malicious packages that have not yet been reviewed by a human are blocked for our users, but we don't try to get them removed until after they have been triaged by a human.
In this incident, we detected the packages quickly, reported them, and they were taken down shortly after. Given how high profile the attack was we also published an analysis soon after, as did others in the ecosystem.
We try to be transparent with how Socket work. We've published the details of our systems in several papers, and I've also given a few talks on how our malware scanner works at various conferences:
I'm not exactly pro-AI, but even I can see that their system clearly works well in this case. If you tune the model to favour false positives, with a human review step (that's quick), I can image your response time being cut from days to hours (and your customers getting their updates that much faster).
You can't catch everything with normal static analysis either. LLM just produces some additional signal in this case, false negatives can be tolerated.
So what? They're not replacing standard tooling like static analysis with it. As they mention, it's being used as additional signal alongside static analysis.
There are cases an LLM may be able to catch that their static analysis can't currently catch. Should they just completely ignore those scenarios, thereby doing the worst thing by their customers, just to stay purist?
What is the worst case scenario that you're envisioning from an LLM hallucinating in this use case? To me the worst case is that it might incorrectly flag a package as malicious, which given they do a human review anyway isn't the end of the world. On the flip side, you've got LLM catching cases not yet recognised by static analysis, that can then be accounted for in the future.
If they were just using an LLM, I might share similar concerns, but they're not.
It's actually pretty easy to detect that something is obfuscated, but it's harder to prove that the obfuscated code is actually harmful. This is why we still have a team of humans review flagged packages before we try to get them taken down, otherwise you would end up with way too many false positives.
Yeah, what I meant is that obfuscation is a strong sign that something needs to be flagged for review. Sadly, there's only a thin line between obfuscation and minification, so I was wondering how many false positives you get.
Thanks for the links in your other comment, I'll take a look!
I think that would be static analysis. After processing the source code normally (looking for net & sys calls), you decode base64, concatenate all strings and process again (until decode makes no change)
Apparently it found this attack more or less immediately.
It seems strange to attack a service like this right after it actively helped keep people safe from malware. I'm sure its not perfect, but it sounds like they deserve to take a victory lap.
Do I need any? Automated tools cannot prevent malicious code being injected. While they can make attempts to evaluate common heuristics and will catch low hanging malware, they are not fool proof against highly targeted attacks.
Either way, the parent post is clearly ambulance chasing rather than having a productive conversation, which should really be about whether or not automatically downloading and executing huge hierarchal trees of code is absolutely fucking crazy, rather than a blatant attempt to make money off an ongoing problem without actually solving anything.
When we find malware on any registry (npm, rubygems, pypi or otherwise), we immediately report it to the upstream registry and try to get it taken down. This helps reduce the blast radius from incidents like this and mitigates the damage done to the entire ecosystem.
You can call it ambulance chasing, but I think this is a good thing for the whole software ecosystem if people aren't accidentally bundling cryptostealers in their web apps.
And regarding not copying massive trees of untrusted dependencies: I am actually all for this! It's better to have fewer dependencies, but this is also not how software works today. Given the imperfect world we have, I think it's better to at least try to do something to detect and block malware than just complain about npm.
I’m all for thinking about second, or third, or fourth order effects of behavior, but unless you have proof that Socket is doing something like lobbying that developers keep using NPM against their own best interests, frankly, I don’t know what your point here is.
> Do I need any? Automated tools cannot prevent malicious code being injected. While they can make attempts to evaluate common heuristics and will catch low hanging malware, they are not fool proof against highly targeted attacks.
So just because a lock isn't 100% effective at keeping out criminals we shouldn't lock our doors?
The more tools that exist to help find vulnerabilities, the better, as long as they're not used in a fully automated fashion. Human vetting is vital, but using tools to alert humans to such issues is a boon.
Signing doesn't protect against maintainer sabotage, but it could theoretically help if the registry were ever compromised. It mainly works to prevent MITM type attacks on the package distribution itself.
In the case of central package managers like rails/npm/cargo/etc., these benefits are very speculative, but there is probably some merit to adopting this approach in distributed ecosystems like go.
This seems really badly argued. The second version seems much worse and harder to extend. Looks like classic ORM style database abstraction wrapped with hand written types. This type of code usually leads to inflexible data models and inefficient n+1 query patterns. Relational algebra is inherently more flexible than OOP/ML-style type systems and its usually better to put as little clutter between your code and the db queries as possible in practice.
It's unclear to me if Bob is working as intended or how we should classify these types of bugs. Threat modeling this sort of prompt injection gets murky, but in general don't put untrusted markdown into your AI agents.
reply