Excited to see models get first-class support! Will be interesting to see what opportunities this opens with this being formalized (other tooling/integrations, etc.).
Note that Compose files using the previous model declarations (using providers) will still work.
(from the Docker DevRel team) Ha! Just had to say... this gave me a good laugh! We're trying to get better... I swear! Comms are hard. We'll get there.
Appreciate it. I'd advise to try hard to not be dismissive of even the more snide and flamey comments in this thread - their interpretations are valid from what I see posted so far.
> Given these statements directly contradict each other
Actually... they aren't contradictory. The organization data will be retained for 30 days and is subject to deletion. That data includes the teams, memberships, etc. But, it wasn't clear what we were going to do about the images. Keeping the public images is important as many other images build on top of them.
> It feels like they changed the actual strategy
We recognize it might feel that way, so apologies. But, that's part of where we are recognize it wasn't clear the technical details... we didn't talk at all about the images. After the feedback, we recognized this, so wanted to make that clear.
If you are deleting the organizational data and effectively archiving [1] all the images, keeping only the option for public images to be pulled but not updated... then how will affected maintainers be able to delete their now out of date public images after the 30 day cut-off? You will have to retain enough of the organization data to allow that to happen.
Keeping the public images available in an archived state is okay for specific image references, but questionable for specific image tags and somewhat irresponsible for the `latest` tag. A `latest` tag that cannot be updated is ... worse than no `latest` tag.
Responsible maintainers that are unable to apply for open-source status or otherwise sponsor their usage of organization public repos should be advised to delete their public repos.
Responsible users of public images on Docker Hub need to have a way to determine which images will be affected, and which will continue to be maintained. Archiving the public repos gives an extended grace period, but users will still need to be prepared to notice if they end up using a now unmaintained, archived repo and migrate to alternative image sources.
The irresponsible thing is making it so the tag exists, but the organization behind it cannot update it.
Let's take for example the "jenkins/jenkins:latest" image.
Jenkins is notorious for having security updates, so in 2 years, if the latest tag is still there and frozen, it will be an attractive nuisance, causing people to download insecure software...
That's what the parent comment is trying to say. It's irresponsible to leave the image that implies it's "up to date and secure" because it's "latest", but is really insecure, and the organization owning it cannot change anything about that without paying $$. It's basically holding users of the image hostage.
You missed his point. He's saying "latest is an anti-pattern". Which is correct. Everyone should be pinning to specific versions or semver to avoid being accidentally upgraded to a release with breaking changes.
Yes, obviously making existing tags immutable is bad. Nobody is disputing that.
There are exceptions though. I'm the kind of person that would pin Jenkins to latest even if it is an antipattern. I'm way more concerned about security flaws than a temporary CI breakage. So for me: Everyone should be pinning Jenkins to latest to avoid accidentally staying on a release with security holes.
It's not* just `latest` tags, it will also affect any other image tag.
If you've been referencing org/image:tag where tag=major-minor, and gets updated when there's a patch, then that's going to stop getting updated.
Without either the tag being deleted (and thus your pulls failing), or going out to find updates on that container - you may not notice that it's fallen out of date and the image/tag is no longer being updated.
With the entire organisation being removed from Dockerhub, it sounds like there's not even going to be a way for people to say "We've moved off Dockerhub, our images/source/etc is now over here".
You'll just have to search and hope you can find where it's moved to.
sometimes I want a container running the latest version of something. maybe i'm integration testing my stuff against that release to make sure stuff still works. or maybe I'm hoping a bug was fixed and will version pin later.
i agree that production software should version-pin all the things, but latest still has a place.
Deleting 'organization data' absolutely read that they would delete everything. Changing direction and back pedaling with a non-apology is borderline insulting.
I understand the need to make money as a company, but it really is biting the hand that fed messing with open source maintainers
I have no horse in this race, but fwiw, I can see how this mistake would be made honestly. Organization data could be easily refer to just the metadata of the organization, and depending on how the product is structured[1], could feel quite different from public images.
[1] Disclaimer: I don't know how the product is structured.
With the organization data gone, will there be a way to update the retained images, like security fixes etc? If not, then this could become very dangerous.
Really, if they delete the org data and images can’t be updated, it might just be better to delete them all just to avoid these inevitable issues (maybe with a longer delay). Just rip the band-aid off and be done with it.
Neither npm nor NuGet are volunteer driven organizations. npm is owned by Github, NuGet is owned by Microsoft, and I'm sure that there are dozens of other examples behind many key pieces of free dev&ops infrastructure that are owned by for-profit companies.
Are they allowed to do it? Of course! Are we allowed them to call them out on their bait & switch tactics? Of course, what else are we supposed to do?
Just because something is common, like building a user base based on implicit promises and then pulling the rug once the service reaches critical mass, doesn't mean it should be accepted and normalized.
You don't have to lecture me about the pitfalls of willingly relying on for-profit companies or the benefits of decentralization, but in the case of Docker we don't really have much of a choice unless you're suggesting that don't use it at all.
Images are published wherever the author decides that they're published and these changes are going to affect everyone who relies on an image that used to be hosted on Docker Hub.
I understand you are in a difficult position, but this is a bit absurd.
> During that period you will maintain access to any of your public images.
The only reason that sentence would be in there is if after that period you would lose access to the public images! And from Merriam-Webster, "access", verb, definition two: "to open or load (a computer file, an Internet site, etc.) a file that can be accessed by many users at the same time".
> it wasn't clear what we were going to do about the images.
No, it was quite clear; after the 30 day period we would not be able to pull the images. That's what the announcement said. It was not ambiguous. That may not have been the policy or what was intended to be announced, but the issue here isn't a lack of clarity.
(Also, letting the images stay accessible but disallowing any changes is only marginally better than just removing them, so the current policy - whether or not it's the same as the originally announced policy - is still terrible.)
Write access is a subset of all access, so I don't think we can really argue that the plain meaning of the original statement was about removing write access.
But yes, a missing word is certainly a plausible explanation for how they issued a statement that meant the opposite of what they apparently intended.
But the original statement did not say all access, it merely said access:
> During that period you will maintain access to any of your public images
Assuming that the you in that sentence is the organization and not the general public (given the use of your organization earlier in the paragraph), the logical interpretation is that they meant write access here, and not all access -- since read access is not limited in any way to the you in that sentence.
Yes, I agree the original messaging was terrible. But claiming that the original can only have meant all access is not consistent with the wording of the announcement.
They are. Your intent may not have been contradictory, but the messages received by everyone else were contradictory. You should own that if you are serious about doing better. Your intent doesn't really matter in these situations.
Yeah, really weird that after an apology announcement they’re still defending the original message at all. Not too hard to say “Yes, those messages contradict each other. The first one did not communicate our actual plan. The second message is a correction and clarification.”
More cynically, the intent might be blaming image maintainers: since obsolete images that appear current are a problem, responsible maintainers will delete them before losing access; then Docker will be able to tell inconvenienced end users that the maintainers autonomously and unnecessarily decided to remove their images.
"During that period" refers to the 30-day period. During that time, the images are accessible. After the 30-day period, they will still be pull-able, but not able to be updated.
So, any public image where the maintainer doesn't jump through hoops gets frozen in time, unable to be updated, and starts accumulating CVEs? This sounds worse than deleting the image.
Any smart FOSS maintainer will find alternate hosting...
Can't believe how soon this announcement came after the redhat "we're killing centos support now, best of luck". It's pretty clear how this industry reacts to major support changes with no heads up.
This is an important implication that needs to be brought up in the FAQ explicitly.
In other words, the public repos are being archived. If I was a maintainer responsible for providing up-to-date and secure images, then I think it would indeed by my duty to delete them, if I am no longer able to update them.
> If you don’t upgrade to a paid subscription, Docker will retain your organization data for 30 days, after which it will be subject to deletion. During that time, you will maintain access to any images in your public repositories, though rate limitations will apply.
Specifically (emphasis mine):
> During that time, you will maintain access to any images in your public repositories
So, the logical conclusion, which literally everyone else on HN had, was that after that time you will lose access to images in your public repositories; access meaning "we can get to the image" in this context, because that's what people f-n care about.
Not to mention the other part, about how Docker will still have images available for pull that can't be changed, for which there is no way to "forward" user pulls elsewhere if the developer chose to not pay the fee; so in affect you're capturing their user base with old software and almost no way to know that.
"DevRel" at Docker failed this week. Just own up to it, take the hit, and don't be evasive. Evasiveness is shady and no one trusts that bullshit.
Nah, it's this guy sitting in here trying to tell us how we're supposed to think, that it wasn't their fault we were stupid, that's the issue I have. I don't give a flying FUCK about Docker as a company; whether they live or die matters less to me than when I'll have to pee next.
No, you were not "stupid", and no one is claiming that!! In fact, Docker has gone out of their way to publicly apologize for saying something that was so inherently confusing that it led people--including myself, btw, and I don't even like or use Docker--to believe something they did not mean (that public image would be deleted). If they had written an article saying "we're sorry you're so dumb" I'd get your vitriol, but they are actively apologizing for their poor communication and trying to fall on their sword for causing a misunderstanding... what more do you want?!
Demanding human treatment in all circumstances constitutes a healthy outlook on life. Letting organizations get away with shady tactics should never be tolerated in any society. It may not change anything but it's still important for society to speak out against bad policies and bad decisions
Keeping them read only is literally the worst solution. Old images that can't be updated and accrue security flaws, all while uninformed users see address still work and assume nothing needs to be changed.
Your corporation picked literally worst way to do it.
>> Your corporation picked literally worst way to do it.
I disagree. The worst way would be to make a blanket decision for all projects on their behalf.
This way they let the project maintainer decide.
For projects that don't get updated, it's better to leave them where they are.
For projects that are changing the maintainers can choose to delete (or move to a paid / OSS plan).
Choice is good, and giving that choice to maintainers is good.
The final act if goodness (and I'm not clear yet) is whether maintainers will be able to delete an image at some point in the future. Like say a year from now. Possibly by creating a paid account, and "reclaiming" that image.
Personally I agree that your advice to delete them may be the best option for most maintainers who have decided to leave. And they currently have the ability to do that.
Hence my assertion that your statement is incorrect.
You didn't address the issue of security. The problem with leaving it up to the projects is that projects won't necessarily respond, and we don't want the foundations of the next Mantis 26M rps botnet to get its start from PULL insecure:latest.
I wonder what a court would think about who'd be legally liable there?
BigCo or GovDepartment gets popped via a known exploit against a fixed bug in an OSS project, but GitHub has prohibited the project from updating the explicable image they host without paying a ransom of $420/year?
Can the images be updated after the organizational data is gone? If not, is there a security concern, since vulns are likely to be discovery in future?
sooooooooooooooooo orgs that didn't want to upgrade are still left with users pinned to old address of the image with no option to push security updates?
(Docker DevRel team here) For clarification, the program grants one year of access, but is indefinitely renewable as long as the program is still compliant with current criteria.
Thank you for the clarification. For reference, are there any open source projects you've seen that suddenly change course to being paid products and would lose open source status according to Docker? Any Linux distros that suddenly went closed source? Are there any sort of general public licenses that might preclude projects from even doing so?
Say a distro sells laptops with the distro pre-installed while also providing the same image online and might also sell consulting services to companies utilizing the distro.
According to my undestanding of the terms of Docker's opend source program, even if said distro was fully open-source and a run by a non-profit they would be non-eligible to participate in the program.
Is this renewal automatic? Having to go recertify with docker once a year for my tiny open source project that we happen to have an image for is literally infinity percent more hassle than I would like to deal with.
Great question! In the demo app we showed yesterday (source here - https://github.com/second-state/microservice-rust-mysql), the Dockerfile is leveraging a multi-stage build where the first stage builds the Wasm module and the second stage extracts the Wasm module into a "FROM scratch" image.
In Compose, the "server" service is running using the image that is produced by that build. It's then handed off to the new Wasm runtime, where the module is extracted and executed. Hope that helps! Feel free to follow up with more questions!
> So docker+WASM does not build a Linux container image with the WASM application inside but it builds a WASM application packaged as a docker image
With this preview, we are leveraging the OCI specification that defines how to build an image. Linux/Windows containers are the most common use of this specification, but many other types of artifacts exist (OPA policies, Helm charts, etc.).
When using any of these other artifact types, tooling has to know how to use that specific artifact type and run it (eg, extract the image and run it as a container or extract the Helm chart and deploy it). In our Wasm use case, we are doing the same thing... package and ship a Wasm module and we'll extract it and run it on the new Wasm runtime. That runtime then "converts" the Wasm module into native machine code for the OS you're running on.
> If this is the case I'd leave docker outside the name of the technology. I imagine the confusion.
That's great feedback! While most know us as "the container company", our mission doesn't even talk about containers. We want to help all developers succeed by reducing app complexity. We can certainly do more to help educate folks between the different types of workloads you might be running. We're still very early on in this process, so stay tuned (and keep the feedback coming)!
'Docker + WASM' definitely gave me the impression that docker is now leveraging wasm to be able to run your normal container workloads directly on the browser.
Like if you have any docker container, you can now take that, with some modification and run that directly on the browser. Reading further, I think this is totally not what it actually is.
You could say 'Docker for WASM applications' or 'Docker to deploy WASM apps' that would make the relationship more clear I think.
There are a few things we're excited about, many of which we fully recognize are used in marketing/buzzword bingo. But, they really are around portability (compile to a Wasm module and run on any architecture), isolated sandboxing (leveraging much of the research and investment used to protect web browsers), and super fast startups (obviously depends on the app too).
There are a few limitations right now, some of which lack of built-in garbage collection and multi-threading. You also have to use a language that can completely compile to Wasm, making it hard (sometimes impossible) to leverage already built libraries/binaries.
But... the tech is growing rapidly and we're excited for it! Happy to dive in or answer more questions if you'd like!
For serverless usecases, cloudflare etc.. found that it would be faster to just call a function from a wasm binary than spin up a whole docker container [1]. Which basically translated to cost savings.
In the last company I worked for [2], we saw wasm as a way to easily ship user code in a cross platform way. We mostly targeted edge-ml use cases. Wasm allowed us to package all the "libraries"/functions needed to run the user code securely. So users created and tested out their ML apps in browser and deployed them to all the platforms we supported. (Mobile, Browser, Embedded). As an added bonus they could write each of their functions in any language that compiled to wasm, and just ship it as one "app".
This seems like the best reason to use WebAssembly. The benefit is for the system integrator / platform and the value prop is clear vs. any other technology out there
Great question! The promise (and excitement) of Wasm is to have portable, secure, quick-to-start, and low resource usage apps. So, write code in Go, Rust, C++, or any other language that can output to Wasm (up to over 40 now!) and you're good to go! The binaries are super small too. Happy to dive in more too!
Are there realistic benchmarks that quantify these gains?
A first view of the promise reads reminiscent to the JVM days, which promised to solve portability. The main difference is JIT vs. AOT, but hidden behind that remains also the complex management of FS access, threading, spawning, SIMD, GPU, and other unportable differences. While I imagine that the shim would avoid having a whole Linux, what do we lose in the change?
Why not run that go/rust/etc code natively on the machine though? Is there extra sandboxing, network/filesystem virtualization or anything gained by compiling to and running in a WASM environment?
One benefit of using wasm is architecture-agnostic binary. Right now you can't run x86 binary on ARM or vice-versa. So basically you need to build your containers twice if you have macbook people and x86 servers. And technically those are different images, so there's chance that you'll hit some non-trivial difference. With wasm everything could be simpler.
This does not work for me. Java image that takes 30 seconds to build on x86 machine took 40 minutes to build on M1 after which I killed it. So this feature essentially does not exist as it's not usable. I don't think that's how most people use it. I, personally, rent x86 VPS just for docker.
Most images nowadays have ARM version, so probably that's how most people use it.
Yes, we need a compiler for Java. Java sources are compiled to Java bytecode along with many other actions for complex projects like downloading dependencies, generating sources, running tests which might run platform-dependent binaries and so on.
> That was the advantage given for the JVM. The .jar ran on x86 and ARM.
Yes, wasm and Java bytecode are the same in that regard. But Java bytecode failed to get adoption outside of Java world. Wasm might not, we will see. One significant improvement of Wasm over Java is good security story. Java sandbox is well known for its CVEs. Browser wasm implementations are battle-tested on billions of devices in the wild Internet. So you basically can't run untrusted Java bytecode without further boundaries like KVM or at least containers. But you definitely can run untrusted Wasm bytecode because that's what your browser does all the time.
And it is starting an additional emulated x86/amd64 vm that is emulated. The battery takes a big hit. Now imagine you can run wasm that doesn‘t need an extra vm and doesn‘t drain your battery much faster.
I think the # of people losing a lot of of battery to this is miniscule, especially since the OSX x86 emulation is pretty fast.
A bigger peeve is that there is no general reasonably fast cross build ability, to eg build, test and debug ARM images (for runnign on ARM servers) on x86 Linux or Windows etc.
Gained by you running your app on your own server? not much really.
Gained by some "serverless" provider who tries to run multiple apps like yours, in parallel on the same machine? Yes. Less process overhead. Probably, less memory footprint too.
A majority of software already does this. WASM seems more like a direct attempt to run binary blobs and to simply "trust" the intrusive binary sandbox while leaking all kinds of information when trustworthy sandbox solutions and multi architecture compilation are nothing new and not a new problem that needs solving. It seems if anything the problem people are trying to solve with WASM is how to get people enthusiastic about handing over more privacy.
Depends on which "this" you're referring to. We (Docker) are trying to make it easier for developers to use the tools and capabilities they know and love to build, share, and run Wasm applications.
As far as Wasm itself, it's designed to provide a fast, lightweight, secure, and portable binary format. While it was originally designed to help bring native code to the browser, it's quickly spreading to the server side. Many folks are using it for edge/IoT, but it's growing into other areas (saw demos today of even using it in databases as psuedo stored procedures). Happy to dive in more if you have more questions!
I'm honestly really confused about the specific role docker is playing here.
What, exactly, is Docker doing? Is it compiling the application? Is it making a runtime for the wasm binary? Is it being the runtime for the wasm binary, so the end user builds through whatever usual build processes and gets a binary they can then easily run?
If the wasm binary is lightweight and portable, why is docker useful?
edit: Given the other comments about "what problem does this solve" I think maybe the blog post has missed its mark slightly
Yeah. Not trying to be cynical, but this feels like someone at Docker said "we need to have a WASM story" and this is what they came up with. I really don't see the point.
Sorry for the delayed response, but the problem we're trying to help solve is how to help lower the barrier of entry to using Wasm apps leveraging the tool many developers are already using. As an example example, with Docker+Wasm you can use a Dockerfile to build the app in a container image, distribute the Wasm bundle as an OCI artifact, and test it locally using the Wasm runtime. While the ability to build using containers has existed all along, the ability to actually run the Wasm app locally side-by-side your other containers (and leverage the same container network) is new. Hope that helps!
This should be the copy of the announcement. Finally it's clear what it does. The announcement doesn't explain at all what docker + WASM is.
Summary of summary: Dockerfile -> docker build -> docker container with the WASM app and runtime inside -> docker push/pull -> docker run container.
However, even if I never compiled anything to WASM I think that it was already possible to build an image with a WASM container and runtime inside (as for any other language/runtime.) So what's the friction this is removing from the process?
That does help thanks. It's still all a bit abstract to me because I (like many others) are still not familiar with WASM itself so this next step is probably going to go unappreciated for a while.
I'm a bit surprised by several comments in this post talking about how WASM could replace containers but I don't have the context around it
Easy orchestration and deployment is the only thing I can think of. Because of docker's infrastructure.
But tbh if orchestration is really the concern, Docker + wasm seems less efficient than having a dedicated app that can orchestrate multiple wasm modules within the same process. But maybe that's something docker can solve later as the actual requirements emerge.
the only thing I see is that after wrapping wasm inside a docker you can leverage all those docker tools(k8n,etc)?
the docker solves portability and scalability issues with a reasonable overhead, so wasm hides inside docker can benefit from some?
I wonder if it could make testing frontend code that uses WASM (but not DOM) fast and easy since you wouldn’t need to fire up a complete browser environment.
I’m not sure that it’s possible at the moment. In the past when I needed to test a WASM integration, I ended up using that approach and it was kind of a pain not to get immediate feedback on the WASM code’s API tests since it was essentially only testable through the complete integration environment.
I like integration tests, but I like smaller and faster test suites for easing development in some conditions as well.
You can currently test a WASM-targeted API if you use unit tests or other language-level testing approaches, but you won’t get the constraints of the WASM runtime as far as I know. Maybe the lack of garbage collection could be a critical constraint to test against.
I suppose you could even test dynamically linked binaries without a browser as well.
I’m sure there’s far more to it that I’m not aware of, and maybe testing really isn’t that useful of a feature here — I’m just guessing based on my own experience.
Great question! There isn't a way to run Docker directly in the browser. But, there are tools (like Play with Docker at play-with-docker.com) that lets you interact with a CLI in the browser to run commands against a remote cloud instance. I personally use this a lot for demos and workshops!
But... certainly a neat idea to think about what Wasm-based applications could possibly look like/run in the browser!
Hey! Peter from Snaplet here. This is really exciting stuff. We created the OSS postgres-wasm (https://github.com/snaplet/postgres-wasm) example a few weeks ago. An idea I'm playing around with is something like:
Edit.com opens a text-editor and terminal where I have access to the NodeJS binary and a connection string to PostgresQL. Want Redis? Open a new tab at https://redis.com/try, where the connection string will appear in the edit.com tab.
I used https://wasm.supabase.com/ to make sure some SQL commands I was writing for a blog were correct. It was super useful and faster than starting docker desktop, looking for the postgres image name, starting it etc..
I miss a feature where I can share a link with some data/schema pre-seeded (maybe from a gist?)
All three links that you posted appears to be either broken or malicious. Are you just trying to explain a concept using example domain names? Consider ".example" or ".example.com" (see RFC 2606) instead of potentially malicious domains.
Is it possible to sandbox the host system from the guests in WASM?
Are there namespaces and cgroups and SECCOMP and blocking for concurrent hardware access in WASM, or would those kernel protections be effective within a WASM runtime? Do WASM runtimes have subprocess isolation?
- TIL about teh Endokernel: "The Endokernel: Fast, Secure, and Programmable Subprocess Virtualization" (2021)
https://arxiv.org/abs/2108.03705#
> The Endokernel introduces a new virtual machine abstraction for representing subprocess authority, which is enforced by an efficient self-isolating monitor that maps the abstraction to system level objects (processes, threads, files, and signals). We show how the Endokernel can be used to develop specialized separation abstractions using an exokernel-like organization to provide virtual privilege rings, which we use to reorganize and secure NGINX. Our prototype, includes a new syscall monitor, the nexpoline, and explores the tradeoffs of implementing it with diverse mechanisms, including Intel Control Enhancement Technology. Overall, we believe sub-process isolation is a must and that the Endokernel exposes an essential set of abstractions for realizing this in a simple and feasible way.
> gVisor is an application kernel, written in Go, that implements a substantial portion of the Linux system surface. It includes an Open Container Initiative (OCI) runtime called runsc that provides an isolation boundary between the application and the host kernel.
Note that Compose files using the previous model declarations (using providers) will still work.