Honestly, it was during my deep dive into p2p networking a year back that I started scratching my head and thought "Huh, you could pretty much replace most of the reasons you'd need Kubernetes and gRPC and Consul with this and make coding 10x simpler..."
I only got to finally try it out just two months ago when Mozilla opened up a program for building MVPs that aim to fix problems with the Internet. Big thanks to them for that!
One of the biggest frustrations I've had prying through Darklang [1], Firebase [2], Supabase [3], and a slew of other low-code projects is that although they do help me build backends faster, they come with a big tradeoff of having to deal with either (a) vendor lock-in, (b) tie-in to specific tools/databases/coding patterns, or (c) a large monthly bill.
At the same time, coding without these tools, I would have to deal with either configuration hell [4] with Kubernetes/Istio/Consul or take the time to stitch a bunch of libraries together and spend a lot of time having to deal with devops/networking/backend architecture myself.
So, I built this framework called Flatend over the last two months for Mozilla Spring MVP Labs, which:
1) lets you write functions using any tools/databases/programming languages/frameworks/hosting environments, and
2) turn them into microservices stitched together into a mesh network with service discovery/load balancing/routing using the Kademlia p2p overlay networking protocol popularized in BitTorrent/Ethereum [5].
Taking advantage of p2p networking, Flatend has taken off a lot of the pains and time spent dealing with bidirectional RPC, service discovery, load balancing, scaling, deployment patterns, and end-to-end-encryption, and has led me to spending more of my time writing productive business logic code in NodeJS/Go.
Using Flatend I've now deployed four different sites/apps racking roughly 200k+ visitors on two 5$ DigitalOcean instances, doing red/blue deployments and upgrades to my microservices really easily.
But that being said, Flatend still is very much experimental and is a heavy work in progress.
Nonetheless, I still do hope for Flatend to be immeasurably useful for the people here at HN looking to speed up the backend development of their next product/service.
I appreciate the dev-friendliness of this (libp2p leaves a lot to be desired last I looked from a dev perspective). How chatty is it? How resilient is it to untrusted peers (e.g. any of skademlia features built in)? Any plans to let the user provide their own net.Conns or io.Reader/Writers (e.g. I might want to run it over Tor)?
All communication is quiescent unless some given node A sends data to node B (apart from TCP keepalive packets of course). For routing messages, it is expected O(log n) much like ordinary Kademlia.
There isn't much resilience carefully integrated into the system for untrusted peers (complete S/Kademlia is in progress though! Static/dynamic crypto puzzles are fully implemented to prevent Sybil attacks on IDs, and the FIND_NODE RPC call is implemented to meet he guarantees documented in S/Kademlia's paper), since this framework is intended to be used in a closed environment for a set of microservices.
The underlying library for networking, Monte, fully supports providing your own net.Conn's and io.Reader/Writers - all you need to do is return your modified net.Conn via implementing the monte.Handshaker interface here: https://github.com/lithdew/monte/blob/master/net.go#L36
Monte will everything else related to bidirectional RPC management. Details of the bare minimum Monte prefixes to your packets is documented on the README for it.
I might be a bit dense, but I don't see what's p2p about Flatend. You have a service that connects to an API gateway by hardcoding its address, then the client queries the gateway directly, and the service replies. Where is Kademlia used here ?
Not trying to be negative, I'm just trying to understand
No worries: Flatend uses Kademlia for helping developers bootstrap the network, and to have nodes discover services from one another.
To clarify, the API gateway I provided is just* another node/peer in the network which advertises a few services that deal with exposing other nodes' services via HTTP/WebSocket/gRPC (rather than an ordinary API gateway server).
This means that you can run several API gateway nodes on different machines, and kill several of them, and they would still discover and process one another's request/data without any extra work, as they are independent nodes that are not directly tied by their addresses to other nodes/microservices.
So, I guess the easiest way to point out the p2p aspect here is on how Flatend deals with the separation of concerns across "services"/nodes.
I think there should be a way to show the p2p aspect in the example; maybe have the go service and the node service talk to each other, without hardcoding anything ? Just to show how it leverages the whole p2p thing
I'm with you on this. I was working on something similar but just found it was a bit early for the P2P networking side of things. We're focusing on cloud first and driving local to platform development and will come back around to the P2P in a few years. A fan of anyone who can make it work.
I only got to finally try it out just two months ago when Mozilla opened up a program for building MVPs that aim to fix problems with the Internet. Big thanks to them for that!