Much needed! Congrats on the launch, Daniel and Matt.
> Matt and I came to this project after we built similar internal billing systems at previous jobs and we realized how error-prone these systems can be—one incident might have even undercharged a client by a few million dollars!
A BigCloud provider (no points for guessing) I worked for found about how they were undercharging customers due to a bug, and so, they fixed the bug for new customers, but continued to undercharge customers grandfathered in.
> However this allows us to Pull, any custom metrics and dimensions directly from your Datastore.
Most SaaS providers would rather push data than have it pulled, is what I'd imagine. Are you hearing otherwise from folks you've been speaking with? For instance, in serverless environments (which is the poison of choice for me, at least), pull is much harder to accomplish, even where possible.
> All of this data is then processed and sent to our backend usage journal, where we store it in an append-only ledger pattern.
Apparently, a BigCloud, in perhaps a case of NIH, ended up creating a highly-parallel event-queue as a direct result of the scale it was dealing with: https://archive.is/IUKvT Curious to hear how you deal with the barrage of multi-dimensional events?
> Additionally, we also help you understand your cost to serve your clients’ usage, and this data allows us to provide your SaaS with usage based billing.
2 cents: Fly.io Machines is a tremendous platform atop which I fully expect businesses to build multiple successful SaaS products; may be that's one niche for you folks to focus on and own.
> We bill based on invoiced revenue (surprise surprise its usage based) and we have a platform fee, roughly it breaks down to 1% of invoiced revenue on Paigo.
This sounds a bit steep. I know for a fact that togai.com are also in private beta (their choice of datastore is TimescaleDB, and event-store is NATS), but unsure what their pricing model is; I'd be surprised if it is the same as paigo's.
I am not cloud provider but in Subscriptions based carpooling. And similar situations we were in.
Point is if existing happy early customers - no point in hiking price due to mistake at our side (Be It our, or our tech vendor or tech integration partner!) provided you are making money and ramen profitable from those early users. :)
> Most SaaS providers would rather push data than have it pulled, is what I'd imagine. Are you hearing otherwise from folks you've been speaking with? For instance, in serverless environments (which is the poison of choice for me, at least), pull is much harder to accomplish, even where possible.
We totally offer Push based as well, all of our workers are just using the same API endpoint to push the data we collect to, its just not the strong highlight since other providers have push offered.
We went down the pull path since during our discovery process about 6 months ago, we were chatting with some DB and Infra companies who just built out and integrated with a billing provider. All of them mentioned being annoyed with the amount of engineering commitment it took for them to measure, persist, and then transmit the usage data to the provider for them to handle the rest. So we wanted to offer a pull based solution to help with this need.
You're totally right that its architecture dependent, and we don't want to cause a huge load (and cost) on a serverless platform. So for some dimensions, push is definitely an option.
> Apparently, a BigCloud, in perhaps a case of NIH, ended up creating a highly-parallel event-queue as a direct result of the scale it was dealing with: https://archive.is/IUKvT Curious to hear how you deal with the barrage of multi-dimensional events?
So to dive into more technical detail, we have an event queue where our workers drop data off to and then it gets persisted into our ledger, by workers reading from the queue. We have the queue hosted by a major cloud platform and its offered as a managed service, similar to kinesis.
For the different dimensions, we have a standard dataformat they need to be in before we can persist them, this transformation typically occurs on the client side, though in some cases we can transform the data from an open standard format (Prometheus https://prometheus.io/docs/concepts/data_model/) to our backend format.
At a criminally high level, this data format consists of a measurement, a value, a field, and a set of metadata tags. Our ledger is built on a schema-less Timeseries DB, so it doesn't matter if the same measurement has a different set of metadata from another. This gives us a boat ton of flexibility when it comes to how we want to query data.
For the different types of dimensions and their different data types, it becomes an issue when wanting to aggregate on them. For instance, a you may want the Total of one dimension, however Average and Count wouldn't make any sense.
To get by this, clients need to tell us what Aggregation method to use per dimension.
This really isn't present in the demo, since its a fairly simplistic version of the whole app, but its a requirement we have implemented into the API.
> 2 cents: Fly.io Machines is a tremendous platform
Thanks for the hot tip, looks awesome! I'll definitely check this out and how we can sneak it into our product.
> Matt and I came to this project after we built similar internal billing systems at previous jobs and we realized how error-prone these systems can be—one incident might have even undercharged a client by a few million dollars!
A BigCloud provider (no points for guessing) I worked for found about how they were undercharging customers due to a bug, and so, they fixed the bug for new customers, but continued to undercharge customers grandfathered in.
> However this allows us to Pull, any custom metrics and dimensions directly from your Datastore.
Most SaaS providers would rather push data than have it pulled, is what I'd imagine. Are you hearing otherwise from folks you've been speaking with? For instance, in serverless environments (which is the poison of choice for me, at least), pull is much harder to accomplish, even where possible.
> All of this data is then processed and sent to our backend usage journal, where we store it in an append-only ledger pattern.
Apparently, a BigCloud, in perhaps a case of NIH, ended up creating a highly-parallel event-queue as a direct result of the scale it was dealing with: https://archive.is/IUKvT Curious to hear how you deal with the barrage of multi-dimensional events?
> Additionally, we also help you understand your cost to serve your clients’ usage, and this data allows us to provide your SaaS with usage based billing.
2 cents: Fly.io Machines is a tremendous platform atop which I fully expect businesses to build multiple successful SaaS products; may be that's one niche for you folks to focus on and own.
> We bill based on invoiced revenue (surprise surprise its usage based) and we have a platform fee, roughly it breaks down to 1% of invoiced revenue on Paigo.
This sounds a bit steep. I know for a fact that togai.com are also in private beta (their choice of datastore is TimescaleDB, and event-store is NATS), but unsure what their pricing model is; I'd be surprised if it is the same as paigo's.