Dexie Cloud Productionizing Suggestions

[dusty-phillips]

Didn't want to file an issue for this as this is pretty much a grab bag and I'm sure you have most of these ideas in mind already. Dexie Cloud has been working great for me for the past few weeks, and I'm starting to imagine what my service would look like in a future of real users. Here is an unordered list of ideas/requests/suggestions:

• Database names or aliases when running dexie-cloud databases and dexie-cloud connect, etc. I will want a development database, a staging database, a production database, etc, and it's not easy to tell the current list apart.
• "Superuser" tooling. I want to be able to view a specific user's data stored data and modify it, especially when the DB gets into an unexpected state. Probably a crud-style admin. Maybe a way to impersonate an existing user.
• Billing features. It's not exactly in dexie-cloud's wheelhouse, but it kind of fits in with the "auth works out of the box" idea. I'd like a "subscriptions out of the box" kind of system, where dexie-cloud (or a partner) automatically takes care of billing for a logged-in user and I just need to set a couple flags in my config to enable stuff like free trials and recurring billing.
• Data locality so that users in EU can have their stuff stored in EU datacentres, users in Canada can have their stuff stored in Canadian datacentres, etc. I can imagine this becoming exceptionally complicated in apps that use Realms to share data between e.g EU and US users.
• Invite-only acc control on the default authentication. As I start to release to test users, I don't want random people able to sign up and log in with OTP.
• Published privacy policy and SLA.
• Updated ETA for the production release of Dexie Cloud. (Probably a blog post is overdue; last was in April or something)

That's it for now, but I'm sure I'll have more suggestions to drop in this thread in the future!

[dfahlander]

Thanks for all the suggestions.
Some of the requirements are easier while some others a bit more challenging.

The billing part would be a great thing but it could involve some legal challenges. Maybe a future partner solution could fix it.

Data locality - as you also say, it would be hard to store things from the same realm on different continents. But tying a database to a region could be possible. We're not really at the point to offer Dexie Cloud in multiple regions yet but it's an interesting aspect. The service is currently hosted in the US. A standard answer would be to buy the server software and install it on customer's own cloud provider in the desired region.

The rest of the requirements are more easily solved and I will definitely have them in thought while approaching production release:

• Database aliasing/naming
• Superuser tooling
• Invite-only access control
• Published privacy policy and SLA

Looking forward to having them implemented.

Regarding an updated ETA for production release - I hear you! The last blog post in February suggested a production release around end of October this year (+ / - 3 months). This is still the target. Maybe more "+" than "-" though.

[dusty-phillips]

I guess at a higher level with the subscription thing I’m looking for a way to store metadata about a user or their db objects that is not under their control. Something like what feature plan they are subscribed to. Currently I’d need to put this in a separate dB (probably fauna, since I would be hosting it on Netlify and they have an add on for fauna. Incidentally a netlify addon for dexie-cloud would probably drive a lot of traffic).

and for that to work, I need a way to associate the dexie user with the on in my “metadata” db. So maybe some kind of web hook? Or some sort of “dexie cloud functions” like aws lambda?

I think right now dexie otp auth uses the email as an identifier. It would be better for this to be an opaque id, since an email is PII, and I’d rather minimize the places PII gets stored.

Maybe this can be handled by realms? Could I have a realm that is read only to the user and configured from a payment gateway?

These are all just musings; as it sits, I think the only viable solution would be to set up dexie with third party auth (netlify identity) and then have some logic going through netlify functions. It would work, but I really like the idea of dexie zero config auth and I hope it can work for wider use cases.

As a tangential high level discussion, I’m not even sure if subscriptions make sense for offline-first apps. I could add logic that prevents access to the app if not subscribed but it would essentially just be a flag that a not-too-sophisticated hacker could disable in the console. But I can’t really see any other way to monitize such an app other than wrapping it up in electron and publishing to an OS-level App Store. So I’m thinking “free for one device, because t if you want sync you need to subscribe, and dexie cloud is the obvious intermediary to gate access to sync. Not specific to dexie cloud of course, but but that’s why I posted a discussion instead of an issue!

[dfahlander]

You can use the current CLI to update individual objects without replacing the entire dataset. By doing so from some kind of backend service, you can create realms that are readonly for the user but that your service can update. The rest endpoint isnt documented but if you use the CLI (dexie-cloud import) with the --verbose flag, you will see what network requests are being generated.

A subscriber tool will be released at some point, where you can write a node daemon or similar that subscribes to all or certain changes in realtime and can act on them in a desired way - similar to a web hook but with a little less traffic as it will use a server-to-server websocket connection. Actions could be things like sending push messages, emails etc.

Here's an example import request to create a readonly realm with some data in it, for [email protected]:

import-file.json

Example import file to use with dexie-cloud import. The schema syntax is the same as the one for db.version(x).stores({...}) but you only need to specify the primary key of each table, and you only need to specify tables that are being modified in the "data" section. The schema itself is not updated - it is only needed for the import endpoint to know the primary key property of the given data.

{
  "schema": {
    "realms": "@realmId",
    "members": "@id",
    "subscriptions": "@id"
  },
  "data": {
    "[email protected]": {
      "realms": {
        "[email protected]": {
          "name": "Readonly realm for [email protected]"
        }
      },
      "members": {
        "mmb-unique-id-for-this-object": {
          "userId": "[email protected]",
          "invite": false,
          "accepted": {"$t":"Date","v":"2021-09-14T00:00:00.000Z"},
          "permissions": {}
        }
      },
      "subscriptions": {
        "sbs-unique-id-for-this-subscription": {
          "subscriptionType": "evaluation",
          "xxx": "yyy"
        }
      }
    }
  }
}

The json is dry coded so there might be some typo. But if you'd run:

npx dexie-cloud import import-file.json --verbose

... then it will insert or update the three objects:

• The realm with id "[email protected]"
• The member relation that ties the realm to the user with zero permissions.
• The "subscription" object with the data you want to put on it.

By using the --verbose flag you should also see the POST request to see the rest API. These are things that will be documented later on. The import will replace or insert objects but only delete objects if they are explicitly set to null in the JSON data.

What's currently not very obvious is how to request partial data. The export argument to the CLI will export the entire database. You can impersonate a user (as being examplified in the docs for how to do custom authentication. You can omit publicKey as the service may not need any refresh token. Using your dexie cloud client-id and client-secret you request an arbritary user token) and then call the sync endpoint to read all data for a particular user (can be reverse engineered using devtools when your normal client does its initial sync call) - but that API still returns the entire dataset for that user - not possible to limit the query to certain realms, tables, ids or criterias.

Just wanted to explain what is possible at current version. Might not be exactly what you were looking for but just so you have an idea what could be done.

I really understand issue of being able to control user subscriptions more than what is possible in the current zero-auth configuration. As you already mentioned, anyone can sign-up - which might not be desired. Having invite-only solves part of that. Another option would be a web hook for controlling sign-ups and accepting or rejecting them, or having some built-in support for free trial accounts that dexie cloud itself would control the expiration of. Ideas are welcome! But as you say - both of these would also be possible by using a custom auth instead.

User-ids can be opaque if using custom authentication. Could have chosen to do that in the zero-auth mode but it haven't decided if that is good or bad. Maybe an option.

[dusty-phillips]

Was thinking about this a bit more:

A subscriber tool will be released at some point, where you can write a node daemon or similar that subscribes to all or certain changes in realtime and can act on them in a desired way - similar to a web hook but with a little less traffic as it will use a server-to-server websocket connection. Actions could be things like sending push messages, emails etc.

I like the idea, but I think I would also like a web hook option, as it would fit in better with a serverless architecture. I’d rather hit a lambda on demand than have to keep a service running full time. I’d also be nervous about websockets in case the service went down and a message was missed (but I assume you’d have retries and queuing for that).

[dusty-phillips]

Nice, this is more than enough for me to work with. Exploring and understanding realms were next on my todo list, so this will be a good intro to that. Will try to break as many things as possible before you exit beta. 😁

FWIW, my general stance with PII is to obfuscate as soon as possible and isolate it to as small a service as possible. It’s kind of like the Unicode or utc sandwich principle.