Future ideas for persistence

[jshaffstall]

Hey Owen, I was wondering what your roadmap looked like for persistence.

At about the same time you were transitioning from the ORM to persistence, I worked on a layer of my own with some specific goals. Persistence achieves some of those but not others. Mine is nowhere near ready for prime-time, partly because of the complexity of some of those goals, and partly because the project I was using it on stalled. Here are a few goals that persistence doesn't seem to hit exactly.

1. Global client-side cache. I like the approach you're taking with persistence, although the cache doesn't contain any objects retrieved. Ideally anything fetched from the server could be retrieved from the cache.

2. Commit/rollback mechanism for local changes made to objects. Allows cancelling changes made locally before they're finalized by the user. It looks like this could be added just by clearing the delta in all objects, although there's not a central spot where that could happen.

3. Detecting stale data. The longer an object lives in the client before being committed, the more likely someone else has already updated it.

4. Defining a schema entirely in code. This was the one that made things most complex. Data tables are a bit more opaque with this one, since the entities are not defined in them. e.g. one data table kept track of entities, another properties, etc. But, it takes away the need to define data tables and associated boiler plate server functions.

Mostly just curious if your future thoughts for persistence included any of those.

[meatballs]

1. True, there's no 'global' cache. Instead, each class handles the cache for its own instance.

although the cache doesn't contain any objects retrieved.

I don't understand what you mean by that. The cache contains anything retrieved by a search unless you explicitly ask for that not to happen.

2. That should be easy to add with a reset method to clear the delta. PRs always welcome!

3. That's next on the list - I'd like to add support for offline data access using indexed db and a services worker. That will involve writing caching strategies to handle stale data and all the other sync wrinkles.

4. I think that's more effort that it's worth. Setting up data tables for an app is a one-off effort and, even using the IDE, it's not that onerous. If I wanted to automate it, I'd probably do something within anvil.yaml

[jshaffstall]

Sorry, typo, that should have been, "the cache doesn't contain all objects retrieved". The cache gets cleared on a search, so even though the client may have references to previously searched objects, only the most recently searched objects are in the cache. It looks like generator fetched objects aren't in the cache at all.

There's not a "cache back to the beginning of the app" sort of cache, and there's no mechanism for searching/filtering the local cache. So the question was whether a more general sort of "cache everything" type of cache was planned.

[meatballs]

There's no plan right now but, if somebody wanted to make an option for a more sophisticated cache, I'd be happy to look at it.

[jshaffstall]

If I can come up with a project to use it for, I'll play with persistence as it is, and see how a more general cache could be worked into it. Going along with that would be batching up a series of persistence operations into a single server call (e.g. create a bunch of objects and add them all at once).

[meatballs]

Well, there's nothing to stop you adding a method to a persisted class that makes a server call and does anything you want with the results.