Add support for backup & restore to the service broker api

[avade]

Problem

As a service user I want to have confidence that my service instance is backed up and know that if something bad happens that I can restore from a backup.

As a service operator I want to offer service users a self-service mechanism to perform backup and restores of the service instance, so I don't have to manage this process.

Workflows

As a service user I can trigger a backup for a given service instance

As a service user I can list all backups for a given service instance

As a service user I can restore to an existing service instance

As a service operator I can choose not to offer self-service backups

As a service operator I can choose not to offer self-service restores

---

Next steps

1. Community feedback on this feature
2. Create proposal google document

[duglin]

No strong opinion on this one yet but wanted to add that I thought these types of operations are normally considered to be extensions to the SB spec. Meaning, a service provider may offer these to their users but they would be too specific to include as part of the core API. So, for example, they may be available via the admin/console URL provided by the service. But definitely something we should discuss.

[avade]

I think backup & restore is supported by the majority of services platforms in a generic way for traditional data services [1].

The improvement to the UX for developers vs going to a dashboard that's different for each service / service provider for a common operation like backup / restore is worth extending the API for.

Agree this is a good discussion item for the next call.

[1] Compose.io is a great example: https://help.compose.com/docs/compose-backups

[duglin]

Just need to be careful in assuming that all services are "data services". I have no idea what % fall into the "data service" category and which don't but I do know that its not 100%.

[avade]

Good point, I would imagine this feature would be similar to bindable in the API today. This is an opt-in feature and brokers that don't support binding do not need to implement the endpoint.

[angarg12]

@avade does this assume that SB is the one delivering the service?

In our use case SB is a gateway to different service providers, so effectively the only state stored in the broker is the instances and bindings information.

In our system this would be implemented via "events". We expose an API where users can send events to service instances. Backup is a perfect example of such an event.

That solution works for us because different providers offer different events (i.e. not all services provide backup) so we keep the API consistent all across the board.

[gberche-orange]

Community feedback on this feature

I have no idea what % fall into the "data service" category and which don't but I do know that its not 100%.

Support for additional life cycle events beyond the current CRUD (Create Read(Bind) Update, Delete) is definitively needed for different use-cases, including:

• backup/restore:
  • see related dingo pgsql requests by @drnic
  • db-dumper-service which which currently works around the issue through arbitrary params conveying side effect semantics in the update/bind verbs and end up providing a custom, non-portable UX through custom CLI plugin
• other services such as autosleep that have diverse needs such as toggling state of services, and would benefit from such enhancements, see related discussion and UX comparison. I'll try to provide some more examples.

An alternative to a normalized backup verb in the SB API, is the notion of service broker action proposed by @benlaplanche . The SB declares a set of additional actions supported (e.g. in a similar way than #59 with a JSON schema) that platform UIs (e.g. CF CLI, or Web UIs) can discover and use to expose extended actions in a uniform ways to platform users.

In our system this would be implemented via "events". We expose an API where users can send events to service instances. Backup is a perfect example of such an event.

@angarg12
The events concept seems interesting to detail, especially if this could be a concept also to also support other use-cases such as broker to app notification events: credentials renewal/rotation, usage quota reaching limit....

Could you please share related spec URL ? How do you envision adapting this to the SB API ? How would this translate into UX, ie how does this gets leveraged in UIs (CLIs and web UI) ?

[avade]

@gberche-orange The broker actions concept is interesting, but it does mean that we would have to rely on standards for broker action types. For example, broker A calls backups action take-backup broker B calls backups action trigger-snapshot, which is confusing for developers. A schema with descriptions would go some way to give clarity to users.

As always the more generic the spec the more decisions we push onto consumers and broker authors. Adding backup & restore as a first class concept would be taking an opinion that it's something the majority of services would want to support with a standard UX for consumers.

@angarg12 how do service users submit events to the broker via CF? As @gberche-orange said, a spec / example that we could see would be helpful.

[paulcwarren]

Alex, this would be useful for persistence too, I think. A developer with an app reading/writing to an NFS volume may very well want to back up that NFS volume, for example. And later on restore it.

[gberche-orange]

@avade

The broker actions concept is interesting, but it does mean that we would have to rely on standards for broker action types.

The same holds for catalog normalization, where CF community-driven-standards appeared to be both flexible and lightweight without putting that much load on service authors.

Adding backup & restore as a first class concept would be taking an opinion that it's something the majority of services would want to support with a standard UX for consumers.

We would need to detail the variations of backups that are required to support the different use-cases:

• backups vs exports (that can be imported somewhere else, such as prod to non-prod)
• data anonymization during exports
• one-time backup vs scheduled backups
  ...

In our experience with db-dumper-service and its cli-plugin two "backup" and "restore" verbs are too limited: we need verbs to list/delete/download backups. We also plan to introduce additional verbs to manage backup stores (a container for backups with their quotas and eviction policies...).

I believe that it would be more useful to support additional generic verbs, than try to normalize the many backup/restore variations which only benefits/applies to data services as @duglin reminded, and would leave lots of other services (e.g. route services such as api gateways) without the needed extended verbs they need.

[angarg12]

@gberche-orange @avade for the record I have asked my colleagues to provide me with the spec/examples on how these actions are implemented in our system. I am waiting for a reply, I will post details once I get them.

[avade]

@gberche-orange good points. I am starting to agree that broker actions may be the way to go. It keeps the API simple but extensible for advanced use cases. We can offer a "community standard" guide as part of the spec, similar to CF metadata.

How about we come up with a 2-4 page proposal for broker actions and reference some of the use cases in the document from @benlaplanche [1]. I have started a new issue for this here [2].

We can then discuss on the call next week if you can join?

[1] https://docs.google.com/document/d/1Y5vwWjvaUIwHI76XU63cAS8xEOJvN69-cNoCQRqLPqU/edit#heading=h.kzfeyua6md59
[2] #114

[paulcwarren]

+1 for broker actions also. We have this story on our backlog that sounds very similar.

[gberche-orange]

thanks @avade sounds great. Proceeding conversation in #114

[mattmcneeney]

@avade Do you want to close this one now that it's being tracked in #114?

[avade]

Yep, added a link in #114 to this thread.