Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service broker can declare supported schema for parameters field #165

Merged
merged 1 commit into from
Sep 8, 2017
Merged

Service broker can declare supported schema for parameters field #165

merged 1 commit into from
Sep 8, 2017

Conversation

mattmcneeney
Copy link
Contributor

See issue #59.

@cfdreddbot
Copy link

Hey mattmcneeney!

Thanks for submitting this pull request! I'm here to inform the recipients of the pull request that you and the commit authors have already signed the CLA.

spec.md Outdated
| [schemas](#SObject) | object | Schema definitions for service instances and bindings for the plan. |


##### Schema Object <a name="SObject"></a> #####
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Markdown automatically creates anchors for you - do you still need this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope; I was just being consistent with the current spec (I know there is an outstanding PR that addresses this). I'll fix in this branch!

@duglin
Copy link
Contributor

duglin commented May 4, 2017

rebase needed (and squash too)

@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented May 4, 2017

@duglin how's this? Not sure if I missed any new RFC keywords.

Copy link
Contributor

@Samze Samze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON schema types must be lowercase

spec.md Outdated
"properties": {
"billing-account": {
"description": "Billing account number used to charge use of shared fake server.",
"type": "String"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"String" should be lowercase "string"

spec.md Outdated
"properties": {
"billing-account": {
"description": "Billing account number used to charge use of shared fake server.",
"type": "String"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"String" should be lowercase "string"

spec.md Outdated
| bindable* | boolean | Specifies whether instances of the service can be bound to applications. This specifies the default for all plans of this service. Plans can override this field (see [Plan Object](#PObject)). |
| metadata | JSON object | An opaque object of metadata for a service offering. Controller treats this as a blob. Note that there are (conventions)[https://docs.cloudfoundry.org/services/catalog-metadata.html] in existing brokers and controllers for fields that aid in the display of catalog data. |
| [dashboard_client](#DObject) | object | Contains the data necessary to activate the Dashboard SSO feature for this service |
| bindable | boolean | Specifies whether instances of the service plan can be bound to applications. This field is OPTIONAL. If specified, this takes precedence over the `bindable` attribute of the service. If not specified, the default is derived from the service. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we allowed to make this optional? Doesn't that break backwards compat?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... of the service. (3rd sentence) - did you mean "plan" instead of "service" here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure where this came from. Rebase mess up I think. Sorting now...

spec.md Outdated

| Response field | Type | Description |
|---|---|---|
| id* | string | An identifier used to correlate this plan in future requests to the broker. This MUST be globally unique within a platform marketplace. Using a GUID is RECOMMENDED. |
| name* | string | The CLI-friendly name of the plan. MUST be unique within the service. All lowercase, no spaces. |
| description* | string | A short description of the plan. |
| metadata | JSON object | An opaque object of metadata for a service plan. Controller treats this as a blob. Note that there are (conventions)[https://docs.cloudfoundry.org/services/catalog-metadata.html] in existing brokers and controllers for fields that aid in the display of catalog data. |
| metadata | object | A list of metadata for a service plan. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why drop the note? Perhaps we should move that note into the spec?

spec.md Outdated
@@ -568,7 +644,7 @@ the resource it creates.
| plan_id* | string | ID of the plan from the catalog. |
| app_guid | string | Deprecated in favor of <code>bind\_resource.app\_guid</code>. GUID of an application associated with the binding to be created. |
| bind_resource | JSON object | A JSON object that contains data for platform resources associated with the binding to be created. Current valid values include <code>app\_guid</code> for [credentials](#types-of-binding) and <code>route</code> for [route services](#route_services). |
| parameters | JSON object | Configuration options for the service binding. An opaque object, controller treats this as a blob. | |
| parameters | JSON object | Configuration options for the service binding. | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's an extra | at the end - was there before too.

@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented May 9, 2017 via email

spec.md Outdated
| free | boolean | When false, instances of this plan have a cost. The default is true |
| bindable | boolean | Specifies whether instances of the service plan can be bound to applications. This field is OPTIONAL. If specified, this takes precedence over the <tt>bindable</tt> attribute of the service. If not specified, the default is derived from the service. |
| bindable | boolean | Specifies whether instances of the service plan can be bound to applications. This field is optional. If specified, this takes precedence over the `bindable` attribute of the service. If not specified, the default is derived from the service. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/optional/OPTIONAL/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed this. It shouldn't have been added in this PR.

spec.md Outdated
| free | boolean | When false, instances of this plan have a cost. The default is true |
| bindable | boolean | Specifies whether instances of the service plan can be bound to applications. This field is OPTIONAL. If specified, this takes precedence over the <tt>bindable</tt> attribute of the service. If not specified, the default is derived from the service. |
| bindable | boolean | Specifies whether instances of the service plan can be bound to applications. This field is OPTIONAL. If specified, this takes precedence over the `bindable` attribute of the service. If not specified, the default is derived from the service. |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is left alone now @duglin

Copy link
Contributor

@pmorie pmorie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this PR is not likely to be merged soon, I think we should move strictly formatting based changes (or as many as is easily feasible) into another PR. I think there are just a couple:

  1. The name of the plan-object anchor changing
  2. Changes to remove unneeded formatting for headers
  3. Spacing changes

I could probably do this for you fairly easily (using git interactive staging with tig status) and give you a branch that has the transformation in it, if you're interested. I'm also happy to show you my git tricks.

As-is, it's not clear to me where the formatting changes stop and the changes in the spec begin. In general I think we should avoid formatting changes in proposal PRs because they will take much longer to merge than normal formatting changes. That said, I can fully appreciate how hard it can be to avoid mixing formatting changes in :)

spec.md Outdated
@@ -568,7 +644,7 @@ the resource it creates.
| plan_id* | string | ID of the plan from the catalog. |
| app_guid | string | Deprecated in favor of <code>bind\_resource.app\_guid</code>. GUID of an application associated with the binding to be created. |
| bind_resource | JSON object | A JSON object that contains data for platform resources associated with the binding to be created. Current valid values include <code>app\_guid</code> for [credentials](#types-of-binding) and <code>route</code> for [route services](#route_services). |
| parameters | JSON object | Configuration options for the service binding. An opaque object, controller treats this as a blob. Brokers SHOULD ensure that the client has provided valid configuration parameters and values for the operation. | |
| parameters | JSON object | Configuration options for the service binding. Brokers SHOULD ensure that the client has provided valid configuration parameters and values for the operation. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an editing/format change or related to this PR?

@mattmcneeney
Copy link
Contributor Author

Totally agree @pmorie. I'll remove formatting changes from this now, and then we can hopefully merge #192 and then rebase this one with the fixed formatting.

@mattmcneeney
Copy link
Contributor Author

This PR has been updated following last weeks' conversations. But we will need to add more to this surrounding the "meta schema" work.

@angarg12
Copy link
Contributor

Will you push the changes to this PR, or would you rather merge this one and start a new conversation in a follow up?

@mattmcneeney
Copy link
Contributor Author

@angarg12 I think we can continue in this PR, it'll just need some more changes before we review

@mattmcneeney mattmcneeney mentioned this pull request May 23, 2017
@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented May 23, 2017

OK, so @Samze and @ablease have been doing some validation around the new schemas proposal and have some interesting feedback for us:

  • Resource-level schemas are:
    • Good for UIs, as they are able to show all config parameters for a service or binding, including those that cannot be changed
    • But makes validation very difficult
      • Validation libraries don't support meta-schems or the additional creatable and updateble flags
      • How do we deal with required on create but not required on update
  • Method-level schemas (original proposal) are:
    • OK for UIs, but they have to be clever when prepopulating forms and can't easily show other resource information (unless we have a separate schema for that)
    • Good for validation libraries (you just have to pass in the create or update schema with the parameters and any library will give you a valid/invalid response)

We know that there will be few UIs to many service brokers, so it's important to keep service brokers easy to implement. Resource-level schemas do not do this, as it makes validation tricky and means broker authors have to understand how to apply the meta-schema to their resource schemas. Method-level schemas are easy to implement, may have duplication (but we don't think this is an important issue), and make building UIs a bit harder, but there are workarounds that don't require super complex logic.

So, in conclusion, we think we should continue with the proposal that was in place before the face-to-face last week. We can talk about this on today's call but wanted to give you all the heads up in case you had time to think about this before the call. Thanks!

@avade avade modified the milestone: 2.12 May 24, 2017
@avade avade modified the milestones: 2.12, 2.13 May 26, 2017
@pmorie
Copy link
Contributor

pmorie commented May 27, 2017

@jwforres, what are your thoughts about the feedback here: #165 (comment)

@jwforres
Copy link

From our perspective we have the same validation concerns as a broker as far as client-side validation of JSON schema. If we try to combine create vs update into the same schema and distinguish between what is required then we are overcomplicating things. There are already great client-side JSON schema form builders and validators, we shouldn't unnecessarily re-invent things in that space. For that reason alone I would agree that sticking with method-level schema makes the most sense.

@mattmcneeney can you clarify what UI benefits you are thinking of that you get with resource-level schema instead. I could maybe see wanting to show parameters that were set during create as read-only fields during an update, which would require knowing which fields were there during create.

@mattmcneeney
Copy link
Contributor Author

@jwforres There were two benefits that we could think of adding schemes at the resource level. The first is what you have mentioned; being able to 'easily' build a form that showed parameters that were set at create time but weren't updatable. The second was that other resource level fields could be added to the schema (i.e. dashboard URL).

However, our investigations showed that combining the schemes didn't make building forms easier for UI teams and made validation much more difficult. Since we want many brokers to add parameter validation (and platforms too) we believe we need to make life easy for broker authors, which we think is method attached schemes.

@mattmcneeney mattmcneeney self-assigned this Aug 31, 2017
Copy link

@skaegi skaegi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mattmcneeney
Copy link
Contributor Author

Alright, if everyone is happy, I think we can move this into review!
@duglin @shalako @pmorie @angarg12 @fmui @vaikas-google @avade

profile.md Outdated

#### Cloud Foundry

The following restrictions are enforced within a Cloud Foundry deployment:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, these are requirements on brokers right? So, are we saying that a broker that wants to be used by both Kube and CF may need to have different schema? I guess I'm asking whether these CF requirements are really CF specific or if they should be true for all platforms/brokers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping they wouldn't be clashing requirements - just additive things like fields that should be expressed (rather than "Broker authors must not do ...").

profile.md Outdated
```
"type": "object"
```
- The `$schema` field MUST be included at the top level of any provided schema.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't we talk about $schema being optional but then letting the platform pick a default? Or am I mixing up conversations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We did, but in CF we really want to know the exact version so we can tell our validation library which version to validate against. I put this in profile.md to save having this requirement in the core spec, though I expect most broker authors to add the $schema key normally anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obviously happy to move both of these to the core spec if everyone agrees with them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duglin do you have any further thoughts on this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer for it to be consistent between platforms otherwise we'll have an interop issue, and if we do that then we might as well just put it in the core spec, no?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we want to be consistent here and think we should consider making appropriate changes to the OSB spec...

After reflecting, the feedback I gave earlier around handling multiple spec versions by inference was ultimately not good. Leaving it up to the platform to infer the JSON Schema version is just not of sufficient value to special case in the spec; it only leads to unexpected behaviour. So... IF an author would like their parameters to be validated according to JSON Schema the contents of "parameters" MUST be a valid JSON Schema document (with a valid "$schema" property value that the platform understands) or else be empty.

Beyond stating that the document must be a valid JSON Schema document I would not stipulate "type": "object" on the schema as this requirement will vary with the schema version. e.g. in v4 only "type": "object" is permitted but in v6 the meta-scheme permits "type": "boolean, object", and who knows what future versions might say here. I would leave this detail up to the schema spec and not in OSB.

spec.md Outdated

| Response field | Type | Description |
| --- | --- | --- |
| parameters | JSON schema object | The schema definition for the input parameters. Platforms MUST support at least [JSON Schema draft v4](http://json-schema.org/), but SHOULD be prepared to support multiple versions. If no schema version is provided by the broker author using `$schema`, platforms MUST minimally interpret definitions using [JSON Schema draft v4](http://json-schema.org/), and if unsuccessful, MAY try using later JSON Schema versions. Each input parameter is expressed as a property within a JSON object. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this statement about schema versions should be more globally stated and not hidden with this one field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe once we start using schemas for more than just parameters?

spec.md Outdated
@@ -557,7 +629,7 @@ It MUST be the ID a previously provisioned service instance.
| context | object | Contextual data under which the service instance is created. |
| service_id* | string | The ID of the service (from the catalog). MUST be globally unique. MUST be a non-empty string. |
| plan_id | string | The ID of the plan (from the catalog) for which the service instance has been requested. MUST be unique within the service. If present, MUST be a non-empty string. If this field is not present in the request message, then the broker MUST NOT change the plan of the instance as a result of this request. |
| parameters | JSON object | Configuration options for the service instance. An opaque object, controller treats this as a blob. Brokers SHOULD ensure that the client has provided valid configuration parameters and values for the operation. If this field is not present in the request message, then the broker MUST NOT change the parameters of the instance as a result of this request. |
| parameters | JSON object | Configuration options for the service instance. Brokers SHOULD ensure that the client has provided valid configuration parameters and values for the operation. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason you dropped the MUST NOT sentence?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, but it was so longer ago I can't remember the reason, so I'll put it back in!

@avade
Copy link
Contributor

avade commented Sep 6, 2017

Ok, it seems like everyone has had a chance to review and give feedback.

I think this is now in a state that we can merge it into the spec and move forward with the 2.13 release.

LGTM

Approved with PullApprove

profile.md Outdated
```
- The `$schema` field MUST be included at the top level of any provided schema.
- No external references are allowed in schemas.
- Schemas MUST be no larger than 64kB.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some special reason why there is a 64kB restriction?

@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented Sep 6, 2017 via email

@skaegi
Copy link

skaegi commented Sep 6, 2017

No external references is reasonable. +1

I'm less comfortable with the size cap but admit that a 64kB schema seems unlikely at this point - most I have used in our platform are less than 1kB. Perhaps we could say that in a less restrictive way? If a schema is larger than 64kB a platform MAY refuse to register the broker...

@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented Sep 6, 2017 via email

@angarg12
Copy link
Contributor

angarg12 commented Sep 7, 2017

This is a big feature, so instead of trying to overengineer it beforehand I would prefer to get it out there and see how different brokers use it. If further requirements/restrictions arise in the future we can address them later.

So I'm in favor of moving the restrictions out of the profiles to the spec.

spec.md Outdated

| Response field | Type | Description |
| --- | --- | --- |
| parameters | JSON schema object | The schema definition for the input parameters. Platforms MUST support at least [JSON Schema draft v4](http://json-schema.org/), but SHOULD be prepared to support multiple versions. Broker auhtors MUST provide the version of JSON Schema they are using. Schemas MUST NOT contain any external references and MUST be no larger than 64kB. Each input parameter is expressed as a property within a JSON object. |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo -- auhtors instead of authors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - fixed!

@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented Sep 8, 2017

Looking for some LGTMs today! (Thanks for the feedback so far)
@angarg12 @duglin @pmorie @skaegi @shalako @vaikas-google @fmui @avade

Approved with PullApprove

@angarg12
Copy link
Contributor

angarg12 commented Sep 8, 2017

LGTM

Approved with PullApprove

@@ -228,10 +228,40 @@ how platforms might expose these values to their users.
| metadata | JSON object | An opaque object of metadata for a service plan. Controller treats this as a blob. Note that there are [conventions](profile.md#service-metadata) in existing brokers and controllers for fields that aid in the display of catalog data. |
| free | boolean | When false, service instances of this plan have a cost. The default is true. |
| bindable | boolean | Specifies whether service instances of the service plan can be bound to applications. This field is OPTIONAL. If specified, this takes precedence over the `bindable` attribute of the service. If not specified, the default is derived from the service. |
| [schemas](#schema-object) | object | Schema definitions for service instances and bindings for the plan. |
Copy link
Contributor

@duglin duglin Sep 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of these objects, should they be JSON object to be consistent with what I above? e.g. metadata

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it should be - thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait - I'm not actually sure it should be. Elsewhere in the spec when the object definition is defined in the spec, we just use object, and use JSON object where the structure is not defined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm I think you're right. That distinction was totally lost on me and I wonder whether its one worth keeping - after all, object is always a JSON object, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that can be a different PR, if we want to change it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't think there are any cases now in the spec where object != JSON object, but we can handle that elsewhere :)

spec.md Outdated

| Response field | Type | Description |
| --- | --- | --- |
| parameters | JSON schema object | The schema definition for the input parameters. Platforms MUST support at least [JSON Schema draft v4](http://json-schema.org/), but SHOULD be prepared to support multiple versions. Broker authors MUST provide the version of JSON Schema they are using. Schemas MUST NOT contain any external references and MUST be no larger than 64kB. Each input parameter is expressed as a property within a JSON object. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with the requirements of this field. I'm just wondering whether it would read better if everything after the first sentence were moved to a non-tabular format. Meaning, put it in a paragraph/section right after this table so that we have more space to be more explicit. For example:

+++
While the presence of schema in the Plan is OPTIONAL, if it is present then the following applies:

  • Platforms MUST support at least JSON Schema draft v4.
  • Platforms SHOULD be prepared to support other versions of JSON Schema well.
  • The $schema keyword MUST be present in the schema with the version of the JSON Schema being used.
  • Schemas MUST NOT contain any external references (e.g. use the $ref keyword).
  • Schemas MUST NOT be larger than 64kB.
  • Each input parameter MUST be expressed as a child of the properties field in the schema.

+++

Just allows us to be a little more verbose and show the exact json entities we're talking about.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you want to mention that type is required? Or is that obvious?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're leaving that up to the spec, since this changes in v6 schema.
Moving the text out of the table - check back in 2 mins!

"type": "object",
"properties": {
"billing-account": {
"description": "Billing account number used to charge use of shared fake server.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want (or need) to place any requirements on the set of fields used to describe the properties? I'm guessing no but wanted to ask to be sure.

@duglin
Copy link
Contributor

duglin commented Sep 8, 2017

LGTM

Approved with PullApprove

1 similar comment
@angarg12
Copy link
Contributor

angarg12 commented Sep 8, 2017

LGTM

Approved with PullApprove

@skaegi
Copy link

skaegi commented Sep 8, 2017

LGTM

@avade
Copy link
Contributor

avade commented Sep 8, 2017

LGTM

Approved with PullApprove

1 similar comment
@vaikas
Copy link
Contributor

vaikas commented Sep 8, 2017

LGTM

Approved with PullApprove

@vaikas vaikas merged commit 49ec60c into openservicebrokerapi:master Sep 8, 2017
@mattmcneeney
Copy link
Contributor Author

mattmcneeney commented Sep 8, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.