Move non-plugin definition keys under a docs or meltanohub key

[pnadolny13]

As part of the 2.0 release upgrades to the hub which included a full refactor of the definition file format we added some new keys that are for rendering documentation. I'm proposing that we move those under something like a docs or meltanohub key so that its clean and clear what parts are plugin definition attributes and should be returned via the API to be stored in lock files vs what should simply be extra metadata for the hub.

For example tap-adwords usage metadata key for the hub docs content is in the definition file.

Then in the API payload generator we exclude them.

The keys are:

• keywords
• maintenance_status
• domain_url
• definition
• next_steps
• settings_preamble
• usage
• prereq

The new schema would be something like:

name: tap-adwords
label: Google Ads
description: Advertising Platform
...
docs:
    domain_url: x
    usage: y
    prereq: z
...

cc @tayloramurphy @aaronsteers @DouweM @alexmarple @z3z1ma

[DouweM]

@pnadolny13 The docs key already exists and is respected by Meltano, but I like the idea and suggest hub.

[tayloramurphy]

@pnadolny13 I quite like that idea. It'd also make it cleaner for folks who may generate the config from the SDK. I'm going to create an issue from this discussion and recommend using the hub key.

[aaronsteers]

I'd like to keep as much of this data in as we can. #609 (comment)

Also, we need to be careful how many times we change the lockfile format - this is going to be another cause for breaking changes, since we already have users who have the old format installed. (In this case, I think we are safe because they likely don't have any other dependencies on these fields, but I want to call this out as an area we'll need to be careful going forward.)

I want to avoid hub as the name, because many of these fields have very valid use cases outside the hub. I'm all for removing elements that become redundant after we render them into the docs page, but otherwise I'd like to keep all the metadata in tact and not overly biased for the Hub use cases.

[tayloramurphy]

we need to be careful how many times we change the lockfile format

@aaronsteers how do we get to a place where we feel comfortable with changing the structure of the YAML when we need to as requirements change? The whole point of the lock file is to give stability to the project so that we can iterate more freely on the Hub. Do we need some API versioning in place on the Hub side for the definitions?

[pnadolny13]

@aaronsteers did you see my link to the to_delete_list in the API generator? We dont send any of this data via the API so its not affecting the schema of the lock file plugin definitions.

As a side note: I mentioned in #597 that I originally wrote the json schemas to validate the schema of the contents of the API, as a sort of unit test for that generator code. I care most about making sure we're not adding or removing data from our SSOT plugin definition API. During the 2.0 push/GitHub migration these schemas started to be used as hub definition schemas AND the API schemas which are different so we'll need to get that cleaned up.

[tayloramurphy]

Ping on a few of the questions in here @aaronsteers. Particularly around the lock file format.

[aaronsteers]

Discussed on other threads, and I think this is now resolved.

Lmk if anything is still outstanding here.

[aaronsteers]

@pnadolny13, @tayloramurphy

I think there are two categories here and I would prefer to keep them separate. First category is those things that render into the generated docs page, and I think we are safe to remove those. Second category is those that are very useful data and should live even in the lock file. We could consider moving them under a heading, but I don't agree we should remove them lock files.

Regarding the second category, I don't see these as Hub-specific, nor do I really think we should have anything that is hub specific in those files - unless it's under a meta or vendor heading, which wouldn't fit for what I would consider critical references.

I think we're agreed on this point: any markdown fields that get rendered into the docs doesn't need to ship with the definition. (Although we probably do then want to inject a link to our generated docs.) Since those elements like are just a workaround for us to automate docs generation without injecting one or more separate markdown files. Those are the only elements I'd feel strongly about removing.

For usage, and prereqs, I think these should just live under docs header and when we run the API generator, we pop them from the docs section and add back the link to where the content was rendered.

For domain_url, keywords and maintenance_status, this seems like it could belong under a meta header, or under the docs header. I don't know if we need to drop those from the lock file though and I think they are also fine where they are. They seem relevant and my preference would be to keep things simple by not introducing unnecessary changes between what's in the source file, what's in the API, and what's in the lock files.

Regarding definition, next_steps, settings_preamble, I don't actually know what these do but if they fit any of the above patterns, I'd suggest same treatment.

Note:

It's important to consider that Meltano UI and also other end-user applications could also have very valid use cases for all of this data (excepting those that become redundant once we render them into the plugin's readme). Meltano UI might have a 'search' capability that locates the correct plugin or related plugins using keywords, and similarly, the Meltano UI might need a docs URL and domain URL to render into links on the metadata page.

It seems like we're somewhat equating the Hub as the only place we'll need docs and metadata info, which I don't think is the long term path here.

[pnadolny13]

@aaronsteers I'm fine with naming the key whatever we want but I want to avoid getting to a place where our Meltano plugin definition data which gets stored in a lock file gets mixed in with extra metadata that we want to add to power the front end. Which is what we have right now. I'm proposing to put them under a single key that can easily be excluded before rendering the API contents or another options could be to remove those fields from the yaml file completely and store somewhere else.

It's important to consider that Meltano UI and also other end-user applications could also have very valid use cases for all of this data (excepting those that become redundant once we render them into the plugin's readme). Meltano UI might have a 'search' capability that locates the correct plugin or related plugins using keywords, and similarly, the Meltano UI might need a docs URL and domain URL to render into links on the metadata page.

I get this point and I was considering it a bit also but it seems better to simply include a link to the hub vs sending all the hub extra metadata into the user's project. Also the current keys that are in these definition files are very hub specific so if we decided we wanted to ship that metadata around (which I personally dont think we should) then we need to make it more generic.

[aaronsteers]

Discussed on other threads, and I think this is now resolved.

Lmk if anything is still outstanding here.