Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "Plugin marketplace" infrastructure to hermes docs and codebase #299

Open
wants to merge 36 commits into
base: develop
Choose a base branch
from

Conversation

zyzzyxdonta
Copy link
Contributor

@zyzzyxdonta zyzzyxdonta commented Dec 11, 2024

This pull requests adds a list of available plugins to the hermes docs page that can act as a first basic "marketplace".

TODO:

  • Add data extraction script
  • Add issue template for people who want to contribute their own plugin to the list
  • List of plugins complete?
  • How to signify "builtin" in schema.org markup? → schema:isPartOf
  • How to add the repo URL to the markup? It seems only SoftwareSourceCode targetProduct SoftwareApplication . is possible. So we would need a {"@type": "SoftwareSourceCode", "targetProduct": {"@type": "SoftwareApplication"}}. This would be valid but not as nice as having all the plugins as top-level elements. Alternative: schema:url
  • Add markup for <hermes> softwareAddOn <each plugin> . ? → Doesn't work well with "one script tag per plugin" paradigm.

The first simple design looks like this:

Screenshot_20241211_092557

In addition to the human-readable text, machine-readable markup is added. Schema.org markup was used. The markup on the page can be checked using the browser extension "OpenLink Structured Data Sniffer" or by pasting the source code of the compiled page into https://validator.schema.org/

Closes #292

@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch 2 times, most recently from 66bf27c to bf824fe Compare December 11, 2024 08:49
@sdruskat
Copy link
Contributor

@zyzzyxdonta Nice one! 🚀

I like this simple impl! A few comments:

  • One thing I'd like to see added to the metadata is the type of the plugin, i.e., the step that this plugin targets. This will help filtering and could be used to display plugins of the same type under dedicated subheadings.
  • In a next step, perhaps we could add another optional field that describes the filetypes (via, e.g., filename or schema) that a plugin works on? This makes sense mainly for harvest plugins, but I'd rather have an optional field that goes unused than introduce plugin metadata inheritance ;).
  • At some point, we should probably provide a JSON schema for the plugin metadata?
  • For now, having a centralized list of plugins makes sense. If at some point this becomes feasible, it could be better to let each plugin have its own metadata in its repo, where the marketplace retrieves it from. This way, plugin authors keep the metadata where the plugin is developed for easier updates. And we have less maintenance work to do, and a simple automated registration of a new plugin would suffice (register via issue, closing issue triggers CI to pull metadata and re-render marketplace).

Thoughts?

@zyzzyxdonta
Copy link
Contributor Author

  • One thing I'd like to see added to the metadata is the type of the plugin, i.e., the step that this plugin targets. This will help filtering and could be used to display plugins of the same type under dedicated subheadings.

Done.

Screenshot_20241211_120537

@led02
Copy link
Member

led02 commented Dec 11, 2024

@zyzzyxdonta Nice one! 🚀

I like this simple impl! A few comments:

  • One thing I'd like to see added to the metadata is the type of the plugin, i.e., the step that this plugin targets. This will help filtering and could be used to display plugins of the same type under dedicated subheadings.

...

Thoughts?

I think it's important to have the steps annotated but I think this is not a one-to-one relationship. I would expect a Python plugin to contain a harvester and maybe a PyPI deposit and even a post-processor. Our CFF post-process could also be coupled with the harvester if it was a plugin...

@zyzzyxdonta
Copy link
Contributor Author

I think it's important to have the steps annotated but I think this is not a one-to-one relationship.

I've taken this into account by allowing multiple steps to be listed. If a plugin targets multiple steps, it appears in all of the related lists for now.

@zyzzyxdonta
Copy link
Contributor Author

zyzzyxdonta commented Dec 13, 2024

I wrote a simple Sphinx extension to get rid of the horrible templating of the JSON-LD markup 🙈

Builtin plugins are now marked using schema:isPartOf <hermes>.

@zyzzyxdonta
Copy link
Contributor Author

  • One thing I'd like to see added to the metadata is the type of the plugin, i.e., the step that this plugin targets. This will help filtering and could be used to display plugins of the same type under dedicated subheadings.

  • In a next step, perhaps we could add another optional field that describes the filetypes (via, e.g., filename or schema) that a plugin works on? This makes sense mainly for harvest plugins, but I'd rather have an optional field that goes unused than introduce plugin metadata inheritance ;).

How about using schema:featureList to mark the targeted steps? And schema:keywords for handled file types?

@zyzzyxdonta
Copy link
Contributor Author

Added hermes-marketplace command which requests the list of known plugins from the website and prints it as follows:

See the detailed list of plugins here: https://hermes.software-metadata.pub#plugins

                  CITATION.cff  (builtin)
                 codemeta.json  (builtin)
             hermes-plugin-git  https://github.com/softwarepub/hermes-plugin-git
          hermes-plugin-python  https://github.com/softwarepub/hermes-plugin-python
                         Merge  (builtin)
                        Curate  (builtin)
               Invenio Deposit  (builtin)
           Invenio RDM Deposit  (builtin)
                Rodare Deposit  (builtin)
           Invenio Postprocess  (builtin)
       Invenio RDM Postprocess  (builtin)

Please note that in order to test this command, the MARKETPLACE_URL in src/hermes/commands/marketplace.py has to be adapted to wherever you deploy your docs pages during testing (likely "http://127.0.0.1:8000")

@zyzzyxdonta
Copy link
Contributor Author

zyzzyxdonta commented Dec 13, 2024

I feel like we could have a common data model (e.g. pydantic model) between the Sphinx extension and the hermes-marketplace command. 👀 Even third-party plugins could then use it to publish their plugin metadata "correctly". But this would require them using Sphinx while a simple JSON file in a repo or a JSON-LD script in the README is way simpler.

@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch 4 times, most recently from 80ab99f to 0b887b1 Compare December 16, 2024 12:43
@zyzzyxdonta
Copy link
Contributor Author

  • At some point, we should probably provide a JSON schema for the plugin metadata?

Done

@zyzzyxdonta
Copy link
Contributor Author

I'll mark this as ready for review now. These are open questions:

  • How to express in Schema.org the Hermes workflow steps targeted by the plugin? schema:keywords? schema:featureList?
  • How to express in Schema.org the file types understood by a harvesting plugin? schema:keywords? schema:featureList?

@zyzzyxdonta zyzzyxdonta marked this pull request as ready for review December 16, 2024 14:49
@zyzzyxdonta zyzzyxdonta changed the title Add plugin "marketplace" to hermes docs Add "Plugin marketplace" infrastructure to hermes docs and codebase Dec 16, 2024
@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch 2 times, most recently from 8b2b64d to 00dc52f Compare December 17, 2024 10:11
@zyzzyxdonta
Copy link
Contributor Author

zyzzyxdonta commented Dec 17, 2024

This is what the issue template looks like (though some fields will be shown as required which doesn't work in the private repo I used for testing, and the unnecessary line break before the marketplace link is fixed):

Screenshot_20241217_111342

@zyzzyxdonta
Copy link
Contributor Author

  • How to express in Schema.org the Hermes workflow steps targeted by the plugin? schema:keywords? schema:featureList?

  • How to express in Schema.org the file types understood by a harvesting plugin? schema:keywords? schema:featureList?

I opted for keywords for everything.

  • Plugin type: hermes-step-harvest, hermes-step-process, ...
  • Filetype: hermes-harvest-citation-cff, hermes-harvest-pyproject-toml, ...

@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch from d330196 to b0ecc1f Compare December 19, 2024 10:35
@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch from b0ecc1f to c4de01c Compare December 19, 2024 10:40
@zyzzyxdonta zyzzyxdonta force-pushed the feature/292-plugin-marketplace branch from dfbb4c2 to bd047f3 Compare December 19, 2024 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Plugin Marketplace
3 participants