OpenAPI definition endpoint #10328

jp-tosca · 2024-02-16T19:22:33Z

What this PR does / why we need it:
On Dataverse 6.0 Payara was updated and this caused the url /openapi to stop working and caused an exception on the server.

This PR restores the functionality by adding a Servlet on /openapi.

We incorporated the SmallRye OpenAPI plugin on our POM (https://github.com/smallrye/smallrye-open-api/tree/main/tools/maven-plugin) which will generate files for YAML and JSON formats and deposit them on edu/harvard/iq/dataverse/openapi/ these files will be provided by this endpoint depending on the format requested.

Which issue(s) this PR closes:

Suggestions on how to test this:

check on the develop branch the url http://localhost:8080/openapi
Update to this branch
Check the same URL

There are additional changes to improve the quality of the OpenAPI document like changing duplicated names from Java classes.

Is there a release notes update needed for this change?:
Yes

on Apr 17 2024: Payara has fixed the original endpoint but it is generating a spec definition that still has errors due duplicated names. We did some testing and regardless of what is on the documentation, the server is not returning the provided files on the META-INF directory, we tested this on the application root META-INF and the currently used META-INF. At this point, we decided to use this solution that will override the original endpoint and server the generated files from the smallrye plugin.

Preview of docs: https://dataverse-guide--10328.org.readthedocs.build/en/10328/api/native-api.html#get-openapi-specification

coveralls · 2024-02-16T19:28:33Z

coverage: 20.569% (-0.01%) from 20.58%
when pulling ef267d0 on openapi-definition-endpoint
into 26be8e1 on develop.

pdurbin

I assume docs are coming. Just a quick initial review. I'm excited that SmallRye is working to create the OpenAPI output!

src/main/resources/edu/harvard/iq/dataverse/openapi/dataverse_openapi.json

src/main/resources/edu/harvard/iq/dataverse/openapi/dataverse_openapi.yaml

src/main/java/propertyFiles/Bundle.properties

src/main/java/edu/harvard/iq/dataverse/api/Info.java

doc/release-notes/10236-openapi-definition-endpoint.md

src/test/java/edu/harvard/iq/dataverse/api/InfoIT.java

Co-authored-by: Philip Durbin <[email protected]>

pdurbin

A little more doc review.

doc/sphinx-guides/source/api/changelog.rst

doc/sphinx-guides/source/api/native-api.rst

pdurbin

I pushed a few commits and have a couple more comments.

pdurbin · 2024-05-29T19:25:36Z

src/main/java/edu/harvard/iq/dataverse/api/Access.java

 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataParam;

+import com.mashape.unirest.request.body.MultipartBody;


I'm the one who added Unirest but it's one of the libraries I'm hoping we can remove some day in favor of using the Jakarta EE standard. Do we need it here? I see a possible solution at https://stackoverflow.com/questions/64079847/how-to-provide-swagger-annotation-for-multipartformdatainput-in-resteasy-with-qu/65339649#65339649 . Is it worth trying it here and other places in this PR where Unirest is being used?

src/main/java/edu/harvard/iq/dataverse/openapi/OpenAPI.java

…-endpoint

github-actions · 2024-06-03T15:32:09Z

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:openapi-definition-endpoint

ghcr.io/gdcc/configbaker:openapi-definition-endpoint

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

pdurbin

Looks good now. API tests are passing. @JR-1991 did note the following...

I just went through the doc and ran a validation. These are the open points:

multipart schema must be an object— All multipart bodies are currently typed as string. This can also be addressed in preprocessing prior to code generation.
items attribute required for arrays in OpenAPI 3.0.X documents- Two array properties do not contain the items key. Also, easy to fix in pre-processing.
duplicate-properties - hTTPUpload (HTTPUpload) will collide with httpUpload (HTTPUpload) [line 11325] when converted to field name- Also addressable in pre-processing.

... in Zulip at https://dataverse.zulipchat.com/#narrow/stream/379673-dev/topic/.2Fopenapi.20broken.20in.206.2E0/near/441630800 BUT my take is that he can live with these problems.

Also, I still think we might want to stop gitignoring the YAML so we always have a version in the guides but we can talk about this at tech hours some day. No need to hold this up.

Approved.

JR-1991 · 2024-06-04T06:07:53Z

@pdurbin - I am totally fine with the current version. The problems are minor and easily addressable in a pre-processing step 🙌

* Plugin initial config * Initial changes to provide OpenAPI definition * Added integration test * Imports fix * Add patchnotes * Update the changelog * Update src/main/java/edu/harvard/iq/dataverse/api/Info.java Co-authored-by: Philip Durbin <[email protected]> * Update doc/release-notes/10236-openapi-definition-endpoint.md Co-authored-by: Philip Durbin <[email protected]> * Update doc/release-notes/10236-openapi-definition-endpoint.md Co-authored-by: Philip Durbin <[email protected]> * Add native API docs * Remove generated definitions * Add to gitignore generated openapi files * Updates to docs * Ignore files correction * Remove files created by the plugin * Changes to move the definition files to META-INF * Changes to move the definitions to WEB-INF * Changes to get the files from META-INF * Changed the phase of execution of the smallrye plugin * Changes of names to improve the generation of the spec * Add support for OpenAPI annotations and documents the version endpoint * Multipart Annotations * Typos correction * Changes for tags * Renaming of methods * Changes to the endpoint * Added test * Add test * Deleted extra import * Docs updated * openapi doc tweaks IQSS#9981 IQSS#10236 * improve release note IQSS#9981 IQSS#10236 * Remove old test and changes response to JSON * stub out guidance on openapi validation IQSS#9981 IQSS#10236 * add InfoIT to list of tests * use description of Dataverse from website * mention status codes in openapi doc * update api faq about changelog, link to breaking changes doc * typo * Change to OpenApi * Changes to docs * Name fix * Removing the multipart from unirest --------- Co-authored-by: Philip Durbin <[email protected]>

DS-INRAE · 2024-06-14T13:36:59Z

Congrats on merging this PR 👍
Does it also close these OpenAPI related issues ?

Or if the issues remaining are minor should the issues be closed as "won't resolve" ?

pdurbin · 2024-06-14T16:10:15Z

@DS-INRA thanks, we've been talking about this in Zulip and we closed both issues as out of date now that the new SmallRye-based /openapi endpoint will be available in the next release.

By the way, I love the issue triage you do for us. To encourage even more, I listed it as a way to contribute: https://dataverse-guide--10532.org.readthedocs.build/en/10532/contributor/index.html#issue-triage

jp-tosca added 4 commits February 15, 2024 14:51

Plugin initial config

e64bc66

Initial changes to provide OpenAPI definition

d717ad1

Added integration test

63df178

Imports fix

77f9c61

jp-tosca self-assigned this Feb 16, 2024

jp-tosca added the Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) label Feb 16, 2024

jp-tosca changed the title ~~Openapi definition endpoint~~ 10236 Openapi definition endpoint Feb 16, 2024

jp-tosca added 2 commits February 16, 2024 14:33

Add patchnotes

2da5b05

Update the changelog

9f528d8

pdurbin reviewed Feb 16, 2024

View reviewed changes

jp-tosca and others added 3 commits February 16, 2024 14:50

Update src/main/java/edu/harvard/iq/dataverse/api/Info.java

a8c4c06

Co-authored-by: Philip Durbin <[email protected]>

Update doc/release-notes/10236-openapi-definition-endpoint.md

a859a50

Co-authored-by: Philip Durbin <[email protected]>

Update doc/release-notes/10236-openapi-definition-endpoint.md

8d41917

Co-authored-by: Philip Durbin <[email protected]>

This comment has been minimized.

Sign in to view

jp-tosca added 2 commits February 16, 2024 15:24

Add native API docs

9405d54

Remove generated definitions

468672e

pdurbin reviewed Feb 16, 2024

View reviewed changes

Add to gitignore generated openapi files

3a0daa6

This comment has been minimized.

Sign in to view

Updates to docs

3a699a8

This comment has been minimized.

Sign in to view

jp-tosca removed their assignment Feb 16, 2024

Remove old test and changes response to JSON

f265744

This comment has been minimized.

Sign in to view

pdurbin added 6 commits May 29, 2024 15:12

stub out guidance on openapi validation #9981 #10236

4ea444a

add InfoIT to list of tests

6fe471f

use description of Dataverse from website

31a65a7

mention status codes in openapi doc

5cbad6d

update api faq about changelog, link to breaking changes doc

0d656a1

typo

eb9cf7c

pdurbin reviewed May 29, 2024

View reviewed changes

jp-tosca added 3 commits May 29, 2024 16:22

Change to OpenApi

678865d

Changes to docs

d23d78c

Name fix

942d473

This comment has been minimized.

Sign in to view

Removing the multipart from unirest

683ea85

This comment has been minimized.

Sign in to view

jp-tosca mentioned this pull request Jun 3, 2024

Feature Request/Idea: Review of used regex #10607

Open

Merge remote-tracking branch 'origin/develop' into openapi-definition…

ef267d0

…-endpoint

pdurbin approved these changes Jun 3, 2024

View reviewed changes

pdurbin removed their assignment Jun 3, 2024

stevenwinship self-assigned this Jun 6, 2024

stevenwinship merged commit c052773 into develop Jun 6, 2024
20 checks passed

stevenwinship removed their assignment Jun 6, 2024

pdurbin added this to the 6.3 milestone Jun 6, 2024

pdurbin mentioned this pull request Jun 14, 2024

OpenAPI specification JSON-LD request bodies are invalid #9982

Closed

pdurbin deleted the openapi-definition-endpoint branch June 14, 2024 16:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenAPI definition endpoint #10328

OpenAPI definition endpoint #10328

jp-tosca commented Feb 16, 2024 •

edited

Loading

coveralls commented Feb 16, 2024 •

edited

Loading

pdurbin left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pdurbin left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pdurbin left a comment

pdurbin May 29, 2024

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 3, 2024

pdurbin left a comment

JR-1991 commented Jun 4, 2024

DS-INRAE commented Jun 14, 2024

pdurbin commented Jun 14, 2024 •

edited

Loading

OpenAPI definition endpoint #10328

OpenAPI definition endpoint #10328

Conversation

jp-tosca commented Feb 16, 2024 • edited Loading

coveralls commented Feb 16, 2024 • edited Loading

pdurbin left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pdurbin left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin May 29, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 3, 2024

pdurbin left a comment

Choose a reason for hiding this comment

JR-1991 commented Jun 4, 2024

DS-INRAE commented Jun 14, 2024

pdurbin commented Jun 14, 2024 • edited Loading

jp-tosca commented Feb 16, 2024 •

edited

Loading

coveralls commented Feb 16, 2024 •

edited

Loading

pdurbin commented Jun 14, 2024 •

edited

Loading