Extension of API {id}/versions and {id}/versions/{versionId} with an optional excludeMetadataBlocks parameter

[johannes-darms]

What this PR does / why we need it:

Extension of API {id}/versions and {id}/versions/{versionId} with an optional excludeMetadataBlocks parameter,
that specifies whether the metadataBlocks should be listed in the output. It defaults to false, preserving backward
compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks
included can dramatically increase the volume of the output).

We have slow response from api/datasets/%s/versions due to the large response body. Most of the information included (metadatablocks) is not needed as we just want to display a dropdown list with all available versions.

Which issue(s) this PR closes:

• Closes Feature Request/Idea: List Versions of a Dataset API reduced response size #10171

Suggestions on how to test this: Call the API once with the flag and once without.

Does this PR introduce a user interface change?: No

Is there a release notes update needed for this change?: Maybe, it is a new optional property.

Preview docs at https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset

[coveralls]

coverage: 22.571% (+1.8%) from 20.791%
when pulling b8d0f3d on johannes-darms:feat/10171-versions-smaller-response
into cf174b2 on IQSS:develop.

[johannes-darms]

@GPortas we are experiencing some performance issues with our SPA when a user requests information about dataset versions, particularly those with many versions. Are you experiencing similar problems? We believe that reducing the payload by omitting the metadata would solve our problem. As we can load the metadata with another query if needed. What do you think?

[GPortas]

@GPortas we are experiencing some performance issues with our SPA when a user requests information about dataset versions, particularly those with many versions. Are you experiencing similar problems? We believe that reducing the payload by omitting the metadata would solve our problem. As we can load the metadata with another query if needed. What do you think?

I'm not sure if we've experienced issues with the metadata blocks, and if we have, they may have been minor, possibly because we don't tend to add complex metadata block configurations in our test datasets.

It's reasonable to think it will improve performance, as additional queries are omitted. This is somewhat similar to what we did with the files, where we added the optional query parameter called excludeFiles.

This makes me wonder if it might be interesting to create a 'reduced information' endpoint instead of continuing to include parameters for excluding properties in the general endpoint.

[pdurbin]

Here are the docs for excludeFiles: https://guides.dataverse.org/en/6.4/api/native-api.html#get-version-of-a-dataset

[johannes-darms]

@GPortas we are experiencing some performance issues with our SPA when a user requests information about dataset versions, particularly those with many versions. Are you experiencing similar problems? We believe that reducing the payload by omitting the metadata would solve our problem. As we can load the metadata with another query if needed. What do you think?

I'm not sure if we've experienced issues with the metadata blocks, and if we have, they may have been minor, possibly because we don't tend to add complex metadata block configurations in our test datasets.

It's reasonable to think it will improve performance, as additional queries are omitted. This is somewhat similar to what we did with the files, where we added the optional query parameter called excludeFiles.

This makes me wonder if it might be interesting to create a 'reduced information' endpoint instead of continuing to include parameters for excluding properties in the general endpoint.

Our problem is only partly caused by the large complex metadata block, the other cause is the amount of versions (we have a dataset where an update is published every day, the file changes but the metadata is the same). So the payload of this API becomes huge and by omitting the metadata blocks we can mitigate the problem while still getting the necessary information about versions without introducing paging.

I'm not a fan of having different endpoints for more or less the same information. It is more code to maintain and more complicated for the user.

This PR is inspired by the excludeFiles feature and the code is quite similar.

[pdurbin]

@GPortas at some point we should probably test the SPA against a dataset with lots of versions. We should have datasets like this on the performance cluster.

[pdurbin]

@johannes-darms overall, looks good. I left some comments. Thanks.

[pdurbin]

@johannes-darms can you please merge the latest from develop? We need this anyway and it will trigger a Jenkins run, which is failing. Also, please consider the suggestions I made in my review. Thanks.

[johannes-darms]

@johannes-darms can you please merge the latest from develop? We need this anyway and it will trigger a Jenkins run, which is failing. Also, please consider the suggestions I made in my review. Thanks.

Sorry for the delay. I've merged, adapted the documentation and wrote a simple test. If you need or want more tests I'm happy to write them.

[pdurbin]

@johannes-darms thanks for adding tests and addressing all my questions! I didn't run the code myself but tests are passing and code and docs look good. Approved!