Merge remote-tracking branch 'IQSS/develop' into DANS-performance

.github/workflows/guides_build_sphinx.yml

@@ -11,6 +11,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - uses: OdumInstitute/sphinx-action@master
+    - uses: uncch-rdmc/sphinx-action@master
       with:
         docs-folder: "doc/sphinx-guides/"

doc/release-notes/10379-MetricsBugsFixes.md

@@ -0,0 +1,10 @@
+
+### Metrics API Bug fixes
+
+Two bugs in the Metrics API have been fixed:
+
+- The /datasets and /datasets/byMonth endpoints could report incorrect values if/when they have been called using the dataLocation parameter (which allows getting metrics for local, remote (harvested), or all datasets) as the metrics cache was not storing different values for these cases. 
+
+- Metrics endpoints who's calculation relied on finding the latest published datasetversion were incorrect if/when the minor version number was > 9. 
+
+When deploying the new release, the [/api/admin/clearMetricsCache](https://guides.dataverse.org/en/latest/api/native-api.html#metrics) API should be called to remove old cached values that may be incorrect.

doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md

@@ -0,0 +1,3 @@
+MDC Citation retrieval with the PID settings has been fixed.
+DOI parsing in Dataverse is case insensitive, improving interaction with services that may change the case.
+Warnings related to managed/excluded PID lists for PID providers have been reduced

doc/release-notes/10886-update-to-conditions-to-display-image_url.md

@@ -0,0 +1,8 @@
+Search API (/api/search) responses for Datafiles include image_url for the thumbnail if each of the following are true:
+1. The DataFile is not Harvested
+2. A Thumbnail is available for the Datafile
+3. If the Datafile is Restricted then the caller must have Download File Permission for the Datafile
+4. The Datafile is NOT actively embargoed
+5. The Datafile's retention period has NOT expired
+
+See also #10875 and #10886.

doc/release-notes/10904-edit-dataverse-collection-endpoint.md

@@ -0,0 +1 @@
+Adds a new endpoint (`PUT /api/dataverses/<identifier>`) for updating an existing Dataverse collection using a JSON file following the same structure as the one used in the API for the creation.

doc/release-notes/10914-users-token-api-credentials.md

@@ -0,0 +1,3 @@
+Extended the users/token GET endpoint to support any auth mechanism for retrieving the token information.
+
+Previously, this endpoint only accepted an API token to retrieve its information. Now, it accepts any authentication mechanism and returns the associated API token information.

doc/release-notes/10919-minor-DataCiteXML-bugfix.md

@@ -0,0 +1 @@
+A minor bug fix was made to avoid sending a useless ", null" in the DataCiteXML sent to DataCite and in the DataCite export when a dataset has a metadata entry for "Software Name" and no entry for "Software Version". The bug fix will update datasets upon publication. Anyone with existing published datasets with this problem can be fixed by [pushing updated metadata to DataCite for affected datasets](https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider) and [re-exporting the dataset metadata](https://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api) or by following steps 9 and 10 in the v6.4 release notes to update and re-export all datasets.

doc/release-notes/10947-unpublished-files-appearing-in-search-results-for-anon-user.md

@@ -0,0 +1,11 @@
+## Unpublished file bug fix
+
+A bug fix was made that gets the major version of a Dataset when all major versions were deaccessioned. This fixes the incorrect showing of the files as "Unpublished" in the search list even when they are published.
+This fix affects the indexing, meaning these datasets must be re-indexed once Dataverse is updated. This can be manually done by calling the index API for each affected Dataset.
+
+Example:
+```shell
+curl http://localhost:8080/api/admin/index/dataset?persistentId=doi:10.7910/DVN/6X4ZZL
+```
+
+See also #10947 and #10974.

doc/sphinx-guides/source/api/native-api.rst

@@ -74,6 +74,58 @@ The request JSON supports an optional ``metadataBlocks`` object, with the follow
 
 To obtain an example of how these objects are included in the JSON file, download :download:`dataverse-complete-optional-params.json <../_static/api/dataverse-complete-optional-params.json>` file and modify it to suit your needs.
 
+.. _update-dataverse-api:
+
+Update a Dataverse Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Updates an existing Dataverse collection using a JSON file following the same structure as the one used in the API for the creation. (see :ref:`create-dataverse-api`).
+
+The steps for updating a Dataverse collection are:
+
+- Prepare a JSON file containing the fields for the properties you want to update. You do not need to include all the properties, only the ones you want to update.
+- Execute a curl command or equivalent.
+
+As an example, you can download :download:`dataverse-complete.json <../_static/api/dataverse-complete.json>` file and modify it to suit your needs. The controlled vocabulary for ``dataverseType`` is the following:
+
+- ``DEPARTMENT``
+- ``JOURNALS``
+- ``LABORATORY``
+- ``ORGANIZATIONS_INSTITUTIONS``
+- ``RESEARCHERS``
+- ``RESEARCH_GROUP``
+- ``RESEARCH_PROJECTS``
+- ``TEACHING_COURSES``
+- ``UNCATEGORIZED``
+
+The curl command below assumes you are using the name "dataverse-complete.json" and that this file is in your current working directory.
+
+Next you need to figure out the alias or database id of the Dataverse collection you want to update.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export DV_ALIAS=dvAlias
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/dataverses/$DV_ALIAS" --upload-file dataverse-complete.json
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/dataverses/dvAlias" --upload-file dataverse-complete.json
+
+You should expect an HTTP 200 response and JSON beginning with "status":"OK" followed by a representation of the updated Dataverse collection.
+
+Same as in :ref:`create-dataverse-api`, the request JSON supports an optional ``metadataBlocks`` object, with the following supported sub-objects:
+
+- ``metadataBlockNames``: The names of the metadata blocks you want to add to the Dataverse collection.
+- ``inputLevels``: The names of the fields in each metadata block for which you want to add a custom configuration regarding their inclusion or requirement when creating and editing datasets in the new Dataverse collection. Note that if the corresponding metadata blocks names are not specified in the ``metadataBlockNames``` field, they will be added automatically to the Dataverse collection.
+- ``facetIds``: The names of the fields to use as facets for browsing datasets and collections in the new Dataverse collection. Note that the order of the facets is defined by their order in the provided JSON array.
+
+To obtain an example of how these objects are included in the JSON file, download :download:`dataverse-complete-optional-params.json <../_static/api/dataverse-complete-optional-params.json>` file and modify it to suit your needs.
+
 .. _view-dataverse:
 
 View a Dataverse Collection

doc/sphinx-guides/source/container/dev-usage.rst

@@ -140,6 +140,56 @@ Alternatives:
 - If you used Docker Compose for running, you may use ``docker compose -f docker-compose-dev.yml logs <service name>``.
   Options are the same.
 
+Accessing Harvesting Log Files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+\1. Open a terminal and access the Dataverse container.
+
+Run the following command to access the Dataverse container (assuming your container is named dataverse-1):
+
+.. code-block::
+
+  docker exec -it dataverse-1 bash
+
+This command opens an interactive shell within the dataverse-1 container.
+
+\2. Navigate to the log files directory.
+
+Once inside the container, navigate to the directory where Dataverse logs are stored:
+
+.. code-block::
+
+  cd /opt/payara/appserver/glassfish/domains/domain1/logs
+
+This directory contains various log files, including those relevant to harvesting.
+
+\3. Create a directory for copying files.
+
+Create a directory where you'll copy the files you want to access on your local machine:
+
+.. code-block::
+
+  mkdir /dv/filesToCopy
+
+This will create a new folder named filesToCopy inside /dv.
+
+\4. Copy the files to the new directory.
+
+Copy all files from the current directory to the newly created filesToCopy directory:
+
+.. code-block::
+
+  cp * /dv/filesToCopy
+
+This command copies all files in the logs directory to /dv/filesToCopy.
+
+\5. Access the files on your local machine.
+
+On your local machine, the copied files should appear in the following directory:
+
+.. code-block::
+
+  docker-dev-volumes/app/data/filesToCopy
 
 Redeploying
 -----------

doc/sphinx-guides/source/developers/version-control.rst

@@ -291,16 +291,16 @@ By default, when a pull request is made from a fork, "Allow edits from maintaine
 
 This is a nice feature of GitHub because it means that the core dev team for the Dataverse Project can make small (or even large) changes to a pull request from a contributor to help the pull request along on its way to QA and being merged.
 
-GitHub documents how to make changes to a fork at https://help.github.com/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork/ but as of this writing the steps involve making a new clone of the repo. This works but you might find it more convenient to add a "remote" to your existing clone. The example below uses the fork at https://github.com/OdumInstitute/dataverse and the branch ``4709-postgresql_96`` but the technique can be applied to any fork and branch:
+GitHub documents how to make changes to a fork at https://help.github.com/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork/ but as of this writing the steps involve making a new clone of the repo. This works but you might find it more convenient to add a "remote" to your existing clone. The example below uses the fork at https://github.com/uncch-rdmc/dataverse and the branch ``4709-postgresql_96`` but the technique can be applied to any fork and branch:
 
 .. code-block:: bash
 
-        git remote add OdumInstitute [email protected]:OdumInstitute/dataverse.git
-        git fetch OdumInstitute
+        git remote add uncch-rdmc [email protected]:uncch-rdmc/dataverse.git
+        git fetch uncch-rdmc
         git checkout 4709-postgresql_96
         vim path/to/file.txt
         git commit
-        git push OdumInstitute 4709-postgresql_96
+        git push uncch-rdmc 4709-postgresql_96
 
 .. _develop-into-develop:
 

doc/sphinx-guides/source/qa/testing-infrastructure.md

@@ -31,6 +31,8 @@ To build and test a PR, we use a job called `IQSS_Dataverse_Internal` on <https:
 
 1. If for some reason it didn't deploy, check the server.log file. It may just be a caching issue so try un-deploying, deleting cache, restarting, and re-deploying on the server (`su - dataverse` then `/usr/local/payara6/bin/asadmin list-applications; /usr/local/payara6/bin/asadmin undeploy dataverse-6.1; /usr/local/payara6/bin/asadmin deploy /tmp/dataverse-6.1.war`)
 
+1. When a Jenkins job fails after a release, it might be due to the version number in the `pom.xml` file not being updated in the pull request (PR). To verify this, open the relevant GitHub issue, navigate to the PR branch, and go to `dataverse > modules > dataverse-parent > pom.xml`. Look for the version number, typically shown as `<revision>6.3</revision>`, and ensure it matches the current Dataverse build version. If it doesn't match, ask the developer to update the branch with the latest from the "develop" branch.
+
 1. If that didn't work, you may have run into a Flyway DB script collision error but that should be indicated by the server.log. See {doc}`/developers/sql-upgrade-scripts` in the Developer Guide. In the case of a collision, ask the developer to rename the script.
 
 1.	Assuming the above steps worked, and they should 99% of the time, test away! Note: be sure to `tail -F server.log` in a terminal window while you are doing any testing. This way you can spot problems that may not appear in the UI and have easier access to any stack traces for easier reporting.

doc/sphinx-guides/source/style/text.rst

@@ -9,4 +9,4 @@ Here we describe the guidelines that help us provide helpful, clear and consiste
 Metadata Text Guidelines
 ========================
 
-These guidelines are maintained in `a Google Doc <https://docs.google.com/document/d/1uRk_dAZlaCS91YFbqE6L9Jwhwum7mOadkJ59XUx40Sg>`__ as we expect to make frequent changes to them. We welcome comments in the Google Doc.
+These guidelines are maintained in `a Google Doc <https://docs.google.com/document/d/1tY5t3gjrIgAGoRxVMWQSCh46fnbSmnFDLQ7aLkNLhJ8/>`__ as we expect to make frequent changes to them. We welcome comments in the Google Doc.

pom.xml

@@ -30,7 +30,7 @@
         <flyway.version>9.22.1</flyway.version>
         <jhove.version>1.20.1</jhove.version>
         <poi.version>5.2.1</poi.version>
-        <tika.version>2.4.1</tika.version>
+        <tika.version>2.9.1</tika.version>
         <netcdf.version>5.5.3</netcdf.version>
 
         <openapi.infoTitle>Dataverse API</openapi.infoTitle>
@@ -68,6 +68,17 @@
                     </exclusion>
                 </exclusions>
             </dependency>
+            <!-- Dependency for Apache Abdera and Apache Tika. Tika needs newer version. -->
+            <dependency>
+                <groupId>org.apache.james</groupId>
+                <artifactId>apache-mime4j-core</artifactId>
+                <version>0.8.10</version>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.james</groupId>
+                <artifactId>apache-mime4j-dom</artifactId>
+                <version>0.8.7</version>
+            </dependency>
         </dependencies>
     </dependencyManagement>
     <!-- Declare any DIRECT dependencies here.
@@ -553,7 +564,7 @@
         <dependency>
             <groupId>org.xmlunit</groupId>
             <artifactId>xmlunit-core</artifactId>
-            <version>2.9.1</version>
+            <version>2.10.0</version>
         </dependency>
         <dependency>
           <groupId>com.google.cloud</groupId>
@@ -615,7 +626,7 @@
         <dependency>
             <groupId>org.xmlunit</groupId>
             <artifactId>xmlunit-assertj3</artifactId>
-            <version>2.8.2</version>
+            <version>2.10.0</version>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -1127,4 +1138,4 @@
             </build>
         </profile>
     </profiles>
-</project>
+</project>

src/main/java/edu/harvard/iq/dataverse/Dataset.java

@@ -485,8 +485,17 @@ public Date getMostRecentMajorVersionReleaseDate() {
         if (this.isHarvested()) {
             return getVersions().get(0).getReleaseTime();
         } else {
+            Long majorVersion = null;
             for (DatasetVersion version : this.getVersions()) {
-                if (version.isReleased() && version.getMinorVersionNumber().equals((long) 0)) {
+                if (version.isReleased()) {
+                    if (version.getMinorVersionNumber().equals((long) 0)) {
+                        return version.getReleaseTime();
+                    } else if (majorVersion == null) {
+                        majorVersion = version.getVersionNumber();
+                    }
+                } else if (version.isDeaccessioned() && majorVersion != null
+                        && majorVersion.longValue() == version.getVersionNumber().longValue()
+                        && version.getMinorVersionNumber().equals((long) 0)) {
                     return version.getReleaseTime();
                 }
             }

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

@@ -891,6 +891,10 @@ public List<DatasetFieldType> findAllDisplayedOnCreateInMetadataBlock(MetadataBl
     }
 
     public List<DatasetFieldType> findAllInMetadataBlockAndDataverse(MetadataBlock metadataBlock, Dataverse dataverse, boolean onlyDisplayedOnCreate) {
+        if (!dataverse.isMetadataBlockRoot() && dataverse.getOwner() != null) {
+            return findAllInMetadataBlockAndDataverse(metadataBlock, dataverse.getOwner(), onlyDisplayedOnCreate);
+        }
+
         CriteriaBuilder criteriaBuilder = em.getCriteriaBuilder();
         CriteriaQuery<DatasetFieldType> criteriaQuery = criteriaBuilder.createQuery(DatasetFieldType.class);
 

src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java

@@ -5,6 +5,7 @@
  */
 package edu.harvard.iq.dataverse;
 
+import edu.harvard.iq.dataverse.authorization.Permission;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
 import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter;
 import edu.harvard.iq.dataverse.dataaccess.StorageIO;
@@ -20,7 +21,6 @@
 
 import jakarta.ejb.EJB;
 import jakarta.enterprise.context.RequestScoped;
-import jakarta.inject.Inject;
 import jakarta.inject.Named;
 
 /**
@@ -33,9 +33,8 @@
 public class ThumbnailServiceWrapper implements java.io.Serializable  {
 
     private static final Logger logger = Logger.getLogger(ThumbnailServiceWrapper.class.getCanonicalName());
-
-    @Inject
-    PermissionsWrapper permissionsWrapper;
+    @EJB
+    PermissionServiceBean permissionService;
     @EJB
     DataverseServiceBean dataverseService;
     @EJB
@@ -49,12 +48,15 @@ public class ThumbnailServiceWrapper implements java.io.Serializable  {
     private Map<Long, DvObject> dvobjectViewMap = new HashMap<>();
     private Map<Long, Boolean> hasThumbMap = new HashMap<>();
 
+    private boolean hasDownloadFilePermission(DvObject dvo) {
+        return permissionService.on(dvo).has(Permission.DownloadFile) ;
+    }
     public String getFileCardImageAsUrl(SolrSearchResult result) {
         DataFile dataFile = result != null && result.getEntity() != null ? ((DataFile) result.getEntity()) : null;
-        if (dataFile == null || result.isHarvested()
+        if (dataFile == null
+                || result.isHarvested()
                 || !isThumbnailAvailable(dataFile)
-                || dataFile.isRestricted()
-                || !dataFile.isReleased()
+                || (dataFile.isRestricted() && !hasDownloadFilePermission(dataFile))
                 || FileUtil.isActivelyEmbargoed(dataFile)
                 || FileUtil.isRetentionExpired(dataFile)) {
             return null;
@@ -105,7 +107,7 @@ public String getFileCardImageAsBase64Url(SolrSearchResult result) {
             }
 
             if ((!((DataFile)result.getEntity()).isRestricted()
-                        || permissionsWrapper.hasDownloadFilePermission(result.getEntity()))
+                        || hasDownloadFilePermission(result.getEntity()))
                     && isThumbnailAvailable((DataFile) result.getEntity())) {
 
                 cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64(