Merge remote-tracking branch 'IQSS/develop' into DANS/bag-ore-updates

doc/release-notes/9880-info-api-zip-limit-embargo.md

@@ -0,0 +1,5 @@
+Implemented the following new endpoints:
+
+- getZipDownloadLimit (/api/info/zipDownloadLimit): Get the configured zip file download limit. The response contains the long value of the limit in bytes.
+
+- getMaxEmbargoDurationInMonths (/api/info/settings/:MaxEmbargoDurationInMonths): Get the maximum embargo duration in months, if available, configured through the database setting :MaxEmbargoDurationInMonths.

doc/sphinx-guides/source/api/native-api.rst

@@ -3320,6 +3320,8 @@ Show Support Of Incomplete Metadata Deposition
 Learn if an instance has been configured to allow deposition of incomplete datasets via the API.
 See also :ref:`create-dataset-command` and :ref:`dataverse.api.allow-incomplete-metadata`
 
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
 .. code-block:: bash
 
   export SERVER_URL=https://demo.dataverse.org
@@ -3332,6 +3334,45 @@ The fully expanded example above (without environment variables) looks like this
 
   curl "https://demo.dataverse.org/api/info/settings/incompleteMetadataViaApi"
 
+Get Zip File Download Limit
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Get the configured zip file download limit. The response contains the long value of the limit in bytes.
+
+This limit comes from the database setting :ref:`:ZipDownloadLimit` if set, or the default value if the database setting is not set, which is 104857600 (100MB).
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl "$SERVER_URL/api/info/zipDownloadLimit"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/api/info/zipDownloadLimit"
+
+Get Maximum Embargo Duration In Months
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Get the maximum embargo duration in months, if available, configured through the database setting :ref:`:MaxEmbargoDurationInMonths` from the Configuration section of the Installation Guide.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl "$SERVER_URL/api/info/settings/:MaxEmbargoDurationInMonths"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/api/info/settings/:MaxEmbargoDurationInMonths"
 
 .. _metadata-blocks-api:
 

doc/sphinx-guides/source/developers/api-design.rst

@@ -0,0 +1,63 @@
+==========
+API Design
+==========
+
+API design is a large topic. We expect this page to grow over time.
+
+.. contents:: |toctitle|
+	:local:
+
+Paths
+-----
+
+A reminder `from Wikipedia <https://en.wikipedia.org/wiki/Uniform_Resource_Identifier>`_ of what a path is:
+
+.. code-block:: bash
+
+          userinfo       host      port
+          ┌──┴───┐ ┌──────┴──────┐ ┌┴┐
+  https://[email protected]:123/forum/questions/?tag=networking&order=newest#top
+  └─┬─┘   └─────────────┬────────────┘└───────┬───────┘ └────────────┬────────────┘ └┬┘
+  scheme          authority                  path                  query           fragment
+
+Exposing Settings
+~~~~~~~~~~~~~~~~~
+
+Since Dataverse 4, database settings have been exposed via API at http://localhost:8080/api/admin/settings
+
+(JVM options are probably available via the Payara REST API, but this is out of scope.)
+
+Settings need to be exposed outside to API clients outside of ``/api/admin`` (which is typically restricted to localhost). Here are some guidelines to follow when exposing settings.
+
+- When you are exposing a database setting as-is:
+
+  - Use ``/api/info/settings`` as the root path.
+
+  - Append the name of the setting including the colon (e.g. ``:DatasetPublishPopupCustomText``)
+
+  - Final path example: ``/api/info/settings/:DatasetPublishPopupCustomText``
+
+- If the absence of the database setting is filled in by a default value (e.g. ``:ZipDownloadLimit`` or ``:ApiTermsOfUse``):
+
+  - Use ``/api/info`` as the root path.
+
+  - Append the setting but remove the colon and downcase the first character (e.g. ``zipDownloadLimit``)
+
+  - Final path example: ``/api/info/zipDownloadLimit``
+
+- If the database setting you're exposing make more sense outside of ``/api/info`` because there's more context (e.g. ``:CustomDatasetSummaryFields``):
+
+  - Feel free to use a path outside of ``/api/info`` as the root path.
+
+  - Given additional context, append a shortened name (e.g. ``/api/datasets/summaryFieldNames``).
+
+  - Final path example: ``/api/datasets/summaryFieldNames``
+
+- If you need to expose a JVM option (MicroProfile setting) such as ``dataverse.api.allow-incomplete-metadata``:
+
+  - Use ``/api/info`` as the root path.
+
+  - Append a meaningful name for the setting (e.g. ``incompleteMetadataViaApi``).
+
+  - Final path example: ``/api/info/incompleteMetadataViaApi``
+

doc/sphinx-guides/source/developers/index.rst

@@ -19,6 +19,7 @@ Developer Guide
    sql-upgrade-scripts
    testing
    documentation
+   api-design
    security
    dependencies
    debugging

doc/sphinx-guides/source/installation/config.rst

@@ -304,6 +304,8 @@ Here are the configuration options for PermaLinks:
 - :ref:`:DataFilePIDFormat <:DataFilePIDFormat>` (optional)
 - :ref:`:FilePIDsEnabled <:FilePIDsEnabled>` (optional, defaults to false)
 
+You must restart Payara after making changes to these settings.
+
 .. _auth-modes:
 
 Auth Modes: Local vs. Remote vs. Both
@@ -2980,6 +2982,8 @@ This setting controls the number of files that can be uploaded through the UI at
 
 ``curl -X PUT -d 500 http://localhost:8080/api/admin/settings/:MultipleUploadFilesLimit``
 
+.. _:ZipDownloadLimit:
+
 :ZipDownloadLimit
 +++++++++++++++++
 

scripts/installer/install.py

@@ -413,7 +413,7 @@
 
    # 3e. set permissions:
 
-   conn_cmd = "GRANT CREATE PRIVILEGES on DATABASE "+pgDb+" to "+pgUser+";"
+   conn_cmd = "GRANT ALL PRIVILEGES on DATABASE "+pgDb+" to "+pgUser+";"
    try:
       cur.execute(conn_cmd)
    except:
@@ -422,7 +422,7 @@
    conn.close()
 
    if int(pg_major_version) >= 15:
-      conn_cmd = "GRANT ALL ON SCHEMA public TO "+pgUser+";"
+      conn_cmd = "GRANT CREATE ON SCHEMA public TO "+pgUser+";"
       print("PostgreSQL 15 or higher detected. Running " + conn_cmd)
       try:
          cur.execute(conn_cmd)

src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java

@@ -708,6 +708,12 @@ protected Response ok( boolean value ) {
             .add("data", value).build() ).build();
     }
 
+    protected Response ok(long value) {
+        return Response.ok().entity(Json.createObjectBuilder()
+                .add("status", ApiConstants.STATUS_OK)
+                .add("data", value).build()).build();
+    }
+
     /**
      * @param data Payload to return.
      * @param mediaType Non-JSON media type.

src/main/java/edu/harvard/iq/dataverse/api/Files.java

@@ -726,6 +726,11 @@ public Response reingest(@Context ContainerRequestContext crc, @PathParam("id")
     public Response redetectDatafile(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("dryRun") boolean dryRun) {
         try {
             DataFile dataFileIn = findDataFileOrDie(id);
+            // Ingested Files have mimetype = text/tab-separated-values
+            // No need to redetect
+            if (dataFileIn.isTabularData()) {
+                return error(Response.Status.BAD_REQUEST, "The file is an ingested tabular file.");
+            }
             String originalContentType = dataFileIn.getContentType();
             DataFile dataFileOut = execCommand(new RedetectFileTypeCommand(createDataverseRequest(getRequestUser(crc)), dataFileIn, dryRun));
             NullSafeJsonBuilder result = NullSafeJsonBuilder.jsonObjectBuilder()

src/main/java/edu/harvard/iq/dataverse/api/Info.java

@@ -25,14 +25,15 @@ public class Info extends AbstractApiBean {
     @GET
     @Path("settings/:DatasetPublishPopupCustomText")
     public Response getDatasetPublishPopupCustomText() {
-        String setting = settingsService.getValueForKey(SettingsServiceBean.Key.DatasetPublishPopupCustomText);
-        if (setting != null) {
-            return ok(Json.createObjectBuilder().add("message", setting));
-        } else {
-            return notFound("Setting " + SettingsServiceBean.Key.DatasetPublishPopupCustomText + " not found");
-        }
+        return getSettingResponseByKey(SettingsServiceBean.Key.DatasetPublishPopupCustomText);
     }
-
+
+    @GET
+    @Path("settings/:MaxEmbargoDurationInMonths")
+    public Response getMaxEmbargoDurationInMonths() {
+        return getSettingResponseByKey(SettingsServiceBean.Key.MaxEmbargoDurationInMonths);
+    }
+
     @GET
     @AuthRequired
     @Path("version")
@@ -41,28 +42,44 @@ public Response getInfo(@Context ContainerRequestContext crc) {
         String[] comps = versionStr.split("build",2);
         String version = comps[0].trim();
         JsonValue build = comps.length > 1 ? Json.createArrayBuilder().add(comps[1].trim()).build().get(0) : JsonValue.NULL;
-        
+
         return response( req -> ok( Json.createObjectBuilder().add("version", version)
                                                               .add("build", build)), getRequestUser(crc));
     }
-    
+
     @GET
     @AuthRequired
     @Path("server")
     public Response getServer(@Context ContainerRequestContext crc) {
         return response( req -> ok(JvmSettings.FQDN.lookup()), getRequestUser(crc));
     }
-    
+
     @GET
     @AuthRequired
     @Path("apiTermsOfUse")
     public Response getTermsOfUse(@Context ContainerRequestContext crc) {
         return response( req -> ok(systemConfig.getApiTermsOfUse()), getRequestUser(crc));
     }
-    
+
     @GET
     @Path("settings/incompleteMetadataViaApi")
     public Response getAllowsIncompleteMetadata() {
         return ok(JvmSettings.API_ALLOW_INCOMPLETE_METADATA.lookupOptional(Boolean.class).orElse(false));
     }
+
+    @GET
+    @Path("zipDownloadLimit")
+    public Response getZipDownloadLimit() {
+        long zipDownloadLimit = SystemConfig.getLongLimitFromStringOrDefault(settingsSvc.getValueForKey(SettingsServiceBean.Key.ZipDownloadLimit), SystemConfig.defaultZipDownloadLimit);
+        return ok(zipDownloadLimit);
+    }
+
+    private Response getSettingResponseByKey(SettingsServiceBean.Key key) {
+        String setting = settingsService.getValueForKey(key);
+        if (setting != null) {
+            return ok(Json.createObjectBuilder().add("message", setting));
+        } else {
+            return notFound("Setting " + key + " not found");
+        }
+    }
 }

src/main/java/edu/harvard/iq/dataverse/ingest/metadataextraction/spi/FileMetadataExtractorSpi.java

@@ -10,9 +10,7 @@
 import java.util.logging.*;
 import java.io.*;
 
-import edu.harvard.iq.dataverse.ingest.plugin.spi.RegisterableService;
 import edu.harvard.iq.dataverse.ingest.plugin.spi.IngestServiceProvider;
-import edu.harvard.iq.dataverse.ingest.plugin.spi.ServiceRegistry;
 import java.nio.MappedByteBuffer;
 import java.util.Locale;
 
@@ -44,22 +42,6 @@ public FileMetadataExtractorSpi(String vendorName, String version) {
         this.version = version;
     }
 
-    public void onRegistration(ServiceRegistry registry,
-                               Class<?> category) {}
-
-
-    public void onDeregistration(ServiceRegistry registry,
-                                 Class<?> category) {}
-
-    public String getVersion() {
-        return version;
-    }
-
-    public String getVendorName() {
-        return vendorName;
-    }
-
-
     public abstract String getDescription(Locale locale);
 
     protected String[] names = null;