Skip to content

Commit

Permalink
Merge branch 'develop' of github.com:IQSS/dataverse into 10280-get-fi…
Browse files Browse the repository at this point in the history 
…le-api-extension
  • Loading branch information
@GPortas
GPortas committed Feb 15, 2024
2 parents 227fe53 + f456e51 commit d7f8040
Show file tree
Hide file tree
Showing 11 changed files with 118 additions and 27 deletions.
14 changes: 14 additions & 0 deletions doc/release-notes/9983-unique-constraints.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
This release adds two missing database constraints that will assure that the externalvocabularyvalue table only has one entry for each uri and that the oaiset table only has one set for each spec. (In the very unlikely case that your existing database has duplicate entries now, install would fail. This can be checked by running

SELECT uri, count(*) FROM externalvocabularyvaluet group by uri;

and

SELECT spec, count(*) FROM oaiset group by spec;

and then removing any duplicate rows (where count>1).




TODO: Whoever puts the release notes together should make sure there is the standard note about reloading metadata blocks for the citation, astrophysics, and biomedical blocks (plus any others from other PRs) after upgrading.
6 changes: 3 additions & 3 deletions scripts/api/data/metadatablocks/astrophysics.tsv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
astrophysics Astronomy and Astrophysics Metadata
#datasetField name title description watermark fieldType displayOrder displayFormat advancedSearchField allowControlledVocabulary allowmultiples facetable displayoncreate required parent metadatablock_id
astroType Type The nature or genre of the content of the files in the dataset. text 0 TRUE TRUE TRUE TRUE FALSE FALSE astrophysics
astroFacility Facility The observatory or facility where the data was obtained. text 1 TRUE TRUE TRUE TRUE FALSE FALSE astrophysics
astroInstrument Instrument The instrument used to collect the data. text 2 TRUE TRUE TRUE TRUE FALSE FALSE astrophysics
astroFacility Facility The observatory or facility where the data was obtained. text 1 TRUE FALSE TRUE TRUE FALSE FALSE astrophysics
astroInstrument Instrument The instrument used to collect the data. text 2 TRUE FALSE TRUE TRUE FALSE FALSE astrophysics
astroObject Object Astronomical Objects represented in the data (Given as SIMBAD recognizable names preferred). text 3 TRUE FALSE TRUE TRUE FALSE FALSE astrophysics
resolution.Spatial Spatial Resolution The spatial (angular) resolution that is typical of the observations, in decimal degrees. text 4 TRUE FALSE FALSE TRUE FALSE FALSE astrophysics
resolution.Spectral Spectral Resolution The spectral resolution that is typical of the observations, given as the ratio \u03bb/\u0394\u03bb. text 5 TRUE FALSE FALSE TRUE FALSE FALSE astrophysics
resolution.Temporal Time Resolution The temporal resolution that is typical of the observations, given in seconds. text 6 FALSE FALSE FALSE FALSE FALSE FALSE astrophysics
coverage.Spectral.Bandpass Bandpass Conventional bandpass name text 7 TRUE TRUE TRUE TRUE FALSE FALSE astrophysics
coverage.Spectral.Bandpass Bandpass Conventional bandpass name text 7 TRUE FALSE TRUE TRUE FALSE FALSE astrophysics
coverage.Spectral.CentralWavelength Central Wavelength (m) The central wavelength of the spectral bandpass, in meters. Enter a floating-point number. float 8 TRUE FALSE TRUE TRUE FALSE FALSE astrophysics
coverage.Spectral.Wavelength Wavelength Range The minimum and maximum wavelength of the spectral bandpass. Enter a floating-point number. none 9 FALSE FALSE TRUE FALSE FALSE FALSE astrophysics
coverage.Spectral.MinimumWavelength Minimum (m) The minimum wavelength of the spectral bandpass, in meters. Enter a floating-point number. float 10 TRUE FALSE FALSE TRUE FALSE FALSE coverage.Spectral.Wavelength astrophysics
Expand Down
2 changes: 1 addition & 1 deletion scripts/api/data/metadatablocks/biomedical.tsv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
studyAssayOtherTechnologyType Other Technology Type If Other was selected in Technology Type, list any other technology types that were used in this Dataset. text 9 TRUE FALSE TRUE TRUE FALSE FALSE biomedical
studyAssayPlatform Technology Platform The manufacturer and name of the technology platform used in the assay (e.g. Bruker AVANCE). text 10 TRUE TRUE TRUE TRUE FALSE FALSE biomedical
studyAssayOtherPlatform Other Technology Platform If Other was selected in Technology Platform, list any other technology platforms that were used in this Dataset. text 11 TRUE FALSE TRUE TRUE FALSE FALSE biomedical
studyAssayCellType Cell Type The name of the cell line from which the source or sample derives. text 12 TRUE TRUE TRUE TRUE FALSE FALSE biomedical
studyAssayCellType Cell Type The name of the cell line from which the source or sample derives. text 12 TRUE FALSE TRUE TRUE FALSE FALSE biomedical
#controlledVocabulary DatasetField Value identifier displayOrder
studyDesignType Case Control EFO_0001427 0
studyDesignType Cross Sectional EFO_0001428 1
Expand Down
2 changes: 1 addition & 1 deletion scripts/api/data/metadatablocks/citation.tsv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
seriesName Name The name of the dataset series text 66 #VALUE TRUE FALSE FALSE TRUE FALSE FALSE series citation
seriesInformation Information Can include 1) a history of the series and 2) a summary of features that apply to the series textbox 67 #VALUE FALSE FALSE FALSE FALSE FALSE FALSE series citation
software Software Information about the software used to generate the Dataset none 68 , FALSE FALSE TRUE FALSE FALSE FALSE citation https://www.w3.org/TR/prov-o/#wasGeneratedBy
softwareName Name The name of software used to generate the Dataset text 69 #VALUE FALSE TRUE FALSE FALSE FALSE FALSE software citation
softwareName Name The name of software used to generate the Dataset text 69 #VALUE FALSE FALSE FALSE FALSE FALSE FALSE software citation
softwareVersion Version The version of the software used to generate the Dataset, e.g. 4.11 text 70 #NAME: #VALUE FALSE FALSE FALSE FALSE FALSE FALSE software citation
relatedMaterial Related Material Information, such as a persistent ID or citation, about the material related to the Dataset, such as appendices or sampling information available outside of the Dataset textbox 71 FALSE FALSE TRUE FALSE FALSE FALSE citation
relatedDatasets Related Dataset Information, such as a persistent ID or citation, about a related dataset, such as previous research on the Dataset's subject textbox 72 FALSE FALSE TRUE FALSE FALSE FALSE citation http://purl.org/dc/terms/relation
Expand Down
7 changes: 6 additions & 1 deletion src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

import jakarta.ejb.EJB;
import jakarta.ejb.Stateless;
import jakarta.ejb.TransactionAttribute;
import jakarta.ejb.TransactionAttributeType;
import jakarta.inject.Named;
import jakarta.json.Json;
import jakarta.json.JsonArray;
Expand All @@ -34,6 +36,7 @@
import jakarta.persistence.NoResultException;
import jakarta.persistence.NonUniqueResultException;
import jakarta.persistence.PersistenceContext;
import jakarta.persistence.PersistenceException;
import jakarta.persistence.TypedQuery;

import org.apache.commons.codec.digest.DigestUtils;
Expand All @@ -46,7 +49,6 @@
import org.apache.http.impl.client.HttpClients;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;

import edu.harvard.iq.dataverse.settings.SettingsServiceBean;

/**
Expand Down Expand Up @@ -448,6 +450,7 @@ public JsonObject getExternalVocabularyValue(String termUri) {
* @param cvocEntry - the configuration for the DatasetFieldType associated with this term
* @param term - the term uri as a string
*/
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void registerExternalTerm(JsonObject cvocEntry, String term) {
String retrievalUri = cvocEntry.getString("retrieval-uri");
String prefix = cvocEntry.getString("prefix", null);
Expand Down Expand Up @@ -518,6 +521,8 @@ public void process(HttpResponse response, HttpContext context) throws HttpExcep
logger.fine("Wrote value for term: " + term);
} catch (JsonException je) {
logger.severe("Error retrieving: " + retrievalUri + " : " + je.getMessage());
} catch (PersistenceException e) {
logger.fine("Problem persisting: " + retrievalUri + " : " + e.getMessage());
}
} else {
logger.severe("Received response code : " + statusCode + " when retrieving " + retrievalUri
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ public void setDisplayOnCreate(boolean displayOnCreate) {
}

public boolean isControlledVocabulary() {
return controlledVocabularyValues != null && !controlledVocabularyValues.isEmpty();
return allowControlledVocabulary;
}

/**
Expand Down
28 changes: 28 additions & 0 deletions src/main/java/edu/harvard/iq/dataverse/api/TestApi.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,34 @@ public Response getExternalToolsforFile(@PathParam("id") String idSupplied, @Que
return wr.getResponse();
}
}

@GET
@Path("datasets/{id}/externalTool/{toolId}")
public Response getExternalToolforDatasetById(@PathParam("id") String idSupplied, @PathParam("toolId") String toolId, @QueryParam("type") String typeSupplied) {
ExternalTool.Type type;
try {
type = ExternalTool.Type.fromString(typeSupplied);
} catch (IllegalArgumentException ex) {
return error(BAD_REQUEST, ex.getLocalizedMessage());
}
Dataset dataset;
try {
dataset = findDatasetOrDie(idSupplied);
JsonArrayBuilder tools = Json.createArrayBuilder();
List<ExternalTool> datasetTools = externalToolService.findDatasetToolsByType(type);
for (ExternalTool tool : datasetTools) {
ApiToken apiToken = externalToolService.getApiToken(getRequestApiKey());
ExternalToolHandler externalToolHandler = new ExternalToolHandler(tool, dataset, apiToken, null);
JsonObjectBuilder toolToJson = externalToolService.getToolAsJsonWithQueryParameters(externalToolHandler);
if (tool.getId().toString().equals(toolId)) {
return ok(toolToJson);
}
}
} catch (WrappedResponse wr) {
return wr.getResponse();
}
return error(BAD_REQUEST, "Could not find external tool with id of " + toolId);
}

@Path("files/{id}/externalTools")
@GET
Expand Down
20 changes: 17 additions & 3 deletions src/main/java/edu/harvard/iq/dataverse/dataaccess/ImageThumbConverter.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ private static boolean generatePDFThumbnail(StorageIO<DataFile> storageIO, int s
// will run the ImageMagick on it, and will save its output in another temp
// file, and will save it as an "auxiliary" file via the driver.
boolean tempFilesRequired = false;
File tempFile = null;

try {
Path pdfFilePath = storageIO.getFileSystemPath();
Expand All @@ -225,7 +226,7 @@ private static boolean generatePDFThumbnail(StorageIO<DataFile> storageIO, int s
}

if (tempFilesRequired) {
InputStream inputStream = null;
InputStream inputStream = null;
try {
storageIO.open();
inputStream = storageIO.getInputStream();
Expand All @@ -234,12 +235,11 @@ private static boolean generatePDFThumbnail(StorageIO<DataFile> storageIO, int s
return false;
}

File tempFile;
OutputStream outputStream = null;
try {
tempFile = File.createTempFile("tempFileToRescale", ".tmp");
outputStream = new FileOutputStream(tempFile);
//Reads/transfers all bytes from the input stream to the output stream.
//Reads/transfers all bytes from the input stream to the output stream.
inputStream.transferTo(outputStream);
} catch (IOException ioex) {
logger.warning("GenerateImageThumb: failed to save pdf bytes in a temporary file.");
Expand Down Expand Up @@ -270,6 +270,12 @@ private static boolean generatePDFThumbnail(StorageIO<DataFile> storageIO, int s
logger.warning("failed to save generated pdf thumbnail, as AUX file " + THUMBNAIL_SUFFIX + size + "!");
return false;
}
finally {
try {
tempFile.delete();
}
catch (Exception e) {}
}
}

return true;
Expand Down Expand Up @@ -371,6 +377,14 @@ private static boolean generateImageThumbnailFromInputStream(StorageIO<DataFile>
logger.warning("Failed to rescale and/or save the image: " + ioex.getMessage());
return false;
}
finally {
if(tempFileRequired) {
try {
tempFile.delete();
}
catch (Exception e) {}
}
}

return true;

Expand Down
16 changes: 16 additions & 0 deletions src/main/resources/db/migration/V6.1.0.3__9983-missing-unique-constraints.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
DO $$
BEGIN

BEGIN
ALTER TABLE externalvocabularyvalue ADD CONSTRAINT externalvocabularvalue_uri_key UNIQUE(uri);
EXCEPTION
WHEN duplicate_table THEN RAISE NOTICE 'Table unique constraint externalvocabularvalue_uri_key already exists';
END;

BEGIN
ALTER TABLE oaiset ADD CONSTRAINT oaiset_spec_key UNIQUE(spec);
EXCEPTION
WHEN duplicate_table THEN RAISE NOTICE 'Table unique constraint oaiset_spec_key already exists';
END;

END $$;
33 changes: 16 additions & 17 deletions src/test/java/edu/harvard/iq/dataverse/api/ExternalToolsIT.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,20 +197,20 @@ public void testDatasetLevelTool1() {
.statusCode(OK.getStatusCode())
.body("data.displayName", CoreMatchers.equalTo("DatasetTool1"));

long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");
Long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");

Response getExternalToolsByDatasetIdInvalidType = UtilIT.getExternalToolsForDataset(datasetId.toString(), "invalidType", apiToken);
getExternalToolsByDatasetIdInvalidType.prettyPrint();
getExternalToolsByDatasetIdInvalidType.then().assertThat()
.statusCode(BAD_REQUEST.getStatusCode())
.body("message", CoreMatchers.equalTo("Type must be one of these values: [explore, configure, preview, query]."));

Response getExternalToolsByDatasetId = UtilIT.getExternalToolsForDataset(datasetId.toString(), "explore", apiToken);
Response getExternalToolsByDatasetId = UtilIT.getExternalToolForDatasetById(datasetId.toString(), "explore", apiToken, toolId.toString());
getExternalToolsByDatasetId.prettyPrint();
getExternalToolsByDatasetId.then().assertThat()
.body("data[0].displayName", CoreMatchers.equalTo("DatasetTool1"))
.body("data[0].scope", CoreMatchers.equalTo("dataset"))
.body("data[0].toolUrlWithQueryParams", CoreMatchers.equalTo("http://datasettool1.com?datasetPid=" + datasetPid + "&key=" + apiToken))
.body("data.displayName", CoreMatchers.equalTo("DatasetTool1"))
.body("data.scope", CoreMatchers.equalTo("dataset"))
.body("data.toolUrlWithQueryParams", CoreMatchers.equalTo("http://datasettool1.com?datasetPid=" + datasetPid + "&key=" + apiToken))
.statusCode(OK.getStatusCode());

//Delete the tool added by this test...
Expand Down Expand Up @@ -271,15 +271,14 @@ public void testDatasetLevelToolConfigure() {
.statusCode(OK.getStatusCode())
.body("data.displayName", CoreMatchers.equalTo("Dataset Configurator"));

long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");

Response getExternalToolsByDatasetId = UtilIT.getExternalToolsForDataset(datasetId.toString(), "configure", apiToken);
Long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");
Response getExternalToolsByDatasetId = UtilIT.getExternalToolForDatasetById(datasetId.toString(), "configure", apiToken, toolId.toString());
getExternalToolsByDatasetId.prettyPrint();
getExternalToolsByDatasetId.then().assertThat()
.body("data[0].displayName", CoreMatchers.equalTo("Dataset Configurator"))
.body("data[0].scope", CoreMatchers.equalTo("dataset"))
.body("data[0].types[0]", CoreMatchers.equalTo("configure"))
.body("data[0].toolUrlWithQueryParams", CoreMatchers.equalTo("https://datasetconfigurator.com?datasetPid=" + datasetPid))
.body("data.displayName", CoreMatchers.equalTo("Dataset Configurator"))
.body("data.scope", CoreMatchers.equalTo("dataset"))
.body("data.types[0]", CoreMatchers.equalTo("configure"))
.body("data.toolUrlWithQueryParams", CoreMatchers.equalTo("https://datasetconfigurator.com?datasetPid=" + datasetPid))
.statusCode(OK.getStatusCode());

//Delete the tool added by this test...
Expand Down Expand Up @@ -594,7 +593,7 @@ public void testFileLevelToolWithAuxFileReq() throws IOException {
.statusCode(OK.getStatusCode())
.body("data.displayName", CoreMatchers.equalTo("HDF5 Tool"));

long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");
Long toolId = JsonPath.from(addExternalTool.getBody().asString()).getLong("data.id");

Response getTool = UtilIT.getExternalTool(toolId);
getTool.prettyPrint();
Expand All @@ -610,13 +609,13 @@ public void testFileLevelToolWithAuxFileReq() throws IOException {
.body("data", Matchers.hasSize(0));

// The tool shows for a true HDF5 file. The NcML aux file is available. Requirements met.
Response getToolsForTrueHdf5 = UtilIT.getExternalToolsForFile(trueHdf5.toString(), "preview", apiToken);
Response getToolsForTrueHdf5 = UtilIT.getExternalToolForFileById(trueHdf5.toString(), "preview", apiToken, toolId.toString());
getToolsForTrueHdf5.prettyPrint();
getToolsForTrueHdf5.then().assertThat()
.statusCode(OK.getStatusCode())
.body("data[0].displayName", CoreMatchers.equalTo("HDF5 Tool"))
.body("data[0].scope", CoreMatchers.equalTo("file"))
.body("data[0].contentType", CoreMatchers.equalTo("application/x-hdf5"));
.body("data.displayName", CoreMatchers.equalTo("HDF5 Tool"))
.body("data.scope", CoreMatchers.equalTo("file"))
.body("data.contentType", CoreMatchers.equalTo("application/x-hdf5"));

//Delete the tool added by this test...
Response deleteExternalTool = UtilIT.deleteExternalTool(toolId);
Expand Down
15 changes: 15 additions & 0 deletions src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2350,6 +2350,21 @@ static Response getExternalToolsForDataset(String idOrPersistentIdOfDataset, Str
}
return requestSpecification.get("/api/admin/test/datasets/" + idInPath + "/externalTools?type=" + type + optionalQueryParam);
}

static Response getExternalToolForDatasetById(String idOrPersistentIdOfDataset, String type, String apiToken, String toolId) {
String idInPath = idOrPersistentIdOfDataset; // Assume it's a number.
String optionalQueryParam = ""; // If idOrPersistentId is a number we'll just put it in the path.
if (!NumberUtils.isCreatable(idOrPersistentIdOfDataset)) {
idInPath = ":persistentId";
optionalQueryParam = "&persistentId=" + idOrPersistentIdOfDataset;
}
RequestSpecification requestSpecification = given();
if (apiToken != null) {
requestSpecification = given()
.header(UtilIT.API_TOKEN_HTTP_HEADER, apiToken);
}
return requestSpecification.get("/api/admin/test/datasets/" + idInPath + "/externalTool/" + toolId + "?type=" + type + optionalQueryParam);
}

static Response getExternalToolsForFile(String idOrPersistentIdOfFile, String type, String apiToken) {
String idInPath = idOrPersistentIdOfFile; // Assume it's a number.
Expand Down

0 comments on commit d7f8040

Please sign in to comment.