Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IQSS 9375 - Retention period #10336

Merged
merged 46 commits into from
May 1, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
5d07f7c
Implemented Retention class and it's persistency in the database
PaulBoon Feb 20, 2024
1cd5968
Implemented set and unset retention API
PaulBoon Feb 21, 2024
c113e66
Implemented minimal retention output on the GUI
PaulBoon Feb 21, 2024
fd4a628
Initial implementation for the retention dialog
PaulBoon Feb 22, 2024
06c7551
Some fixes for the retention dialog
PaulBoon Feb 26, 2024
fa0ddf8
Added missing property for file.assignedRetention.success
PaulBoon Feb 27, 2024
677704b
Default Retention constructor now has 1000 years period
PaulBoon Feb 27, 2024
a353418
Remove referenced retentions with file deletion
PaulBoon Feb 27, 2024
f571f68
Initial implementation for making files unavailable after the retenti…
PaulBoon Feb 29, 2024
8a720a1
Disable access request for files after the retention period
PaulBoon Feb 29, 2024
013eee2
Disable access for superusers after the retention period
PaulBoon Mar 5, 2024
07c944c
Changed retention wording; files are unavailable when the retention h…
PaulBoon Mar 5, 2024
cc4c42a
Initial documentation for the retention
PaulBoon Mar 6, 2024
4a55b0a
Fixed isPubliclyDownloadable for the retention
PaulBoon Mar 7, 2024
d77c28f
Implemented unit tests for the file retention
PaulBoon Mar 7, 2024
3432afb
Added minimal integration test for the file retention
PaulBoon Mar 12, 2024
2be3c0a
Added file retention info to metadata export for OAI_ORE and datavers…
PaulBoon Mar 12, 2024
5b2f397
Use minimal retention period to initialise the date unavailable on th…
PaulBoon Mar 12, 2024
7d51ec4
Do not allow full-text indexing for files with a retention, even if i…
PaulBoon Mar 12, 2024
1077a4f
Merged with upstream develop
PaulBoon Mar 12, 2024
52c5be5
Undo unwanted change in astrophysics.properties
PaulBoon Mar 20, 2024
c03e635
Improved retention section of the dataset management guide
PaulBoon Mar 20, 2024
ada8e0f
Improved retention section of the native api guide
PaulBoon Mar 20, 2024
005c823
Fixe file indexing with embargo and or retention dates
PaulBoon Mar 20, 2024
ac96deb
Added searching with the RetentionPeriodExpired file access status
PaulBoon Mar 20, 2024
3257b87
Added RetentionPeriodExpired to the native api guide
PaulBoon Mar 20, 2024
21db794
Change 'Retention Expired' to 'Retention Period Expired'
PaulBoon Mar 21, 2024
81cd03d
Fix metadata display of end dates of embargo and retention
PaulBoon Mar 25, 2024
6005657
File status background red (label-danger) for retention expired
PaulBoon Mar 25, 2024
d95cd7f
Changed the retention metadata GUI text
PaulBoon Mar 26, 2024
9587c1f
minor text changes and removing extra changes in schema.xml
qqmyers Mar 27, 2024
a760bb2
Fixing file publication date metadata field display
PaulBoon Mar 27, 2024
e29fcfc
Merge pull request #7 from GlobalDataverseCommunityConsortium/Retenti…
PaulBoon Mar 27, 2024
83ab52a
Removed unused embargoDate.label from the Bundle.properties
PaulBoon Mar 27, 2024
0c665eb
Improved file publication date metadata field display
PaulBoon Mar 28, 2024
d5b5331
Use label-warning for status labels at dataset level and label-danger…
PaulBoon Mar 28, 2024
8826e7c
Improved file publication date metadata field display
PaulBoon Mar 28, 2024
857833f
Merge branch 'develop' into RetentionPeriod
PaulBoon Apr 15, 2024
00f9dc1
Renamed flyway SQL upgrade script for the retention
PaulBoon Apr 15, 2024
8758557
Added release notes for the retention period
PaulBoon Apr 18, 2024
35f9da0
Merge branch 'develop' of github.com:IQSS/dataverse into RetentionPeriod
PaulBoon Apr 29, 2024
3a3f6cd
Changed selected file download 'warning' messages to include the expi…
PaulBoon Apr 30, 2024
43fbbf2
Improve messages for the set-retention and unset-retention API
PaulBoon May 1, 2024
3cdd3c6
Improve input validation with response messages for the set-retention…
PaulBoon May 1, 2024
639c162
More improvements on input validation with response messages for the …
PaulBoon May 1, 2024
8f65a46
Improvements on input validation with response messages for the unset…
PaulBoon May 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/main/java/edu/harvard/iq/dataverse/DataFile.java
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,18 @@ public void setEmbargo(Embargo embargo) {
this.embargo = embargo;
}

@ManyToOne
@JoinColumn(name="retention_id")
private Retention retention;

public Retention getRetention() {
return retention;
}

public void setRetention(Retention retention) {
this.retention = retention;
}

public DataFile() {
this.fileMetadatas = new ArrayList<>();
initFileReplaceAttributes();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1366,7 +1366,10 @@ public Embargo findEmbargo(Long id) {
DataFile d = find(id);
return d.getEmbargo();
}


public boolean isActivelyRetended(FileMetadata fm) {
PaulBoon marked this conversation as resolved.
Show resolved Hide resolved
return FileUtil.isActivelyRetended(fm);
}
/**
* Checks if the supplied DvObjectContainer (Dataset or Collection; although
* only collection-level storage quotas are officially supported as of now)
Expand Down
218 changes: 215 additions & 3 deletions src/main/java/edu/harvard/iq/dataverse/DatasetPage.java
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,8 @@ public enum DisplayMode {
@Inject
EmbargoServiceBean embargoService;
@Inject
RetentionServiceBean retentionService;
@Inject
LicenseServiceBean licenseServiceBean;
@Inject
DataFileCategoryServiceBean dataFileCategoryService;
Expand Down Expand Up @@ -3660,6 +3662,25 @@ public String deleteFiles() throws CommandException{
}
}

//Remove retentions that are no longer referenced
//Identify which ones are involved here
List<Retention> orphanedRetentions = new ArrayList<Retention>();
if (selectedFiles != null && selectedFiles.size() > 0) {
for (FileMetadata fmd : workingVersion.getFileMetadatas()) {
for (FileMetadata fm : selectedFiles) {
if (fm.getDataFile().equals(fmd.getDataFile()) && !fmd.getDataFile().isReleased()) {
Retention ret = fmd.getDataFile().getRetention();
if (ret != null) {
ret.getDataFiles().remove(fmd.getDataFile());
if (ret.getDataFiles().isEmpty()) {
orphanedRetentions.add(ret);
}
}
}
}
}
}

deleteFiles(filesToDelete);
String retVal;

Expand All @@ -3669,12 +3690,14 @@ public String deleteFiles() throws CommandException{
} else {
retVal = save();
}


//And delete them only after the dataset is updated

// And delete them only after the dataset is updated
for(Embargo emb: orphanedEmbargoes) {
embargoService.deleteById(emb.getId(), ((AuthenticatedUser)session.getUser()).getUserIdentifier());
}
for(Retention ret: orphanedRetentions) {
retentionService.delete(ret, ((AuthenticatedUser)session.getUser()).getUserIdentifier());
}
return retVal;

}
Expand Down Expand Up @@ -6278,6 +6301,195 @@ private boolean containsOnlyActivelyEmbargoedFiles(List<FileMetadata> selectedFi
return true;
}

public Retention getSelectionRetention() {
return selectionRetention;
}

public void setSelectionRetention(Retention selectionRetention) {
this.selectionRetention = selectionRetention;
}


private Retention selectionRetention = new Retention();

public boolean isValidRetentionSelection() {
//If fileMetadataForAction is set, someone is using the kebab/single file menu
if (fileMetadataForAction != null) {
if (!fileMetadataForAction.getDataFile().isReleased()) {
return true;
} else {
return false;
}
}
//Otherwise we check the selected files
for (FileMetadata fmd : selectedFiles) {
if (!fmd.getDataFile().isReleased()) {
return true;
}
}
return false;
}

/*
* This method checks to see if the selected file/files have a retention that could be removed. It doesn't return true of a released file has a retention.
*/
public boolean isExistingRetention() {
if (fileMetadataForAction != null) {
if (!fileMetadataForAction.getDataFile().isReleased()
&& (fileMetadataForAction.getDataFile().getRetention() != null)) {
return true;
} else {
return false;
}
}
for (FileMetadata fmd : selectedFiles) {
if (!fmd.getDataFile().isReleased() && (fmd.getDataFile().getRetention() != null)) {
return true;
}
}

return false;
}

public boolean isActivelyRetended(List<FileMetadata> fmdList) {
return FileUtil.isActivelyRetended(fmdList);
}

public boolean isRetentionForWholeSelection() {
for (FileMetadata fmd : selectedFiles) {
if (fmd.getDataFile().isReleased()) {
return false;
}
}
return true;
}

private boolean removeRetention=false;

public boolean isRemoveRetention() {
return removeRetention;
}

public void setRemoveRetention(boolean removeRetention) {
boolean existing = this.removeRetention;
this.removeRetention = removeRetention;
//If we flipped the state, update the selectedRetention. Otherwise (e.g. when save is hit) don't make changes
if(existing != this.removeRetention) {
logger.fine("State flip");
selectionRetention= new Retention();
if(removeRetention) {
logger.fine("Setting empty retention");
selectionRetention= new Retention(null, null);
}
PrimeFaces.current().resetInputs("datasetForm:retentionInputs");
}
}

public String saveRetention() {
if (workingVersion.isReleased()) {
refreshSelectedFiles(selectedFiles);
}

if(isRemoveRetention() || (selectionRetention.getDateUnavailable()==null && selectionRetention.getReason()==null)) {
selectionRetention=null;
}

if(!(selectionRetention==null || (selectionRetention!=null && settingsWrapper.isValidRetentionDate(selectionRetention)))) {
logger.fine("Validation error: " + selectionRetention.getFormattedDateUnavailable());
FacesContext.getCurrentInstance().validationFailed();
return "";
}
List<Retention> orphanedRetentions = new ArrayList<Retention>();
List<FileMetadata> retentionFMs = null;
if (fileMetadataForAction != null) {
retentionFMs = new ArrayList<FileMetadata>();
retentionFMs.add(fileMetadataForAction);
} else if (selectedFiles != null && selectedFiles.size() > 0) {
retentionFMs = selectedFiles;
}

if(retentionFMs!=null && !retentionFMs.isEmpty()) {
if(selectionRetention!=null) {
selectionRetention = retentionService.merge(selectionRetention);
}
for (FileMetadata fmd : workingVersion.getFileMetadatas()) {
for (FileMetadata fm : retentionFMs) {
if (fm.getDataFile().equals(fmd.getDataFile()) && (isSuperUser()||!fmd.getDataFile().isReleased())) {
Retention ret = fmd.getDataFile().getRetention();
if (ret != null) {
logger.fine("Before: " + ret.getDataFiles().size());
ret.getDataFiles().remove(fmd.getDataFile());
if (ret.getDataFiles().isEmpty()) {
orphanedRetentions.add(ret);
}
logger.fine("After: " + ret.getDataFiles().size());
}
fmd.getDataFile().setRetention(selectionRetention);
}
}
}
}
if (selectionRetention != null) {
retentionService.save(selectionRetention, ((AuthenticatedUser) session.getUser()).getIdentifier());
}
// success message:
String successMessage = BundleUtil.getStringFromBundle("file.assignedRetention.success");
logger.fine(successMessage);
successMessage = successMessage.replace("{0}", "Selected Files");
JsfHelper.addFlashMessage(successMessage);
selectionRetention = new Retention();

save();
for(Retention ret: orphanedRetentions) {
retentionService.delete(ret, ((AuthenticatedUser)session.getUser()).getUserIdentifier());
}
return returnToDraftVersion();
}

public void clearRetentionPopup() {
logger.fine("clearRetentionPopup called");
selectionRetention= new Retention();
setRemoveRetention(false);
PrimeFaces.current().resetInputs("datasetForm:retentionInputs");
}

public void clearSelectionRetention() {
logger.fine("clearSelectionRetention called");
selectionRetention= new Retention();
PrimeFaces.current().resetInputs("datasetForm:retentionInputs");
}

public boolean isCantDownloadDueToRetention() {
if (getSelectedNonDownloadableFiles() != null) {
for (FileMetadata fmd : getSelectedNonDownloadableFiles()) {
if (FileUtil.isActivelyRetended(fmd)) {
return true;
}
}
}
return false;
}

public boolean isCantRequestDueToRetention() {
if (fileDownloadHelper.getFilesForRequestAccess() != null) {
for (DataFile df : fileDownloadHelper.getFilesForRequestAccess()) {
if (FileUtil.isActivelyRetended(df)) {
return true;
}
}
}
return false;
}

private boolean containsOnlyActivelyRetendedFiles(List<FileMetadata> selectedFiles) {
for (FileMetadata fmd : selectedFiles) {
if (!FileUtil.isActivelyRetended(fmd)) {
return false;
}
}
return true;
}

public String getIngestMessage() {
return BundleUtil.getStringFromBundle("file.ingestFailed.message", Arrays.asList(settingsWrapper.getGuidesBaseUrl(), settingsWrapper.getGuidesVersion()));
}
Expand Down
Loading
Loading