Basic Duplicate Detection

[kshepherd]

References

• Replaces Duplicate Detection ported from DSpace CRIS 7 Angular #1732
• Requires Basic Duplicate Detection in submission and workflow DSpace#9265

The initial work on this feature was supported by TU Berlin. This work was supported by FHNW and ZHAW.

Description

This feature introduces basic Duplicate Detection into DSpace submission and item links, using Solr's ability to search by levenshtein distance.

A new solr field item_signature is indexed for in-progress and archived items, with a normalised version of the title to allow it to be treated as a single term using the ~n operator, where n is a configurable edit distance.

The potential duplicates detected by this search will be displayed to submitters or reviewers as a submission step, for their information (they could then choose to ignore the report if they know this is a new different item, or cancel their submission if there is a legitimate duplicate already present).

If a potential duplicate is in workspace, only the submitter of an item can see this information.

Workflow reviewers are given a small hint indicating how many potential duplicates are detected for a pooled or claimed task, and directing them to the 'Edit' (form) where the step will show details. If the potential duplicate is in workflow, only a reviewer or admin will be able to see this information.

If versioning is used, only the latest version is included in comparison.

Instructions for Reviewers

• Make sure you are using the backend PR Basic Duplicate Detection in submission and workflow DSpace#9265
• Configure submission to include the new duplicate step
• Enable duplication in dspace/config/modules/duplicate-detection.cfg and configure as desired
• Add items with the exact same (or play around with edit distance) titles, and see what shows up in the Duplicate section

Checklist

This checklist provides a reminder of what we are going to look for when reviewing your PR. You need not complete this checklist prior to creating your PR (draft PRs are always welcome). If you are unsure about an item in the checklist, don't hesitate to ask. We're here to help!

• My PR is small in size (e.g. less than 1,000 lines of code, not including comments & specs/tests), or I have provided reasons as to why that's not possible.
• My PR passes ESLint validation using yarn lint
• My PR doesn't introduce circular dependencies (verified via yarn check-circ-deps)
• My PR includes TypeDoc comments for all new (or modified) public methods and classes. It also includes TypeDoc for large or complex private methods.
• My PR passes all specs/tests and includes new/updated specs or tests based on the Code Testing Guide.
• If my PR includes new features or configurations, I've provided basic technical documentation in the PR itself.
• If my PR fixes an issue ticket, I've linked them together.

[MW3000]

Works as intended also using the new search endpoint.

[kshepherd]

@MW3000 thank you for the further review. I have fixed the issues identified.

[github-actions]

Hi @kshepherd,
Conflicts have been detected against the base branch.
Please resolve these conflicts as soon as you can. Thanks!

[atarix83]

Thanks @kshepherd for this PR.

I've reviewed this PR and found out this is not ported as is from implemented on DSpace-CRIS.

I agree to keep it simple than the most comprehensive functionality that is present on DSpace-CRIS, but i've couple of suggestions i'd like to be ported here in this PR:

• in my opinion is not necessary to show the submission step when no duplicates were detected because I don't think it gives any added value to the submission process
  
  We hide mandatory section without section data here on DSpace-CRIS https://github.com/4Science/dspace-angular/blob/e4e919307924a2f912bdcc53cf00fd20fc18e8c0/src/app/submission/objects/submission-objects.effects.ts#L87C25-L88C70
• when a duplicate is detected i'd highlight the message with a warning alert, something like
  
  In this case i'd also change the section status to warning

Moreover in console there is a repetitive error which occurs when the submission is deposited due to the detect duplicate step

[pnbecker]

@atarix83 thank you for the review! Yes, this isn‘t a port anymore. As the port didn‘t got reviewed for several DSpace versions, we reimplemented this to get much smaller PRs. We hoped to get it reviewed this time by reducing the PR to roughly 20% of the original size.

[pnbecker]

@atarix83 we addressed your feedback. Can you please confirm?

[kshepherd]

@atarix83 thanks for your review, I have addressed your feedback as below:

1. Warning alert panel for "potential duplicates" message in submission section, claimed task and pooled task

2. Section data is not assumed to exist in the component template so those errors reading from missing section data will not occur
3. I made the default behaviour of the submission section to hide the duplicate section (at form load and after parsing save response, using very similar approach to the CRIS code you linked) when there are no duplicates to display.

However, if the angular env config submission.duplicateDetection.alwaysShowSection is set to true (default false) then the section will show the "no duplicates" message, in case that fits the user requirements. The message is displayed in an info alert panel:

(ignore that black border, that is my screenshot utility, not the actual web display)

[atarix83]

@pnbecker @kshepherd

i'm sorry i didn't realize that the rest part is fully changed and my thought was that only the Angular interface has been simplified.

also based on what @abollini pointed out in this comment DSpace/RestContract#252 (review) I'd not approve this PR.

There are any changes the old porting PR is resumed?

[kshepherd]

force pushed a large rebase (lots of conflicts with modules now that other submission PRs are pushed and this one has so many commits), and includes changes to use searchBy endpoint again as per last reversion/change in backend PR

[kshepherd]

@tdonohue although code cov looked ok after i made my last changes, i also added new tests for the data service and the object reducer / actions, to keep things in line with general good practice and get coverage up a bit more, hope that works well

[tdonohue]

👍 Thank you @kshepherd ! I retested this today and it works perfectly from the Submission form & the Workflow tasks page. I've also verified that errors no longer appear in DevTools when this feature is disabled. The new specs/tests also look good & bring this up to our CodeCov requirements

Verified feedback has been addressed

[tdonohue]

Flagging as needs documentation until Documentation is added for this new feature to the pre-8.0 docs: https://wiki.lyrasis.org/display/DSDOC8x/Submission+User+Interface

[tdonohue]

Removing "needs documentation" flag as docs now exist in https://wiki.lyrasis.org/pages/viewpage.action?pageId=328958055