Add EAD collection import #6297

solth · 2024-11-07T23:57:14Z

This pull request adds the option to import EAD collections from uploaded XML files. The import creates one (parent) process for the collection and one (child) for each "file" contained in the XML file, thus allowing complete collections ("Bestände) into individual processes from a single file upload.

To be able to use this EAD import, first a ruleset and an Import configuration of type "File Upload" has to be configured with metadata format EAD:

Accordingly, the mapping file used in the import configuration has to map EAD to the Kitodo internal metadata format:

A separate mapping file can be configured to map the collection level of the imported EAD XML file, in the field "Mapping file for parent record":

This pull request adds example mapping files for the parent and child processes (eadParent2kitodo.xsl and ead2kitodo.xsl) as well as a corresponding ruleset (ruleset_ead.xml) that can be used for this purpose and that of cause can also extended when required.

The user can then use this import configuration to upload an EAD XML file containing the EAD collection. The upload dialog also allows to select the EAD level of the element that is to be transformed into the parent process from COLLECTION, CLASS and SERIES and the EAD level of items that are transformed into child processes from FILE and ITEM:

Additionally, a limit for the maximum number or child processes that can be processed interactively in the import mask can now be configured in kitodo_config.properties using the parameter maxNumberOfProcessesForImportMask. (this value defaults to 5). If the number of child elements in the uploaded file exceeds this limit, the user is offered with a choice to relocate the import of all processes to a background task:

If the user has the corresponding permission, the progress of this background import can then be monitored on the "Task manager" page of Kitodo:

Fixes #5984

henning-gerhardt

Some notes from reading the changes.

henning-gerhardt · 2024-11-11T18:18:54Z

Kitodo/src/main/java/org/kitodo/production/forms/createprocess/CreateProcessForm.java

@@ -108,6 +122,10 @@ public CreateProcessForm() {
        this.priorityList = priorityList;
    }

+    public void calculateNumberOfEadElements() throws XMLStreamException {


Missing JavaDoc

Kitodo/src/main/java/org/kitodo/production/forms/createprocess/CatalogImportDialog.java

henning-gerhardt · 2024-11-11T18:22:15Z

Kitodo/src/main/java/org/kitodo/production/model/LazyHitModel.java

    private SearchResult searchResult = null;

    /**
     * Empty default constructor. Sets default catalog and search field, if configured.
     */
-    public LazyHitModel() {
+    public LazyHitModel(CatalogImportDialog dialog) {


Javadoc should be adjusted.

henning-gerhardt · 2024-11-11T18:22:49Z

Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

-        if (Objects.isNull(parentID)) {
-            this.parentTempProcess = null;
-            return;
+    public void checkForParent(String parentID, Ruleset ruleset, int projectID) {


JavaDoc should be adjusted.

Kitodo/src/main/java/org/kitodo/production/thread/ImportEadProcessesThread.java

henning-gerhardt · 2024-11-11T18:26:59Z

Kitodo/src/main/java/org/kitodo/production/forms/createprocess/CatalogImportDialog.java

+    /**
+     * Set searchTerm.
+     *
+     * @param searchTerm as java.lang.String


java.lang.String could be simplified to string or the whole wording should be adjusted.

henning-gerhardt · 2024-11-11T18:28:45Z

Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

+                StringConstants.LEVEL, level);
+    }
+
+    public boolean isMaxNumberOfRecordsExceeded(String xmlString, String tagName) throws XMLStreamException {


Missing JavaDoc.

henning-gerhardt · 2024-11-11T18:32:32Z

Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

@@ -409,8 +412,7 @@ public String getParentID(Document document, String higherLevelIdentifier, Strin
     */
    public TempProcess createTempProcessFromDocument(ImportConfiguration importConfiguration, Document document,
                                                     int templateID, int projectID)
-            throws ProcessGenerationException, IOException, TransformerException, InvalidMetadataValueException,
-            NoSuchMetadataFieldException {
+            throws ProcessGenerationException, IOException, TransformerException {


JavaDoc should be adjusted.

BartChris · 2024-11-13T15:04:27Z

While testing this i first made a dumb configuration error. I associated the wrong XSL-file with the EAD-Mapping-Definition (i associated the pica2ktodo.xsl).
This brought up the following exception message:

This error is produced by the fact that the internalRecord does only contain the XML header and no real content at that point.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

Line 894 in c947be2

    
           resultDocument = XMLUtils.parseXMLString((String) internalRecord.getOriginalData());

It is a dumb mistake on my side, but maybe we can assist by also check if we also check if the importDocument is empty (file has no real content) - as we do later with the resultDocument - and the importer has to check the mapping file configuration.

BartChris · 2024-11-13T16:01:03Z

I tested with the given test ead file. A problem appears, when i try to import the file two times. I then get a Null pointer for the createProcessLocation call of the child processes and they are not created.
The problem is that when processTempProcess is called for a child process (with an already existing title)

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

Line 1227 in c947be2

    
           processTempProcess(tempProcess, rulesetManagement, acquisitionStage, priorityList, null);

the process titles are considered invalid because they do already exist.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

Lines 1342 to 1344 in c947be2

    
           if (!ProcessValidator.isProcessTitleCorrect(processTitle)) { 
        
               throw new ProcessGenerationException(String.format("Unable to create process (invalid process title '%s')", 
        
                       processTitle));

This should throw a ProcessGenerationException, but it seems as if this exeption does not stop the execution in the CreateProcessForm. The execution continues and we have processes without an ID.

Then the Nullpointer is thrown in the FileService, because the file location is based on the missing ID.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/file/FileService.java

Lines 242 to 246 in c947be2

    
           public URI createProcessLocation(Process process) throws IOException, CommandException { 
        
               URI processLocationUri = fileManagementModule.createProcessLocation(process.getId().toString()); 
        
               createProcessFolders(process, processLocationUri); 
        
               return processLocationUri; 
        
           }

When processing the list in the background (because the number of imported processes is greater than the configured limit) the process stops with a ProcessGenerationException.

For the background processing the behaviour should probably be more like in the mass import, where items with duplicate titles are skipped. In the form Kitodo should catch invalid titles and force the user to correct them.

BartChris · 2024-11-14T10:44:56Z

Kitodo/src/main/java/org/kitodo/production/helper/XMLUtils.java

+     * @return number of EAD elements with given level
+     * @throws XMLStreamException when parsing XML string fails
+     */
+    public static int getNumberOfElements(String xmlString, String eadLevel) throws XMLStreamException {


Suggested change

public static int getNumberOfElements(String xmlString, String eadLevel) throws XMLStreamException {

public static int getNumberOfEADElements(String xmlString, String eadLevel) throws XMLStreamException {

This can only be used with EAD elements. Maybe a generic function for getting the number of elements is useful as well, but right now it does not work like that.

@BartChris yes, you are right. This was intended to be used in more general imports, but at the moment only works for the EAD import. I have renamed the method as you suggested.

BartChris · 2024-11-14T10:59:01Z

Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java

+
+        if (parentElements.size() != 1) {
+            throw new ProcessGenerationException(
+                    String.format("EAD XML does not contain exactly one element of parent level '%s'!",


This should maybe be translated as it is displayed in the UI.

BartChris · 2024-11-14T11:31:16Z

While inspecting the Pull request i noticed a general problem:
#6303

solth · 2024-11-15T10:45:19Z

When processing the list in the background (because the number of imported processes is greater than the configured limit) the process stops with a ProcessGenerationException.

For the background processing the behaviour should probably be more like in the mass import, where items with duplicate titles are skipped. In the form Kitodo should catch invalid titles and force the user to correct them.

@BartChris thanks for your analysis. Actually I implemented it this way deliberately. The goal here was to restore the original state - from before the import started - when an error occurs so that the user can first check the uploaded file for problems, fix those problems and then re-try the upload with the updated file. Therefore, all processes that were already created up to the point where the exception is thrown are "cleaned up", e.g. removed.

I do understand, though, that a differnt behavior could be preferable, where importing collections does not fail with individual problematic EAD data records, but instead skips those processes and proceeds with the next "items" from the EAD file. If there is no consense on which option is better, we could ideally introduce a parameter in kitodo_config.properties to control the error handling that way.

henning-gerhardt · 2024-11-15T10:57:38Z

The goal here was to restore the original state - from before the import started - when an error occurs so that the user can first check the uploaded file for problems, fix those problems and then re-try the upload with the updated file. Therefore, all processes that were already created up to the point where the exception is thrown are "cleaned up", e.g. removed.

Maybe nothing for now: but could here help a dry run on importing without a real save on database, index and file system? Deleting processes can even cause additional work in the background.

solth · 2024-11-15T11:47:13Z

Maybe nothing for now: but could here help a dry run on importing without a real save on database, index and file system? Deleting processes can even cause additional work in the background.

I wouldn't do that in this particular case. Uploaded EAD XML files are parsed using a SAX parser in order to be able to handle large XML files without increasing memory requirements. To keep this advantage no lists of complex objects should be retained during the import of an EAD collection (which would grow with XML files containing large collections). Instead each process is completely handled (including saving it to filesystem, index and database) when reaching its corresponding "end event" in the main while loop of ImportEadProcessesThread.

I added a parameter that controls how the system should handle exceptions during EAD import. The default behavior is now as @BartChris described, so that erroneous records are skipped and the import is continued with the following EAD element. In this case, no intermediate processes are removed, so no additional deletion of processes is performed anyway.

BartChris · 2024-11-15T13:27:39Z

I tested with the given test ead file. A problem appears, when i try to import the file two times. I then get a Null pointer for the createProcessLocation call of the child processes and they are not created. The problem is that when processTempProcess is called for a child process (with an already existing title)

I still have the problem, that when using the standard import way (without background task) i get a NullPointer exception, when i try to import the same EAD collection multiple times.

solth · 2024-11-15T14:06:45Z

I tested with the given test ead file. A problem appears, when i try to import the file two times. I then get a Null pointer for the createProcessLocation call of the child processes and they are not created. The problem is that when processTempProcess is called for a child process (with an already existing title)

I still have the problem, that when using the standard import way (without background task) i get a NullPointer exception, when i try to import the same EAD collection multiple times.

This seems to be a problem in the hierarchical import in general that also occurs in the master branch and is not related to the changes in this pull request. There (in the master branch), I just tried to import a process hierarchy from Kalliope (ID DE-2060-BE-13-C35, import depth = 1, import children = true) twice and on the second try got the same NullPointerException you documented above.

@BartChris If this is easy to resolve I can try to include a fix in this pull request, otherwise I would suggest to handle that issue separately.

solth · 2024-11-15T15:00:13Z

I still have the problem, that when using the standard import way (without background task) i get a NullPointer exception, when i try to import the same EAD collection multiple times.

@BartChris I think this will take a little more time to properly fix. I would therefor suggest to deal with this problem separately. I will create a ticket for this problem.

BartChris · 2024-11-15T15:13:29Z

@solth I agree and approve the Pull request. I think the real usage in the archives has to show, wether we need additional adjustments, but it works good for me.

Some additional ideas for the future:

Also allow the background processing for the "normal" mass import
Think about a way to provide a report after a background processed mass import (EAD and normal) to report the result of the processing (Skipped processes? Reason?) in a way which is usable for non-admins

solth · 2024-11-15T15:29:40Z

Some additional ideas for the future:

* Also allow the background processing for the "normal" mass import

* Think about a way to provide a report after a background processed mass import (EAD and normal) to report the result of the processing  (Skipped processes? Reason?) in a way which is usable for non-admins

Yes, these are definitely the next two incremental steps that we should try to achieve and that I already had in mind as well. Thanks a lot for the review!

solth requested review from thomaslow and BartChris November 8, 2024 12:16

henning-gerhardt reviewed Nov 11, 2024

View reviewed changes

BartChris reviewed Nov 14, 2024

View reviewed changes

solth requested a review from BartChris November 15, 2024 11:47

solth added 5 commits November 15, 2024 14:52

Add EAD collection import

8030d71

Implement change requests regarding JavaDoc

2d691c5

Translate error message

685c12b

Rename method 'getNumberOfElements'.

41c5e21

Add parameter to control execption handling during EAD collection import

75dd227

solth force-pushed the ead-import branch from 2612405 to 75dd227 Compare November 15, 2024 14:07

BartChris approved these changes Nov 15, 2024

View reviewed changes

solth merged commit d84f92c into kitodo:master Nov 15, 2024
5 checks passed

solth deleted the ead-import branch November 15, 2024 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add EAD collection import #6297

Add EAD collection import #6297

solth commented Nov 7, 2024 •

edited

Loading

henning-gerhardt left a comment

henning-gerhardt Nov 11, 2024

henning-gerhardt Nov 11, 2024

henning-gerhardt Nov 11, 2024

henning-gerhardt Nov 11, 2024

henning-gerhardt Nov 11, 2024

henning-gerhardt Nov 11, 2024

BartChris commented Nov 13, 2024 •

edited

Loading

BartChris commented Nov 13, 2024 •

edited

Loading

BartChris Nov 14, 2024 •

edited

Loading

solth Nov 14, 2024

BartChris Nov 14, 2024 •

edited

Loading

solth Nov 14, 2024

BartChris commented Nov 14, 2024

solth commented Nov 15, 2024

henning-gerhardt commented Nov 15, 2024

solth commented Nov 15, 2024

BartChris commented Nov 15, 2024

solth commented Nov 15, 2024

solth commented Nov 15, 2024

BartChris commented Nov 15, 2024 •

edited

Loading

solth commented Nov 15, 2024 •

edited

Loading

	public static int getNumberOfElements(String xmlString, String eadLevel) throws XMLStreamException {
	public static int getNumberOfEADElements(String xmlString, String eadLevel) throws XMLStreamException {

Add EAD collection import #6297

Add EAD collection import #6297

Conversation

solth commented Nov 7, 2024 • edited Loading

henning-gerhardt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BartChris commented Nov 13, 2024 • edited Loading

BartChris commented Nov 13, 2024 • edited Loading

BartChris Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BartChris Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BartChris commented Nov 14, 2024

solth commented Nov 15, 2024

henning-gerhardt commented Nov 15, 2024

solth commented Nov 15, 2024

BartChris commented Nov 15, 2024

solth commented Nov 15, 2024

solth commented Nov 15, 2024

BartChris commented Nov 15, 2024 • edited Loading

solth commented Nov 15, 2024 • edited Loading

solth commented Nov 7, 2024 •

edited

Loading

BartChris commented Nov 13, 2024 •

edited

Loading

BartChris commented Nov 13, 2024 •

edited

Loading

BartChris Nov 14, 2024 •

edited

Loading

BartChris Nov 14, 2024 •

edited

Loading

BartChris commented Nov 15, 2024 •

edited

Loading

solth commented Nov 15, 2024 •

edited

Loading