How to identify error source?

[bondjimbond]

I've been working on my Twig template and CSV template. Metadata templating is coming out just fine, but when I reach the Ingest step, I get the following error:

messageRows with issue(s) that could not be translated to Objects: 1,2

I've only got the one row, so I know which row the error is in... but how can I identify what the actual problem is? I'm not doing that much different from when I was testing using the example files, just adding more metadata fields.

OBJ maps to the "filename" column.

I would of course like help identifying the source of the error in my CSV, but I would like to see IMI provide a bit more detail. I see that it prints an array to the Drupal log; perhaps the error could refer more precisely to the exact position in the array where it's hitting an error and log it?

arca_imi.zip
sample.mp4.zip

[bondjimbond]

Nothing useful in the Apache log either:

[Fri Apr 03 13:11:20.743897 2020] [:error] [pid 2391] [client 10.0.2.2:60744] Running Multi Importer getIngestInfo, referer: http://localhost:8000/multi_importer

[DiegoPino]

@bondjimbond let's do a walk through together. IMI produces an entry inside your Drupal Log (see your admin/reports) that has all the settings you passed during an IMI ingest setup (its like a full ARRAY dump). Can you share that with me? That will allow me to retrace your steps one by one

There are two types of errors:

PREPROCESS: (the ones you are seeing)

Most of the time those errors are either:

• ZIP is wrong, can not be read/opened.
• Wrong parent collection/PID settings/mappings (Either your data is wrong or your settings in "Object Properties mapping". If Islandora can not assign a PID, the parent does not exist, or the row number used to reference another row to define membership/relationship does not exist or is wrong

Actual ingest

• MODS can not be generated because the TWIG template is too permissive, does not check encoding, etc or is simply wrong.
• Out of space/remote URL for an OBJ could not downloaded, Derivatives die, wrong CMODEL, Missing XSLT mapping (XML form thing...)

All these try to generate verbose messages that go into your Drupal Log.

All of that, is generated in this file (e.g a line there)

islandora_multi_importer/includes/islandora_multi_batch.inc

Line 407 in 4c48f58

'error' => array_keys($invalid),

and you will see there are a lot of messages going out. We can make them better for sure.

Will wait for that Drupal Log output and test your data, i don't see anything obvious wrong on your CSV so better that way.

[bondjimbond]

Absolutely, thanks for helping. I'm using the same sample.mp4.zip that I had used successfully with the stock CSV and Twig templates, so I don't think there's a problem with the object/zip file itself.

I had initially set the incorrect parent collection, but fixed that, and am still having these problems.

Parameters for Set 6
Array
(
    [type] => ZIP
    [source_data] => 25
    [cmodel_source_field_index] => 3
    [cmodel_mapping] => Array
        (
            [islandora:sp_videoCModel] => Array
                (
                    [dsid] => Array
                        (
                            [rows] => Array
                                (
                                    [DC] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => xlst|0
                                                )

                                        )

                                    [MODS] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => 
                                                )

                                        )

                                    [OBJ] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => column|1
                                                )

                                        )

                                    [TN] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => derivative|0
                                                )

                                        )

                                    [OGG] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => column|0
                                                )

                                        )

                                    [MKV] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => derivative|0
                                                )

                                        )

                                    [TECHMD_FITS] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => column|0
                                                )

                                        )

                                    [MP4] => Array
                                        (
                                            [method] => Array
                                                (
                                                    [data] => derivative|0
                                                )

                                        )

                                )

                        )

                )

        )

    [object_maping] => Array
        (
            [pidmap_row] => Array
                (
                    [pidmap] => 0
                    [pidtype] => 1
                )

            [parentmap_row] => Array
                (
                    [parentmap] => 0
                    [parenttype] => 1
                )

            [labelmap_row] => Array
                (
                    [labelmap] => 5
                )

            [sequencemap_row] => Array
                (
                    [sequencemap] => 2
                )

            [dsmap_row] => Array
                (
                    [dsremote] => ZIP
                )

            [relsext_row] => Array
                (
                    [relsext_mode] => NEW
                )

        )

    [computed_cmodels] => Array
        (
            [islandora:sp_videoCModel] => Array
                (
                    [DC] => Array
                        (
                            [id] => DC
                            [mime] => Array
                                (
                                    [0] => application/xml
                                )

                            [optional] => 
                        )

                    [RELS-EXT] => Array
                        (
                            [id] => RELS-EXT
                            [mime] => Array
                                (
                                    [0] => application/rdf+xml
                                )

                            [optional] => 1
                        )

                    [RELS-INT] => Array
                        (
                            [id] => RELS-INT
                            [mime] => Array
                                (
                                    [0] => application/rdf+xml
                                )

                            [optional] => 1
                        )

                    [MODS] => Array
                        (
                            [id] => MODS
                            [mime] => Array
                                (
                                    [0] => application/xml
                                )

                            [optional] => 1
                        )

                    [OBJ] => Array
                        (
                            [id] => OBJ
                            [mime] => Array
                                (
                                    [0] => video/mp4
                                    [1] => video/quicktime
                                    [2] => video/m4v
                                    [3] => video/x-matroska
                                    [4] => video/ogg
                                    [5] => video/x-msvideo
                                    [6] => video/avi
                                )

                            [optional] => 1
                        )

                    [TECHMD] => Array
                        (
                            [id] => TECHMD
                            [mime] => Array
                                (
                                    [0] => application/xml
                                )

                            [optional] => 
                        )

                    [TN] => Array
                        (
                            [id] => TN
                            [mime] => Array
                                (
                                    [0] => image/jpeg
                                    [1] => image/png
                                    [2] => image/gif
                                )

                            [optional] => 
                        )

                    [OGG] => Array
                        (
                            [id] => OGG
                            [mime] => Array
                                (
                                    [0] => video/ogg
                                )

                            [optional] => 
                        )

                    [MKV] => Array
                        (
                            [id] => MKV
                            [mime] => Array
                                (
                                    [0] => video/x-matroska
                                )

                            [optional] => 
                        )

                    [TECHMD_FITS] => Array
                        (
                            [id] => TECHMD_FITS
                            [mime] => Array
                                (
                                    [0] => application/xml
                                )

                            [optional] => 
                        )

                    [MP4] => Array
                        (
                            [id] => MP4
                            [mime] => Array
                                (
                                    [0] => video/mp4
                                )

                            [optional] => 
                        )

                )

        )

    [source_binaries] => 26
    [action] => ingest
)

[DiegoPino]

@bondjimbond hi,np,

ok, first thing (not related to your error), but i don't see your Twig Template being used as source for the MODS datastream, its not mapped in those settings.
Twig templates can generate any Datastream so you have to tell IMI, for each CMODEL which template is going to produce which XML/TEXT datastream (i was even tempted to allow RELS-EXT but then i had to acknowledge giving online support would kill me!). Right now your MODS is unmapped. FYI, you could let it that way.. but it kinda defeats the purpose

OK, i used your CSV, added your template to the IMI template list

Then went step by step mapping (also remember please to fix the VIDEO CMODEL XML, its like totally wrong, let me know if you need help with it... )

And i got this

Which means your settings are off.

Other than the missing Template mapping for MODS i see this one

[parentmap_row] => Array
                (
                    [parentmap] => 0
                    [parenttype] => 1
                )

Which for your CSV should be

  [parentmap_row] => Array
                (
                    [parentmap] => 4
                    [parenttype] => 1
                )

4 there means which column index (from 0 to) and 4 is the numeric index for parent and your 0 is using the key column, which is just a 1, and you have no 1 object, nor can you make that object be parent of the header row! This settings is the "parent object" setting in the "Object properties mapping" tab.

Give it a shot again. I feel a short tutorial (Zoom or so) next week could help you. You are pretty much there, just probably just moving too fast between settings, and i know, there are a lot of them

[DiegoPino]

@McFateM is a pretty veteran (in terms of knowledge, he is pretty young!) user, maybe he can give you a hand and i also know ISLE has done some work around documenting and hacking IMI. Maybe he can help here too?

[bondjimbond]

Ah, right, I think it was the parent thing. Somehow I was misremembering how I originally did it and thought that both of the checkboxes meant that those settings would be ignored (instead of just the first one).

And good to know that I have to select the Twig template again -- I had thought that by selecting it on the Templating section, that would be how it's selected for the whole thing.

What are the odds of some of these configurations being preselected, or at least prepopulating the selections based on input from earlier on in the process?

[SummittDweller]

Sorry I'm so late to this conversation... I'm not as young as Diego suggests. 8^) And very busy.

I have a hook and some IMI customization that pre-populates EVERYTHING. It's something I wrote just before IslandoraCon last Fall. I think there's some mention, maybe even some "documentation" in my blog from back then. If you're interested I can take a look back sometime this afternoon and post links here to anything I find?

[bondjimbond]

@SummittDweller Thanks, that would be helpful. It'd be nice to have some prepopulating done, as it's easy to skip over small but important settings by accident.

[DiegoPino]

@bondjimbond the first twig template selection is more like a playground/testing, not really needed, specially given the fact, that you could eventually want to have different twig templates for different CMODELS or even force your own DC template overriding what Islandora does. Means you can end with many twig templates in use at the same time.

That preset thing has been in the wish list for quite some time but i always stumble in the same issue, what happens if the CSV changes? Probable @SummittDweller solved that one! (cool new handle by the way!)

[SummittDweller]

@DiegoPino Err, no I didn't solve that problem. However, in my case, the CSV layout is ALWAYS the SAME, no matter what CModel. I did remember correctly, there is some "documentation", but it's paper thin. @bondjimbond You can find it at https://dlad.summittdweller.com/en/posts/049-granular-isle-customization-implementing-imi-hooks/, and specifically in the last link (a gist) in that post.

Have a look if you like, and let me know if you're interested in the same, I might be able to make a little time to assist with implementation.

[bondjimbond]

Thanks, I'll look at that when I have the time!

I recorded a brief screencast on using IMI for my members; might be useful to others if they ask for help figuring this out (they can learn from my stumbling): https://www.youtube.com/watch?v=LhgK5-doWJE