Related object block shows incorrect objects

[patdunlavey]

From @petermacdonald:
I have set the IW block to show up in the right side-panel (see image below), but when I'm viewing the whole collection it is displaying every item in the collection -- most of which are not submissions at all.

[patdunlavey]

Additonal comments from email thread:
Pat:
Hi Peter,

The block is filtered by rels-ext relation name, so my off-the-top-of-my-head guess is that you used the same relation name for your ingested submissions as the collection content model uses for its children. If true, and if changing the relation name is not possible/practical, and if you are using a consistent namespace for your parent objects, you could use that in the block visibility section.

Peter:
Pat:

Thanks for your suggestion, but...

the RELS-EXT datastream in the ingested submission Fedora objects all have an "isAnnotationOf" relation, as the following shows.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:fedora="info:fedora/fedora-system:def/relations-external#" xmlns:fedora-model="info:fedora/fedora-system:def/model#" xmlns:islandora="http://islandora.ca/ontology/relsext#">
<rdf:Description rdf:about="info:fedora/yhmKerEph:453">
<fedora-model:hasModel rdf:resource="info:fedora/islandora:sp_example_text">/fedora-model:hasModel
<fedora:isAnnotationOf rdf:resource="info:fedora/yhmKerEph:23">/fedora:isAnnotationOf
/rdf:Description
/rdf:RDF

but none of these Fedora objects show up in the IW block. Instead, the IW block shows only ones with an "isMemberOf" relation:

<rdf:Description rdf:about="info:fedora/yhmKerEph:25">
<islandora:isPageOf rdf:resource="info:fedora/yhmKerEph:23">/islandora:isPageOf
islandora:isSequenceNumber2/islandora:isSequenceNumber
islandora:isPageNumber2/islandora:isPageNumber
islandora:isSection1/islandora:isSection
<fedora:isMemberOf rdf:resource="info:fedora/yhmKerEph:23">/fedora:isMemberOf
<fedora-model:hasModel rdf:resource="info:fedora/islandora:pageCModel">/fedora-model:hasModel
/rdf:Description

And these don't have any "isAnnotationOf" relations at all.

It looks to me like the code supporting the block is not picking up the "isAnnotationOf" setting in my webform.

Pat:
Hmmm... What happens if you re-save the webform's Islandora Settings (the page where this predicate is defined)? And if the answer is nothing, what happens if you save with a different relationship, followed by saving again with the original one?

Peter:
I changed the setting "Relationship to current object" from "Is Annotation Of" to "Is Derivation Of" and got the message (which looks normal):

Islandora configuration saved.
5 objects had their relations updated from "isAnnotationOf" to "isDerivationOf"
5 objects had their relations updated from "isAnnotationOf" to "isDerivationOf"
5 objects had their relations updated from "isAnnotationOf" to "isDerivationOf"
5 objects had their relations updated from "isAnnotationOf" to "isDerivationOf"
5 objects had their relations updated from "isAnnotationOf" to "isDerivationOf"

I then created a new submission with this webform, ingested it, and the new Fedora object has the correct relationship statement:

<fedora:isDerivationOf rdf:resource="info:fedora/yhmKerEph:23">/fedora:isDerivationOf

However, when looking at the original object, the block shows images for pages 1 and 2 of that object (neither of which has "isDerivationOf" in the RELS-EXT) -- and the submission object is not displayed (see screenshopt below).

Test 2: I changed it back to "Is Annotation Of", submitted a new webform, but the submission objects still don't appear in the IW block when viewing the object -- only ones with "isMemberOf" do.

[DiegoPino]

Hi @petermacdonald and @patdunlavey, can you confirm the delta for the block is correct? Did you clear your drupal cache after modifying the relations?
Reading the code, i see the block "delta" is the actual rdf relation used to fetch the objects. So, if the block was generated initially using a "wrong" delta, then even when you change the default relation used by the ingest webform, the already in place block could still be using the initially configured one.

Basically, what is the link (mouse over the iw block you are using) when you go to admin/structure/block?

Hope i'm not adding confusion to the topic, but since hook_block_info() defines the deltas initially(a.k.a relations), i assume that even when those are changed on the fly, the already configured ones could still live in the DB or cache using a wrong delta and as a result display wrong relations.

[patdunlavey]

@petermacdonald and @DiegoPino - Peter has stumbled into what I believe is a critical flaw in the ingested objects rels-ext logic that can wreck all of your repository's collections in one false-move! So, first, if this is on your production repository, I would back it up now. Though it may be too late!

You can set ingested objects to use the "Is Member Of Collection" relation to the target object. That's not a problem yet. The catastrophic problem occurs (may occur - I haven't actually tested it) if you then modify your webform ingest settings to use a different relation predicate. In that case, all objects in the repository that are members of a collection will have that relation replaced by the new one - thus wrecking all of your collections!

As I say, I haven't tested this, but I think it is true. And it would explain the behavior that Peter is seeing.

@DiegoPino Can you think of a way to prevent this from happening? Can we add some flag to the rels-ext record to identify it as generated by this particular ingest configuration, and then when the ingest configuration is changed, only change those rels-ext records associated with it?

[patdunlavey]

@DiegoPino has proposed a solution and we have given him the go-ahead to work on it. So I'm assigning this task to him.

[patdunlavey]

@petermacdonald: Diego thinks he can implement a method for fixing member of collection relationships that have been lost. Probably best if you wait to hear back from us before attempting to change any more webform ingest relationship predicate settings.

Thanks, Pat

[McFateM]

I have a fairly robust and flexible set of Tuque-based Drush scripts that I can run against a repository to effect things like RELS-EXT changes too. If they might be useful here just let me know.

-Mark

From: Pat Dunlavey <[email protected]mailto:[email protected]>
Reply-To: commonmedia/islandora_webform <[email protected]mailto:[email protected]>
Date: Thursday, July 2, 2015 at 11:31 AM
To: commonmedia/islandora_webform <[email protected]mailto:[email protected]>
Subject: Re: [islandora_webform] Related object block shows incorrect objects (#10)

@petermacdonaldhttps://github.com/petermacdonald: Diego thinks he can implement a method for fixing member of collection relationships that have been lost. Probably best if you wait to hear back from us before attempting to change any more webform ingest relationship predicate settings.

Thanks, Pat

Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-118086609.

[DiegoPino]

Great @McFateM, i appreciate your offer, i will give a local try first using my tools and will also contact you tomorrow to get some info on how we can identify the affected objects so we can roll everything back and fix this. Thanks!

[McFateM]

I'm off work (holiday) tomorrow (Friday, July 3), but if you still need or want to talk on Monday please let me know. Take care and have a great weekend.

-Mark

From: Diego Pino Navarro <[email protected]mailto:[email protected]>
Reply-To: commonmedia/islandora_webform <[email protected]mailto:[email protected]>
Date: Thursday, July 2, 2015 at 2:24 PM
To: commonmedia/islandora_webform <[email protected]mailto:[email protected]>
Cc: Mark McFate <[email protected]mailto:[email protected]>
Subject: Re: [islandora_webform] Related object block shows incorrect objects (#10)

Great @McFateMhttps://github.com/McFateM, i appreciate your offer, i will give a local try first using my tools and will also contact you tomorrow to get some info on how we can identify the affected objects so we can roll everything back and fix this. Thanks!

Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-118140085.

[DiegoPino]

Hi @McFateM, @petermacdonald and @patdunlavey. I just committed a new branch so we can test some new functionality and ideas.
Basically

• i removed the block->existing webform association dependency. So now you get blocks for every defined predicate in this module. This also allows you to navigate to objects that are no longer related to a particular form anymore. Block functionality is similar and don't discriminative (if block $delta is A, the you get all via A related objects, webform or not webform ingested). but you can also set to show only IW ingested ones. Nice!
• Also, on object ingest, i added some IW predicates to keep track of which webform generated the object. This predicates can be indexed in Solr, allowing to make some nice views or extra interfaces, but more important, they allow us to make predicate changes only on those tagged objects, not messing around anymore on normal Repo objects.
• At last, we give you some "power" when making a predicate change in an existing form. If you already have objects created by a particular webform in place and decide to make a predicate change, you get a few options to deal with them. You can migrate to a new predicate, replacing the old one, do nothing(my favourite) or move every tagged object (related to this webform) independently of which predicates they already have, replacing only "one" with the "new one".
  It's almost perfect until to use other modules to make relations. So use with care.

It's a lot of code, some invisible fixes but proactive (means if we start from 0 everything flows). So we need now a retroactive fix for current ingested objects. There are many different ways we could handle this, from deleting -> re ingesting to batch identifying and updating. And i can help on this.

I did many, many border case tests, but i'm sure you will find a much more border case that i have not resolved, so i would love you could clone this branch and start playing around.

[patdunlavey]

Wow Diego, you did a lot of work! (I think you measure time differently down there!)

We should probably have another skype call to discuss strategies for retroactive fix.

It will take some time to properly understand and test all this work. But on first run-through, I had a couple questions related to this screenshot:

• Does 'Only those objects that are using the former predicate "[predicate]"' mean that all objects with that predicate will be modified, or only those objects ingested under this webform with this predicate? Should we have separate options for those two possible preferences?
• The messages area indicates that the other settings have been saved, which seems a little strange on a confirmation page. Do you know if at this stage of the form submission those settings have been saved yet? If not, I think we should just change when the message appears. Otherwise, it may not be worth the trouble to modify the form submission process to save the other settings after the confirmation page.

[DiegoPino]

Hi Pat, well, let's forget about timing this time =)

About your questions:

• The whole migration confirmation form works (every option there) only on objects ingested (using this version of the module) using this webform (node id) association, so we will never touch objects that are not marked using IW predicates. More info on this on the Sparql queries. We could inform this also in the form?
• Maybe the word "confirmation" is not the most exact definition. It's not a "save confirmation", but a "migration confirmation". Settings are being saved after submitting the first form, as the status messages correctly inform.
  I separated saving from migration. This means, you could just close this page (same as "leave existing objects untouched") and the previous settings would stick. I also did so, because the previous form is defined in two different modules and i did not wanted to modify the main one at that extend. (also not simple to do when you have multiple submit handlers in place). You see also i omitted a back button and the previous form submit button still says "save settings".

On Jul 8, 2015, at 7:58 AM, Pat Dunlavey [email protected] wrote:

Wow Diego, you did a lot of work! (I think you measure time differently down there!)

We should probably have another skype call to discuss strategies for retroactive fix.

It will take some time to properly understand and test all this work. But on first run-through, I had a couple questions related to this screenshot:

Does 'Only those objects that are using the former predicate ""' mean that all objects with that predicate will be modified, or only those objects ingested under this webform with this predicate? Should we have separate options for those two possible preferences?
The messages area indicates that the other settings have been saved, which seems a little strange on a confirmation page. Do you know if at this stage of the form submission those settings have been saved yet? If not, I think we should just change when the message appears. Otherwise, it may not be worth the trouble to modify the form submission process to save the other settings after the confirmation page.
—
Reply to this email directly or view it on GitHub.

[patdunlavey]

Thanks Diego, that makes sense. I'll think about improving the wording on that page.

I'm going to be occupied with other things today. Will you have any time tomorrow (Friday) for a skype call about retroactive approaches?

[patdunlavey]

@petermacdonald does the original problem described in this issue still exists for you?