Add Whole Tale to Dataverse demo site

[craig-willis]

I was trying to demo the Whole Tale/Dataverse integration today and noticed that the external tools integration is no longer available.

[pdurbin]

@craig-willis thanks for opening this issue. It should be relatively straightforward to follow Whole Tale's excellent documentation (screenshot below) at https://wholetale.readthedocs.io/en/stable/users_guide/integration.html#dataverse-external-tools via http://guides.dataverse.org/en/4.18.1/admin/external-tools.html

[kcondon]

Having trouble running this on demo:
curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file wholetale.json
{"status":"ERROR","message":"Transaction aborted"}

Plus Wholetale command line has a typo, says -upload-file should say --upload-file.

[pdurbin]

{"status":"ERROR","message":"Transaction aborted"}

Bloops. Good catch. This is my fault. I created this issue about this:

• Unable to add dataset level external tool due to non null constraint for contenttype Unable to add dataset level external tool due to non null constraint for contenttype #6459

And then this pull request:

• contenttype can be non-null because dataset tools do not require it contenttype can be null because dataset tools do not require it #6459 #6460

[pdurbin]

At standup today we discussed how now that pull request #6460 has been merged, this issue should be unblocked.

@scolapasta asked me to do one last check but I got a strange result. This is what I did:

• Spin up Dataverse 4.16.
• Upgrade to Dataverse 4.17 (I followed the release notes).
• Try to add the Whole Tale tool.

At expected something like {"status":"ERROR","message":"Transaction aborted"} as described above. Instead the tool was added. Here's the output I saw:

curl -X POST -H 'Content-type: application/json' --upload-file wholetale.json http://localhost:8080/api/admin/externalTools

{"status":"OK","data":{"id":20,"displayName":"Whole Tale","description":"Analyze in Whole Tale","type":"explore","scope":"dataset","toolUrl":"https://data.wholetale.org/api/v1/integration/dataverse","toolParameters":"{\"queryParameters\":[{\"datasetPid\":\"{datasetPid}\"},{\"siteUrl\":\"{siteUrl}\"},{\"key\":\"{apiToken}\"}]}"}}

The question I was going to try to answer was "If you run the SQL from pull request #6460 manually and then let Flyway run it, are there any errors or problems deploying?" Instead, I have more questions, especially, "Why was I able to add the Whole Tale tool on 4.17 without the SQL script?" and "Do we need to update the release notes for 4.17 or not?" As I mentioned at standup, I already added a stub to the bottom of the 4.17 release notes about #6459 so I'll tag @djbrooke to make sure we talk more about this.

[pdurbin]

When I look at my installation of Dataverse that I upgraded from Dataverse 4.16 to 4.17 the "externaltool" table already allows contenttype to be null:

dvndb=# \d+ externaltool
                                                         Table "public.externaltool"
     Column     |          Type          |                         Modifiers                         | Storage  | Stats target | Description 
----------------+------------------------+-----------------------------------------------------------+----------+--------------+-------------
 id             | integer                | not null default nextval('externaltool_id_seq'::regclass) | plain    |              | 
 contenttype    | text                   |                                                           | extended |              | 
 description    | text                   |                                                           | extended |              | 
 displayname    | character varying(255) | not null                                                  | extended |              | 
 toolparameters | character varying(255) | not null                                                  | extended |              | 
 toolurl        | character varying(255) | not null                                                  | extended |              | 
 type           | character varying(255) | not null                                                  | extended |              | 
 scope          | character varying(255) | not null                                                  | extended |              | 
Indexes:
    "externaltool_pkey" PRIMARY KEY, btree (id)

Again, this is without me running the SQL script in pull request #6460. So I'm not sure if it's really necessary. Very confusing.

[pdurbin]

@scolapasta can you please take a look at my last two comments? Thanks!

[scolapasta]

@pdurbin I'm not sure why it would be not "not null" in your 4.16 -> 4.17 instance. That said, that would work for the use case of checking that the flyaway script works when the not null has been removed manually.

[pdurbin]

I just clicked "Explore" from a dataset in https://demo.dataverse.org and I was able to successfully run some code in Whole Tale. Below I'll put some screenshots.

$ cd ~/Downloads/5dfa85577bf5ca3bf54ac41f
$ find .
.
./tagmanifest-md5.txt
./bagit.txt
./bag-info.txt
./tagmanifest-sha256.txt
./README.md
./run-local.sh
./manifest-md5.txt
./data
./data/LICENSE
./data/workspace
./data/workspace/code
./data/workspace/code/language-count.sh
./data/workspace/run.sh
./data/workspace/results
./data/workspace/results/language-count.tsv
./data/workspace/results/language-count.tab
./data/workspace/data
./data/workspace/data/2019-02-25.tab
./data/workspace/data/2019-02-25.tsv
./manifest-sha256.txt
./metadata
./metadata/manifest.json
./metadata/environment.json
./fetch.txt

[craig-willis]

Thanks all. It will be great to be able to demonstrate this integration using the demo Dataverse instance.

[pdurbin]

The question I was going to try to answer was "If you run the SQL from pull request #6460 manually and then let Flyway run it, are there any errors or problems deploying?"

I just did some more testing and the answer seems to be "no". That is to say, while running 4.17 I ran the SQL upgrade script. Then I upgraded to the latest in develop dc103c4. No problems deploying. I think we're good.

@djbrooke I went ahead and updated the live 4.17 release note. I didn't update doc/release-notes/4.17-release-notes.md but I'd be happy to make a pull request.