Implement loading to staging/live virtuoso

[Bjwebb]

No description provided.

[Bjwebb]

The original plan was to using pyodbc for this, and I even updated the Virtuoso patch to support v3 Bjwebb/pyodbc@459382d, and I got as far as acquiring a cursor - NRGI/resource-projects-etl@8db4a99 - however, using that cursor caused errors.

Instead, I'm now trying to add sparql using virtuoso's sparql-graph-crud-auth web endpoint. I also had trouble using "requests" for this (see 96fa2fc#diff-d6efc1c3c41753d42e33120dc96d54bcR73 for the erroring code). So for now I'm using subprocess to talk to curl.

This now successfully loads into virtuoso.

Some of the remaining tasks:

• Add the staging/live graphs to different graph groups
  • [10 Sep] Check whether we can avoid using shell=True and/or composing our own virtuoso sql. Particularly, I don't know if errors will be caught in our current piping setup. - split out to Construct virtuoso SQL more securely NRGI/resource-projects-etl#44
• Better variables for virtuoso configuration - Better variables for virtuoso configuration NRGI/resource-projects-etl#45
• Tests

[Bjwebb]

I've looked into graph groups, and as far as I can tell this involves running SQL rather than SPARQL commands. There is no nice REST endpoint for SQL that I can see. Thus I think we have 3 options:

1. Get pyodbc working - not sure how much work this is, but my exprience so far suggests it won't be straightforward.
2. Do the cookie based auth to get at http://localhost:8890/conductor/isql_main.vspx over HTTP - this is inelegant and brittle
3. Call virtuoso's isql command using subprocess. This requires running Cove in an environment that has access to virtuoso's isql - e.g. by building it in a container inherited from the virtuoso one.
   I'm investigating option (3) for now.

[timgdavies]

+1 to Option 3.