Database

Setup

1. Download the Solr TGZ:

   wget http://archive.apache.org/dist/lucene/solr/6.5.0/solr-6.5.0.tgz
   tar xfz solr-6.5.0.tgz

2. Add .../solr-6.5.0/bin to PATH

3. Clone the Database schemas

   git clone [email protected]:ECP-CANDLE/Database.git

4. Start solr as yourself with these schemas
   Specify the Solr home directory by using the -Dsolr.solr.home setting

   ./bin/solr start -Dsolr.solr.home=.../Database

   All data is stored in .../Database

5. Start inserting data with the examples in Database/examples

Basic operation reference

You can access the database using curl/wget or via pysolr.

Command line tools

• A curl-based insertion is here

• A wget-based query is here

Python

We are using pysolr to access the database from workflows.

Setup

1. Install pip

2. Do pip install requests

3. Download:

wget https://pypi.python.org/packages/47/d6/91dd269b4443c77905ac5f347318435bfeaa2825ce2763d936e0945f29e4/pysolr-3.6.0.tar.gz

1. Untar:

tar xfz pysolr-3.6.0.tar.gz

1. Set PYTHONPATH to point to that directory. You only need pysolr.py, you can delete everything else.

   • A sample insertion is here

   • A sample query is …​

   • Our WIP pysolr abstraction is run_insert.py

The CANDLE DB tool

The tool is called candle_db. It contains a nice Python API for accessing the CANDLE Solr database. It can also be used on the command line.

Insert from command line

./candle_db update run run_id=<run_id> parameters=<parameters> <other key/values...>

Workflow access from Python

Cf. The SwiftExamples sweep.py, which is called from Swift

import candle_db
candle_db.update_run(run_id=run_id,
                     parameters=params_string,
                     experiment_id=sweep_experiment_id)

Sample transcript after workflow execution

What have I run so far?

$ candle_db ls -c experiment
workflow.swift-276
workflow.swift-279

I ran two workflows. The second looks familiar. What is its full metadata?

$ candle_db query experiment experiment_id:workflow.swift-276
results: 1
date_inserted    = 2017-04-19T18:24:37.033Z
benchmark_id     = unknown
experiment_id    = workflow.swift-276
date_modified    = 2017-04-19T18:24:37.033Z
experiment_title = untitled

Ok, what samples did it perform?

$ candle_db ls run experiment_id:workflow.swift-276
results: 4
workflow.swift-276:N1=1,NE=6 = N1=1,NE=6
workflow.swift-276:N1=1,NE=5 = N1=1,NE=5
workflow.swift-276:N1=2,NE=6 = N1=2,NE=6
workflow.swift-276:N1=2,NE=5 = N1=2,NE=5

The N1=1,NE=5 sample looks relevant. What was its metadata?

$ candle_db query run run_id:workflow.swift-276\\:N1=1,NE=5
results: 1
date_inserted = 2017-04-19T18:24:37.292Z
benchmark_id  = unknown
experiment_id = workflow.swift-276
parameters    = [u'N1=1,NE=5']
run_id        = workflow.swift-276:N1=1,NE=5
date_modified = 2017-04-19T18:24:37.292Z