Merge pull request #104 from JeffersonLab/dev0.9

v0.9

.github/workflows/python-tests.yml

@@ -8,12 +8,12 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies

.gitignore

@@ -122,3 +122,7 @@ cpp/\.idea/workspace\.xml
 rcdb_prereqs_version.xml
 
 .idea/
+
+.fleet/
+
+sql/schema.mwb.bak

doc/.gitignore

@@ -0,0 +1,4 @@
+node_modules/
+.cache/
+public
+.idea/

doc/Database-Installation.md

@@ -0,0 +1,57 @@
+## Prerequisites  
+> Evironment variables have to be set.  
+    TL;DR; run `environment.*` script located in the RCDB root dir([more details](Installation)).  
+    `RCDB_CONNECTION` must also be set (see below). 
+
+Lets assume that one wants to install RCDB to the following database:
+
+* database location: `localhost`
+* database name: `rcdb`
+* database user name: `rcdb`
+* database user password: `password` 
+
+then the connection string is
+
+```
+mysql://rcdb:password@localhost/rcdb
+```
+
+and ```RCDB_CONNECTION``` must be: 
+
+```bash
+export RCDB_CONNECTION='mysql://rcdb:password@localhost/rcdb'
+```
+
+## Create MySQL database  
+(RCDB is tested for MySQL and MariaDB. Also RCDB should work with other MySQL forks while it hasn't been tested)
+
+1. Create DB(aka schema) and a user with privileges:
+
+    ```bash    
+    mysql -u root -p
+    ```
+
+    > note, that root privileges are required to run ```mysql - u root ...``` for MariaDB by default. Run something like `sudo mysql -u root -p` for MariaDB
+
+    ```sql
+    CREATE SCHEMA rcdb;
+    CREATE USER 'rcdb'@'localhost' IDENTIFIED BY 'password';
+    GRANT ALL PRIVILEGES ON rcdb.* TO 'rcdb'@'localhost';
+    ```
+
+## Create or update DB structure
+
+Both creating schema for the first time for a fresh database or updating the existing schema is done with
+[Alembic](https://pypi.python.org/pypi/alembic)
+
+Just run:
+
+```bash
+cd $RCDB_HOME
+./alembic_rcdb upgrade head
+```
+
+> Since there where problems with installing alembic on some machines in counting house RCDB ships a copy of it within itself; `alembic_rcdb` command runs this embedded alembic version. 
+
+
+

doc/Deployment.md

@@ -0,0 +1,106 @@
+Deploying RCDB MySQL database on local machine. 
+
+### 0. Clone RCDB repo 
+
+Here is a very brief instruction for it (for bash shell):
+
+```bash
+git clone https://github.com/JeffersonLab/rcdb.git
+cd rcdb
+source environment.bash
+
+# now to check that everything works, test connect to HallD RCDB
+export RCDB_CONNECTION=mysql://[email protected]/rcdb
+rcnd
+```
+
+rcnd should answer something like:
+
+```
+Runs total: 10462
+Last run  : 42387
+Condition types total: 53
+Conditions: 
+<list of conditions>
+```
+
+More information is in [Installation chapter](https://github.com/JeffersonLab/rcdb/wiki/Installation)
+
+
+### 1. Create MySQL database
+
+```bash
+mysql -u root -p
+```
+
+```mysql
+CREATE SCHEMA rcdb;
+CREATE USER 'rcdb'@'localhost';
+GRANT ALL PRIVILEGES ON rcdb.* TO 'rcdb'@'localhost';
+```
+
+### 2. Create DB structure
+And fill in a minimal set of common conditions. 
+
+```bash 
+python $RCDB_HOME/python/create_empty_db.py -c mysql://rcdb@localhost/rcdb --i-am-sure --add-def-con
+```
+
+The right answer should be:
+```
+creating database:
+database created
+default conditions filled
+```
+
+### 3. Testing the installation
+
+Now check rcdn 
+
+```
+export RCDB_CONNECTION=mysql://rcdb@localhost/rcdb
+rcnd
+```
+
+The output should be like:
+
+```
+Runs total: 0
+Condition types total: 13
+Conditions: 
+<list of conditions>
+```
+
+Lets create a first run (run 1) and fill event_count=12345 for run 1
+
+```bash
+rcnd --new-run 1
+rcnd --write 12345 --replace 1 event_count
+
+#lets check conditions for run 1
+rcnd 1
+```
+
+Output should be like:
+
+```
+event_count = 12345
+```
+More about rcnd command and command line tools one could find in 
+[CLI basics chapter](https://github.com/JeffersonLab/rcdb/wiki/rcnd).
+
+### 4. Test run the web site
+
+```bash
+python $RCDB_HOME/start_www.py
+```
+
+The output should be like:
+
+```
+* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
+* Restarting with reloader
+```
+
+Now follow to http://127.0.0.1:5000/ in the browser to see the site. 
+

doc/Development.md

@@ -0,0 +1,33 @@
+# 1. Establishing integration tests
+
+1. Create MySQL database for the test purposes
+
+    ```bash
+    mysql -u root -p
+    ```
+
+    ```mysql
+    CREATE SCHEMA test_rcdb;
+    CREATE USER 'test_rcdb'@'localhost' IDENTIFIED BY 'test_rcdb';
+    GRANT ALL PRIVILEGES ON test_rcdb.* TO 'test_rcdb'@'localhost';
+     ```
+
+2. Set RCDB_MYSQL_TEST_CONNECTION environment variable
+
+    ```bash
+    export RCDB_MYSQL_TEST_CONNECTION="mysql://test_rcdb@localhost/test_rcdb"
+
+    ```
+
+3. run create_test_database.py
+
+    ```bash
+    python $RCDB_HOME/python/tests/create_test_database.py $RCDB_MYSQL_TEST_CONNECTION
+    ```
+
+
+4. Run ```test_all_rcdb```
+
+
+
+

doc/README.md

@@ -0,0 +1,79 @@
+# RCDB
+
+Run Configuration/Conditions Database (RCDB) stores run related information and conditions. It uses MySQL or SQLite databases to store information about runs and provides interfaces to search runs, manage data, automatic CODA integration, etc.
+
+One can consider two main aspects of what conceptually RCDB is designed for: 
+
+1. Add data to database   
+2. Provide convenient way to read, introspect and analyze DB stored data providing interfaces:
+   - Web site
+   - Command line interface (CLI)
+   - Python API
+   - C++ API
+   - Possibly JAVA API
+
+## Concepts 
+
+[daq_concepts](daq/daq_concepts.md ':include')
+
+Software wise:
+
+   - Can work with multiple databases (MySQL, SQLite, possibly others)
+   - Data queries
+   - Administration tools
+   - DAQ module tools
+   - Introspection tools
+
+
+## Demo:
+
+One can visit HallD RCDB Web site as demo:
+https://halldweb.jlab.org/rcdb/
+
+Daily updated SQLite database is available here:
+https://halldweb.jlab.org/dist/rcdb.sqlite
+
+
+## Conditions
+
+Run conditions is the way to store information related to a run (which is identified by run_number everywhere).
+From a simplistic point of view, run conditions are presented in RCDB as **name**-**value** pairs attached to a run number. For example, **event_count** = **1663** for run **100**.
+
+One of the major use cases of RCDB is searching for runs matching required conditions. It is done using simple, python-if-like queries. The result of ```event_count > 100000``` is all runs, that, obviously, have **event_count** more than **100000**
+
+Lets see how API-s would look for the examples above.
+
+Python:
+```python
+import rcdb
+
+# Open SQLite database connection
+db = rcdb.RCDBProvider("sqlite:///path.to.file.db")
+
+# Read value for run 100
+event_count = db.get_condition(100, "event_count").value
+
+# Select runs 
+result = db.select_runs("event_count > 100000")
+for run in result:
+   print run.number   # Or do whatever you want with runs
+```
+
+CLI:
+
+```bash
+export RCDB_CONNECTION=mysql://rcdb@localhost/rcdb
+rcnd --help                            # Gives you self descriptive help
+rcnd 1000 event_count                  # See exact value of 'event_count' for run 1000
+rcnd --write 1663 100 event_count      # Write condition value to run 100
+rcnd --search "event_count > 500"      # Select runs 
+
+```
+
+
+What RCDB conditions are not designed for? - They are not designed for large data sets that change rarely (value is the same for many runs).
+That is because each condition value is independently saved (and attached) for each run.
+
+In the case of bulk data, it is better to save it using other RCDB options. RCDB provides the files saving mechanism as example.
+
+[//]: # (<img src="demo.png" style="max-height: 300px"/>)