Merge branch 'develop'

CHANGELOG.md

@@ -1,3 +1,33 @@
+# Version 9.0.0
+*2024-03-26*
+
+See the [Version 9 upgrade notes](UPGRADE-v9.md) for notes on the upgrade process.
+
+* Setting `samples.privacy_precision` to 0 is now treated as a special value which causes the
+  sample to be hidden in default report behaviour. See https://github.com/BiologicalRecordsCentre/ABLE/issues/468.
+  There is a new field in the cache tables, `hide_sample_as_private`, to support this.
+* New features in the REST API for authorisation:
+  * New user interface in the warehouse under Admin -> REST API Clients, which allows new clients
+    for the REST API to be added without having to edit the config files. Clients are stored in the
+    database and define the keys required for JWT authentication. Clients contain one or more
+    available connections for that connection which define the set of records and actions which are
+    authorised for use by that connection. Using the config files to define clients is still
+    supported but is deprecated so may be dropped in a future version.
+  * Configuration for a project or connection in the REST API list of clients can specify a filter
+    ID in order to define the accessible records. This applies to connections configured in the
+    warehouse UI and stored in the database as well as projects configured in the config files.
+  * Ability to use a JWT token to authenticate as a client register in the warehouse UI.
+* Some fixes to the HTTP status response from the REST API, for example returning 401 Forbidden
+  rather than 404 Unauthorized when the user account is authorised but access to the requested
+  resource or operation is denied.
+* Add a Record Status control to the Edit Sample form on the warehouse.
+* Adds Elastic Stack containers to the Docker development system. Security
+  features have been enabled by way of a demonstration though these are not
+  needed in a dev environment.
+    - Elasticsearch indexes are configured for samples and occurrences.
+    - Logstash pipelines are configured to populate the indexes.
+    - Kibana can be used to explore the indexes.
+
 # Version 8.26.0
 *2024-02-19*
 

README.md

@@ -28,12 +28,15 @@ up of the indicia schema. E.g. on Ubuntu you can
 
 ### Starting
 If you clone this repo, `cd docker` and execute `./compose.sh` it will start
-5 docker containers offering these services.
+8 docker containers offering these services.
 1. A postgres database with postgis installed.
 1. pgAdmin for administering the database.
 1. A mock mail server.
 1. A webserver running the warehouse code.
 1. GeoServer for sharing spatial data in OGC standard format.
+1. Elasticsearch for storing an index of warehouse data.
+1. Kohana for exploring and managing Elasticsearch
+1. Logstash for populating Elasticsearch with warehouse data.
 On first run, it offers to initialise the indicia database schema.
 If you choose this option you will later login in as user `admin` having
 password `password`.
@@ -48,10 +51,17 @@ Once running you can browse the warehouse at http://localhost:8080.
 You can examine the database with pgAdmin at http://localhost:8070.
 Any mail sent by the warehouse can be viewed at http://localhost:8025.
 GeoServer can be configured at http://localhost:8090/geoserver.
+The Elasticsearch API is accessible at https://localhost:9200.
+Kohana is accessed by browsing https://localhost:5601.
 
 #### PgAdmin
-To connect pgAdmin to the database, configure the connection with
- - Host name: The docker container name e.g. indicia_postgres_1
+To log in , the default credentials are
+ - Email: [email protected]
+ - Password: password
+
+To connect pgAdmin to the database, add a new server and configure the
+connection with
+ - Host name: The docker container name e.g. indicia-postgres-1
  - Port: 5432
  - Username: postgres
  - Password: password
@@ -62,7 +72,13 @@ To list the container names and ports you can execute the command
 To log in, the default credentials are:
  - Username: admin
  - Password: geoserver
-
+
+#### Elasticsearch and Kohana
+Note that security is enabled so use https. To log in the default credentials
+are
+ - Username: elastic
+ - Password: password
+
 ### Unit testing
 There is a separate Docker configuration for unit testing which can be
 run up by `cd docker` then `./phpunit.sh`. This replicates the unit

UPGRADE-v9.md

@@ -0,0 +1,92 @@
+# Upgrading to version 9 of the warehouse.
+
+Version 9 adds fields to the reporting cache tables and the code that populates these fields will
+error if the upgrade scripts which add the fields have not been run. In order to avoid errors for
+posted records during the upgrade process, you have 2 options. Either take all client sites
+offline during the upgrade (e.g. by putting Drupal in maintenance mode), or by running the
+following script before the upgrade so that the fields are ready in the database. Note that the
+UPDATE statements in particular may take a long time, depending on the number of records and
+samples in your database.
+
+```sql
+ALTER TABLE cache_samples_functional
+ADD COLUMN IF NOT EXISTS hide_sample_as_private boolean;
+
+ALTER TABLE cache_occurrences_functional
+ADD COLUMN IF NOT EXISTS hide_sample_as_private boolean;
+
+-- Disable tracking increments, so doesn't force a complete ES refresh.
+SET application_name = 'skiptrigger';
+
+ALTER TABLE cache_occurrences_functional
+ALTER COLUMN hide_sample_as_private SET DEFAULT false;
+
+ALTER TABLE cache_samples_functional
+ALTER COLUMN hide_sample_as_private SET DEFAULT false;
+
+UPDATE cache_samples_functional SET hide_sample_as_private=false;
+
+UPDATE cache_occurrences_functional SET hide_sample_as_private=false;
+
+ALTER TABLE cache_samples_functional
+ALTER COLUMN hide_sample_as_private SET NOT NULL;
+
+ALTER TABLE cache_occurrences_functional
+ALTER COLUMN hide_sample_as_private SET NOT NULL;
+```
+
+After the upgrade the warehouse will ask you to run the 2nd part of this script using pgAdmin, if
+you have already run it there is no need to run it a second time.
+
+## Elasticsearch
+
+If you are using Elasticsearch, then before upgrading you should add the mappings required for new
+fields. You can run the following using the Dev tools in Kibana, replacing your index name:
+
+```
+PUT /my_occurrence_index/_mapping
+{
+  "properties": {
+    "metadata.hide_sample_as_private": {
+      "type": "boolean"
+    }
+  }
+}
+```
+
+Repeat this step for your samples index if you are also storing samples in Elasticsearch.
+
+You also need to add information about this new field to each of your occurrence *.conf files used
+by Logstash. Edit the files and search for a comment which starts `# Convert our list of fields`
+which should be just above a `mutate` block. Insert the following code before the comment:
+
+```yaml
+  mutate {
+    add_field => {
+      "hide_sample_as_private" => false
+    }
+  }
+  # Set hide_sample_as_private using privacy_precision value.
+  translate {
+    source => "[privacy_precision]"
+    target => "[hide_sample_as_private]"
+    override => true
+    dictionary => {
+      "0" => true
+    }
+    fallback => false
+  }
+```
+
+Also, in the list of rename operations in the mutate block just below, add the following after the
+rename operation for `privacy_precision`:
+
+```yaml
+  "hide_sample_as_private" => "[metadata][hide_sample_as_private]"
+```
+
+Now save the config file and repeat for any other pipeline configuration files that you have set
+up. Finally, restart the Logstash process or service as appropriate.
+
+One the above steps have been completed, it is safe to update your warehouse code then visit the
+home page in order to follow the link to upgrade the database.

application/config/version.php

@@ -29,14 +29,14 @@
  *
  * @var string
  */
-$config['version'] = '8.26.9';
+$config['version'] = '9.0.0';
 
 /**
  * Version release date.
  *
  * @var string
  */
-$config['release_date'] = '2024-03-13';
+$config['release_date'] = '2024-03-25';
 
 /**
  * Link to the code repository downloads page.

application/controllers/indicia.php

@@ -66,7 +66,7 @@ class Indicia_Controller extends Template_Controller {
    *
    * @var array
    */
-  protected mixed $auth_filter;
+  protected $auth_filter;
 
   /**
    * Name of the main template view file.
@@ -372,11 +372,12 @@ protected function getModelValues() {
   }
 
   /**
-   * Retrieve default values for an edit form.
+   * Retrieve default values for a create form.
    *
-   * Constructs an array of the default values required when loading a new
-   * edit form. Each entry is of the form "model:field => value". Loads both
-   * the defaults from this controller's main model, and any supermodels it has.
+   * Constructs an array of the default values required when loading a form for
+   * creating a new record. Each entry is of the form "model:field => value".
+   * Loads both the defaults from this controller's main model, and any
+   * supermodels it has.
    *
    * @return array
    *   List of default values keyed by fieldname.

application/controllers/termlists_term.php

@@ -92,13 +92,17 @@ private function internal_index($termlist_id) {
       'termlists_term:termlist_id' => $termlist_id
     );
     $this->upload_csv_form->returnPage = $termlist_id;
-    // Apply permissions. If core admin, or a private website-owned termlist, then its editable.
+    // Apply permissions. If core admin, or a private website-owned termlist,
+    // then its editable.
     // @todo: Could possibly allow editing of a termlist if public but only used by 1 site
     $this->view->readonly = !$this->termlist_authorised($termlist_id);
   }
 
   /**
-   * Define non-standard behaviuor for the breadcrumbs, since this is accessed via a term list
+   * Define non-standard behaviour for the breadcrumbs.
+   *
+   * Edit form is accessed via a termlist so breadcrumb should point back to
+   * the termlist.
    */
   protected function defineEditBreadcrumbs() {
     $this->page_breadcrumbs[] = html::anchor('termlist', 'Term Lists');

application/helpers/MY_valid.php

@@ -102,7 +102,8 @@ public static function date_in_past($date) {
    * @param string $column_value
    *   Column value to test.
    * @param array $args
-   *   Table name, table column, ID of current record.
+   *   Table name, table column, ID of current record. Optionally an extra
+   *   SQL filter in the last parameter.
    *
    * @return bool
    *   TRUE if valid.
@@ -111,12 +112,17 @@ public static function unique($column_value, array $args) {
     $db = new Database();
     $idFilter = empty($args[2]) ? '' : "AND id<>$args[2]";
     $value = pg_escape_literal($db->getLink(), $column_value);
+    $extraFilters = '';
+    if (count($args) > 3) {
+      $extraFilters = 'AND ' . $args[3];
+    }
     $qry = <<<SQL
 SELECT 1 AS hit
 FROM $args[0]
 WHERE deleted=false
 AND LOWER($args[1]) = LOWER($value)
 $idFilter
+$extraFilters
 LIMIT 1
 SQL;
     $found = $db->query($qry)->count();

application/helpers/report_standard_params_occurrences.php

@@ -625,7 +625,7 @@ public static function getParameters() {
         'description' => 'Exclude sensitive records?',
         'wheres' => [
           [
-            'sql' => "o.sensitive=false",
+            'sql' => "o.sensitive<>true",
           ],
         ],
       ],