Configuration simplification

[carlhiggs]

As per #309 , we still have an issue with our configuration process being over-complex.

Related changes to consider include:

• do away with the 'process' folder (make everything there in the root directory)
• have a 'region' folder, that is in effect a data folder
  • to configure a new region, a folder is created for that region containing a self-named user-text file (this could be initialised with a help utility)
  • data for that region could be stored in that folder, but configuration paths could also allow for data to be accessed in other folders optionally for advanced users running analyses for multiple cities (ie. the root directory is 'regions' for all data paths)
  • an output directory is created in the region folder, so configuration, inputs and outputs are all kept together in a parent region folder, while being clearly identified (perhaps a utility could provide a way to collate and package outputs from multiple processed regions)
  • some of the config.yml parameters could go into the region configuration, perhaps as optional parameters (implicit if not specified, but can be over-ridden, optionally)

Other things to think about

• would it make it easier if the code were packaged in the Docker image, and the directory it is launched from is used as the directory with region configuration, inputs and outputs?
• could provide an alternative configuration file format (eg formatted Excel, that if a yml is not present may be identified as a fall back and parsed in an equivalent way --- may be easier for some users to use this; we could generate the equivalent yml file in the outputs folder as a record of provenance)

An illustrative directory structure for the folder where a user launches the process, as viewed from within the GHSCI container process itself:

User project folder
├───.settings  # contains configuration stuff most users are unlikely to modify
├───.subprocesses # contains code that most users are unlikely to access
├───regions # contains all the things a user will want to interact with
│   └───Example  # a provided example, illustrating how users can initialise folders and configurations
│       ├───resources # input resources and documentation
│       ├───outputs # a generated folder, with all the outputs generated through the process
│       └───ES_Las_Palmas_2023_example.yml # configuration file,  perhaps a simplified yml?
└───ghsci.py

In this approach,

• we could distribute the command for launching the GHSCI docker image as an executable where the user specifies their project directory
  • they could also choose whether to launch as a terminal shell for scripting, a web app GUI, or Jupyter Lab (the latter could be optionally launched from the shell)
  • the process retrieves and runs the GHSCI image and PostGIS image, with the GHSCI process launching in the specified directory
• Users could store their region specific Jupyter notebooks or scripts in their region folder (or any other folder they chose to).

---

The above paradigm assumes the continuation of the current coupling with Postgis and running in a Docker image. An alternate (or complementary) approach would be to re-do the package as a user-installable python library. That would require them to provide their own database solution etc, and for us to provide the flexibility for that to be done.

I'm not sure about what approach should be taken to rethink things, if any. But the above are placeholder thoughts that we can discuss, @global-healthy-liveable-cities/software-spatial . Perhaps once we come up with some good options, we could propose these to our early adopters and solicit feedback/preferences for change vs status quo. Keen to hear your thoughts.

The tricky thing with wondering about all these things is, we also need to better document our configuration process --- we previously made this simpler, so it is possible to demonstrate this now, but I keep deferring this to add new features, which makes it easier to explain things, but further defers the documentation itself... Where do we draw the line or how do we manage the documentation / feature adding / simplification?

[carlhiggs]

Just a quick comparison of the possibility sketched above for a simpler data structure (that might help configuration),

├───regions # contains all the things a user will want to interact with
│   └───Example  # a provided example, illustrating how users can initialise folders and configurations
│       ├───resources # input resources and documentation
│       ├───outputs # a generated folder, with all the outputs generated through the process
│       └───ES_Las_Palmas_2023_example.yml # configuration file,  perhaps a simplified yml?

This is an excerpt of the current structure for those equivalent folders:

├───process
│   ├───configuration
│   │   └───regions  # a provided example, illustrating how users can initialise folders and configurations
│   │       └───ES_Las_Palmas_2023_example.yml # configuration file,  perhaps a simplified yml?
│   └───data 
│       ├───_study_region_outputs # a generated folder, with all the outputs generated through the process
│       │   └───ES_Las_Palmas_2023_example
│       │       └─── ... # output files
│       ├─── .... # other input data folders

For a person configuring a new region, debugging an existing one or gathering outputs they have to skip back and forth between these folders. The output data folder is 4 folders deep in both configurations, but in the first one, all the relevant parameters (configuration, inputs, outputs) are clustered and clearly identified in the third folder (root > regions > named_region).

In effect, only one directory needs to be remembered or open. I think that structure might be more intuitive. It would be interesting to hear if others think that might make configuring and using the software easier.