GHSCI spatial and software update

[carlhiggs]

Hi @global-healthy-liveable-cities/software-spatial team ,

I just wanted to share a quick update of what I've been up to ahead of our working group meeting next week -- will be great to get your thoughts on both of these things:

a bug with GTFS (as per issue)

urbanaccess, or our implementation of it (but i think its the package) apparently doesn't deal properly with calendar_dates, which is an issue for some cities - like Barcelona)

• I drafted a possible solution for this briefly on gtfs branch using gtfs-lite (see thread for instructions; need to manually install package for now, as I haven't changed docker) --- I think its promising, but as per above thread, it either hasn't solved the problem or its caused new problems
• anyway - this isn't resolved and if you have thoughts/interest you're welcome to look into it!

an enormous new feature boost (we can run our analysis as imported libraries in python, including in Jupyter notebook - as per demo)

• this is on enhancements branch
• a focus on the basic functionality, for now; I know the demo isn't pretty yet
• changes are basically to instructions and usage
  • previously the software only really worked from commandline; that's how it was coded; there are other usage options now
  • regions are now constructed using a Region class
  • I completely rewrote the setup script (perhaps I'll remove it or write over it); for now its superceded by the ghsci.py library; to create a study region: r = ghsci.Region(codename). The r object contains all the config relating to the study region, while project settings are found in ghsci.settings and ghsci.indicators etc.
• all scripts were re-written to allow the functions and classes to be imported as modules in other scripts or notebooks
• the command line behaviour remains the same
• I renamed our 4 fundamental scripts: configure.py, analysis.py, generate.py, compare.py
• I created aliases for these, so you run like
  • configure codename
  • analysis codename
  • generate codename
  • compare codename comparison_codename
• you can still run them as python scripts of course
• but >>> you can run them in a jupyter notebook, and maybe that's better for a lot of people (see linked demo above)
• so, jupyter lab can now be launched from the main container
  • should probably create an alias because currently it uses this command: jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --notebook-dir=/home/ghsci/work/process --allow-root
  • on the other hand, you should also be able to use the "global_indicators.bat jupyter" method (launches it directly without going to the command prompt at all)
• Jupyter Lab has the side benefit that users can use it to edit their yml configuration files, so our demonstration videos should make use of it to show how that would be done.

Here's what the new changes look like:

Running the notebook (no pictures - just text outputs like in terminal --- but its kind of useful, as it directs people like before what to do, but the jupyter interface means the text is broken up into the sections of analysis so its more readable; nice to have file browser on the side for our demonstrations)

Editing some yaml

Anyway - I've literally only just finished implementing this and its not ready for a pull request. There will be bugs e.g.
tqdm needs a widget update to work better with jupyter (you can see the error in the demo screenshot).

I suspect some of the docker compose launch commands used may assume use of a recent version of Docker desktop but I haven't been able to investigate this fully yet. Ihaven't tested on linux/MacOS.

But in broad strokes, I think this is pretty useful and helps address at least part way the need for different ways of using for different users.

The new approach also means it will be easier if we did want to create a barebones graphical user interface (GUI). But then again, once we've tested this, maybe we could go to our early adopters and see if they feel they need one, or if providing a series of example/template jupyter notebooks and/or a means to generate them for their cities would be sufficient.

I know this involved a lot of changes on the enhancements branch -- but again, having this done will make the future use of a more test driven and continuous integration/continuous development approach easier to implement.

That's all -- keen to hear your thoughts some time

[carlhiggs]

By the way - for now in the analysis script the subprocesses are all still launched as subprocesses (invoked to be run as seperate processes by python) --- but in principle all scripts can now be imported directly, and we/our users could run them that way if they wanted.

A lot more options with workflow orchestration with the new architecture.