Skip to content

Commit

Permalink
Update readme documents for new data organisation and preparation
Browse files Browse the repository at this point in the history
  • Loading branch information
@StevePem
StevePem committed Feb 7, 2024
1 parent dbc5987 commit c19d673
Show file tree
Hide file tree
Showing 4 changed files with 143 additions and 100 deletions.
23 changes: 13 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,19 @@ Adjustable parameters are listed under the Parameters sub-heading.
Running the algorithm requires an input parameter 'city', and adjustable parameters must be completed for that city, specifying locations of relevant input files and the applicable CRS. If running for a location for which 'city' parameters have not already been defined, then these must be added, using existing city parameters as a template.

The city parameters are as follows.
* region - required if OSM extract is to be downloaded, or destinations (see below) are to be extracted. This must be the location of a file in sqlite format which defines the boundary of the area for which the OSM extract is required.
* outputCrs - specify the appropriate EPSG coordinate reference system number for the region.
* osmGpkg - the location to which an OSM extract in .gpkg format will be saved, or where an existing .gpkg file is stored if already held.
* unconfiguredSqlite - the location to which an unconfigured network in .sqlite format will be saved, or where an existing unconfigured network is stored if already held.
* cropAreaPoly - an optional parameter for cropping the OSM extract to a smaller test area.
* demFile - required if 'addElevation' is set to 'T'. This must be the location of a digital elevation model raster file in the same CRS as the the network.
* ndviFile - required if 'addNDVI' is set to 'T'. This must be the location of a raster file with NDVI values in the same CRS as the network.
* gtfs_feed - required if 'addGtfs' or 'addDestinationLayer' is set to 'T'. This must be the location of a zip file containing GTFS data.

The parameters assume that the relevant files are stored in the a 'data' subdirectory.

| Parameter | Parameter type | Requirements |
|--------------------|----------------|-----------------------------------------|
| region | .sqlite file | Required if OSM extract is to be downloaded, or destinations (see below) are to be extracted. This must be the location of a file in sqlite format which defines the boundary of the area for which the OSM extract is required.|
| outputCrs | CRS |Specify the appropriate EPSG coordinate reference system number for the region.|
| osmGpkg | .gpkg file | The location to which an OSM extract in .gpkg format will be saved, or where an existing .gpkg file is stored if already held.|
| unconfiguredSqlite | .sqlite file | The location to which an unconfigured network in .sqlite format will be saved, or where an existing unconfigured network is stored if already held.|
| cropAreaPoly | area location | An optional parameter for cropping the OSM extract to a smaller test area. See https://github.com/JamesChevalier/cities/tree/master/australia/victoria for available locations. |
| demFile | .tif file | Required if 'addElevation' is set to 'T'. This must be the location of a digital elevation model raster file in the same CRS as the the network. |
| ndviFile | .tif file | Required if 'addNDVI' is set to 'T'. This must be the location of a raster file with NDVI values in the same CRS as the network. |
| gtfs_feed | gtfs .zip file | Required if 'addGtfs' or 'addDestinationLayer' is set to 'T'. This must be the location of a zip file containing GTFS data. |

**The parameters assume that the region file and (if used) the demFile, ndviFile and gtfs_feed are stored in the 'data' subdirectory.** See `data/README.md` for more detail on obtaining the data files.

The algorithm will do the following:
* if 'downloadOsm' is set to 'T', download an OSM extract for the selected 'region' and save it as a .gpkg file.
Expand Down
67 changes: 33 additions & 34 deletions data/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,37 +1,36 @@
# Update required
# Network generation data

The information below requires update, once final data location and structure is determined.
This directory contains input files required to generate MATSim networks for Bendigo and Melbourne, which can be found [here](https://osf.io/ajycn/).

# Network generation data
## Files to download

Download the following files for the relevant network location.

### Bendigo
| File | Description |
|--------------------------|---------------------------------------------------|
| greater_bendigo.sqlite | Boundary of the Greater Bendigo Local Government Area |
| [to come] | Digital elevation model data for the Greater Bendigo area |
| [to come] | [NDVI] data for the Greater Bendigo area |
| gtfs.zip | GTFS feed for Victoria as at 20 October 2023 |


### Melbourne
| File | Description |
|--------------------------|---------------------------------------------------|
| greater_melbourne.sqlite | Boundary of the Greater Melbourne Greater Capital City Statistical Area |
| [to come] | Digital elevation model data for the Greater Melbourne area |
| [to come] | [NDVI] data for the Greater Melbourne area |
| gtfs.zip | GTFS feed for Victoria as at 20 October 2023 |


## Other files

The directory also contains the following other files, from which the region boundary [and DEM] files above were created, using the code contained in `data/data prep tools.R`. That code may also be useful to generate similar data input files for other locations if required.

| File | Description |
|---------------------------------|--------------------------------------------|
| LGAs.zip | Local government areas of Victoria (VICMAP) |
| GCCSA_2021_AUST_SHP_GDA2020.zip | Greater capital city statistical areas (ABS) |
| [to come] | Digital elevation model data for Victoria |

This directory contains inputs files required to generate a MATSim network for Melbourne, which can be found [here](https://cloudstor.aarnet.edu.au/plus/s/ssLkX8Uez64rV3D). Alternatively, you can use the `./prepare.sh` command as described below to download the data you need.

## How to populate

To populate this directory with the required data files, use the `./prepare.sh` with the relevant arguments. Valid arguments and their descriptions are presented in the table below:

| Argument | Input file | Description |
|----------|------------------------------|-----------------------------------------------|
| -region | studyRegion.sqlite | Greater Melbourne region |
| -osm19 | melbourne.osm | Raw OSM file for Melbourne, 2019 |
| -melb | melbourne.sqlite | Road attributes |
| -net | network.sqlite | non-planar edges and nodes |
| -gtfs19 | gtfs_au_vic_ptv_20191004.zip | GTFS feed - 2019-10-04 |
| -demx10 | DEMx10EPSG28355.tif | Digital Elevation Model data (x10, EPSG28355) |
| -A | all of the above | It Will download all the input files, (~1.2gb)|

As an example, to start from processing raw OSM (step 1), and generating a network without elevation and public transport, you need to run the following to get the required input:
```
./prepare.sh -osm19
```

Alternatively, if you want to skip processing raw OSM and start directly from `makeNetwork.sh`, and generate a network that has elevation and PT network from GTFS, you need to run the following to download required inputs:
```
./prepare.sh -melb -net -gtfs19 -demx10
```
If you are not sure about which inputs are required, just simply run the following to download all the inputs:
```
./prepare.sh -A
```

If any issues with the script, please download each required file directly.
97 changes: 97 additions & 0 deletions data/data prep tools.R
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Tools for creating input data files for use in NetworkGenerator
# Currently these tools are applicable for Victoria only

library(tidyverse)
library(sf)
library(terra)

# 1 Region boundaries from LGA or GCCSA files ----
# -----------------------------------------------------------------------------#
# function for extracting specific region from an administrative districts file
getRegion <- function(input.file, input.field, input.name,
output.filename, outputCrs) {

input.transformed <- input.file %>%
st_transform(outputCrs)

output <- input.transformed %>%
filter(.data[[input.field]] == input.name)

st_write(output, paste0("./data/", output.filename, ".sqlite"))
}

# function to extract zipped shapefile
# Note: 'subpath' is the string between the top zipped file and the ultimate file, eg "/gda2020_vicgrid/esrishape/whole_of_dataset/victoria/VMTRANS"
# 'file' not needed for files that don't have layers (eg shapefiles) if there is only one in the directory
# use 'file' (rather than 'layer') for shapefiles and mapinfo files; use both for gpkg and sqlite
read_zipped_GIS <- function(zipfile, subpath = "", file = NULL, layer = NULL) {
temp <- tempfile()
unzip(zipfile, exdir = temp)
if (is.null(layer)) {
st_read(paste0(temp, subpath, file))
} else {
st_read(paste0(temp, subpath, file), layer)
}
}

# Greater Bendigo - from LGAs
getRegion(input.file = read_zipped_GIS(zipfile = "./data/LGAs.zip",
subpath = "/gda2020_vicgrid/esrishape/whole_of_dataset/victoria/VMADMIN"),
input.field = "NAME", input.name = "GREATER BENDIGO",
output.filename = "greater_bendigo", outputCrs = 7899)


# Greater Melbourne - from GCCSA
getRegion(input.file = read_zipped_GIS(zipfile = "./data/GCCSA_2021_AUST_SHP_GDA2020.zip"),
input.field = "GCC_NAME21", input.name = "Greater Melbourne",
output.filename = "greater_melbourne", outputCrs = 7899)



# 2 Elevation from whole of state file ----
# -----------------------------------------------------------------------------#
# function for extracting region's elevation from whole of state file
getRegionDem <- function(dem.location, region.location,
output.filename, outputCrs) {
dem <- rast(dem.location) %>%
project(., outputCrs)

region <- st_read(region.location)

dem.cropped <- terra::crop(x = dem, y = region %>% st_buffer(1))

writeRaster(dem.cropped, paste0("./data/", output.filename, ".tif"),
gdal = "COMPRESS = DEFLATE", overwrite = TRUE)
}

# Bendigo
getRegionDem(dem.location = "./data/vmelev_dem10m_ESRI_grid_GDA94_VicGrid/vmelev_dem10m_ESRI_grid_GDA94_Vicgrid/vmelev_dem10m/dem10m/hdr.adf",
region.location = "./data/greater_bendigo.sqlite",
output.filename = "DEM_bendigo", outputCrs = 7899)

# Melbourne
getRegionDem(dem.location = "./data/vmelev_dem10m_ESRI_grid_GDA94_VicGrid/vmelev_dem10m_ESRI_grid_GDA94_Vicgrid/vmelev_dem10m/dem10m/hdr.adf",
region.location = "./data/greater_melbourne.sqlite",
output.filename = "DEM_melbourne", outputCrs = 7899)



# from other file
dem <- rast("./data/vmelev_dem10m_ESRI_grid_GDA94_VicGrid/vmelev_dem10m_ESRI_grid_GDA94_Vicgrid/vmelev_dem10m/dem10m/hdr.adf") %>%
# transform to project crs
project(., "EPSG:28355")

# load network
network <- st_read("./data/melbourne_network_unconfigured.sqlite")

# crop dem to network (will crop to bounding box)
dem.network <- terra::crop(x = dem, y = network %>% st_buffer(1))

# save output
writeRaster(dem.network,
"./data/DEM_melbourne.tif",
gdal = "COMPRESS = DEFLATE",
overwrite = TRUE)



56 changes: 0 additions & 56 deletions data/prepare.sh
View file

This file was deleted.

0 comments on commit c19d673

Please sign in to comment.