Skip to content

Latest commit

 

History

History
20 lines (15 loc) · 1.14 KB

README.md

File metadata and controls

20 lines (15 loc) · 1.14 KB
Raw

Dam-Level-Data-Scraper

Scrape weekly dam water level data from the South African Department of Water and Sanitation using the Internet Archive for use in modeling.

Data source: http://www.dwa.gov.za/Hydrology/Weekly/Province.aspx

The script scraper.py will insert each province's data into a seperate CSV where the rows are ordered by date. It will not normalize the data, which has to be done seperately using the functions in normalize.py. Normalizing will squeeze the FSC (Full Storage Capacity) between 0.0 and 1.0 and the percentage values will be converted into corresponding decimal values for the percentages.

usage: scraper.py [-h] [--output OUTPUT] [--provinces PROVINCES [PROVINCES ...]] [--no_override]

Scrape South African dam water level data from the Internet Archive

optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
                        directory where the CSVs will go
  --provinces PROVINCES [PROVINCES ...], -p PROVINCES [PROVINCES ...]
                        ids of the specific provinces to download
  --no_override, -n     will not redownload data that is already present