This repository contains instructions for you to get started contributing to projects under Project Sidewalk. It has instructions for you to setup each of the components needed for the dev environment.
If you are running Linux, you can also set up your development environment using Docker. See the Project Sidewalk docker repo for Docker configuration files and instructions.
We use Postgres (or PostgreSQL) for persistent data storage of user data and GIS data. We choose PostgreSQL over other databases (e.g., MySQL) primarily because of its geographical data support. You can either install Postgres and plug-ins directly on your computer (not recommended), or use a virtual machine with Postgres already installed.
- (Recommended) If you are installing them using a virtual machine, see Virtual Box and Vagrant below.
- If you are installing Postgres directly into your computer, see the following pages:
- Windows:
- PostGIS : PostGIS's official web page. Follow the instruction to install PostgreSQL as well as PostGIS (> 2.0.0).
- OS X:
- Kyng Chaos : Kyng Chao offers a PostgreSQL 9.4 binary for OS X that include PostgreSQL, PostGIS, and pgRouting. This would be the easiest way to go for the OS X user. Once installed, start the database with the following command.
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/. To start the postgres, run/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data/
- Kyng Chaos : Kyng Chao offers a PostgreSQL 9.4 binary for OS X that include PostgreSQL, PostGIS, and pgRouting. This would be the easiest way to go for the OS X user. Once installed, start the database with the following command.
- Windows:
- Start by installing VirtualBox (www.virtualbox.org/wiki/Downloads) and Vagrant (www.vagrantup.com).
Windows users should also install an SSH client as well (or the chance is you already have it if you are using git.
Add
C:\Program Files (x86)\Git\binto the PATH. See here and here for more information.) - Check out this repo (on your local machine) by typing
git clone https://github.com/ProjectSidewalk/Instructions.git - To get to the Instructions folder, navigate directories using command line commands (e.g. cd). Type the following command in the directory (Instructions) where the
Vagrantfileis located:vagrant up. This will create a new Ubuntu Trusty 64 Bit virtual machine and configure it for this project. This step will take a while. - When the set up completes, you should be able to log into the virtual machine you've just installed by typing:
vagrant ssh. If you get a connection timeout error, and/or a "ssh_exchange_identification: read: Connection reset by peer" message, it may be the case that you need to enable hardware acceleration in the BIOS. See this post. Another issue may be that you need to disable Secure Boot. - When you are done working, stop the virtual machine by running
vagrant suspend. When you want to restart working again, use:
vagrant resume
vagrant ssh
Once you have set up VirtualBox and Vagrant, send an email to Manaswi Saha ([email protected]) or Mikey Saugstad ([email protected]) so they can send you the data to be imported into the database (<sidewalk-dump-file>). Once you get the data, put it under the directory resources, which you should create in the Instructions folder.
To import data, you should run the following commands (you may need to run it as a super user. Run: sudo su - postgres):
$ vagrant ssh
vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant/resources
vagrant@vagrant-ubuntu-trusty-64:~$ createdb -T template0 sidewalk
vagrant@vagrant-ubuntu-trusty-64:~$ pg_restore -d sidewalk <sidewalk-dump-file>
When the import completes, you can expect the following warning: WARNING: errors ignored on restore: 2.
If you received more than 2 errors, it is likely either because the size of virtual disk for your VM is too small OR because of a recent issue we've had where the pgrouting library isn't installed on the VM like it should be.
Start by trying to install the pgrouting library. From this installation guide:
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql-9.3-pgrouting
Then try dropping the database you had tried to restore by running dropdb sidewalk. Then try running the three import commands again.
If you are trying to load a very large data dump or multiple data dumps, you may need to increase the size of the virtual disk. You can either get a smaller dump of the database or try using the vagrant-disksize plugin to expand the size of the virtual disk (we have only tried this using on Ubuntu 16.04 using Vagrant 1.9.7 and VirtualBox 5.1.26).
Once you install Postgres using Vagrant, you should be able to access it at port 5432. Test if your database is up and running. Run the following commands on the directory where you have the Vagrant file:
$ vagrant ssh
vagrant@vagrant-ubuntu-trusty-64:~$ psql -d sidewalk
Check if the user name sidewalk exists on the database:
psql -d sidewalk
\du
Once you import the data, you should be able to access it via command line or a database client. Download one of the Postgres clients listed below (e.g., Postico), and access the database using the following credential information:
Host: localhost:5432
User: sidewalk
Password: sidewalk
Database: sidewalk
Windows users who download pgAdmin 3 can access the database by opting to "Add a new server" and filling the form with the following information:
Name: server_name_of_your_choice
Host/Host Address: localhost
Port: 5432
Maintenance DB: postgres
Username: sidewalk
Password: sidewalk
To exporting the data in the server, run the following after logging into the server with ssh.
scl enable postgresql92 bash
pg_dump -h <hostname> -U <username> -Fc <database> -f dump.sql
The first line will drop you into a subshell with the environment variables setup to point you at the pgsql 9.2 installation instead of the system default. The second line dumps all the data in the database into dump.sql. See more on: http://www.postgresql.org/docs/9.2/static/app-pgdump.html
Follow this set-up instruction to contribute to the projects that use Java or Scala.
- Install Java Development Kit version 7 (JDK 7). JDK 8 is backward compatible and our code should work on it too.
- Install
activator(orsbt).
- Install
npm, a package manager for JavaScript. I think the easiest way for Mac uses isbrew install node. For Windows users, follow this guide. - (Optional) Install
Grunt, a task runner for JavaScript.
Install Python. I recommend installing Anaconda, a Python distribution with all the scientific packages (e.g., numpy) bundled by default.
Effectively using an IDE for development is an important skill you need. When you are working on this project, I highly recommend you to use IDE. My favorites are those offered by JetBrains (e.g., IntelliJ for Scala and Java, PyCharm for Python). They also have student license which gives a free access to a set of their products.
This is based on this stackoverflow post.
- In a terminal, go to the root directory of your Play project. Then run:
activator -jvm-debug 9998 - Start the play application by entering
runin the activator console. - Set up IntelliJ's debug setting. Go to Run > Edit Configurations...
- Add a new configuration; click "+" and select "Remote".
- Choose Transport: Socket, Debugger mode: Attach, Host: localhost, Port: 9998, and select the appropriate module's classpath. Click "Ok."
- Run a debugger and set break points. Then you should be good to go!
As you can see, we use Git and GitHub for version control and collaboration. See below for the list of concepts you should know:
- Branching:
- Create a branch to work on your task. You can create a branch by "git checkout -b "
- Make changes to the code in your branch
- Once you are done with editing the code, issue a "pull request"
- Wait for a code to be reviewed. (You can work on other stuff by branching )
- Merge the code once reviewed. If there are conflicts, resolve it. See: this stackoverflow post
- Pull Request:
- For more information, see:
- GitHub Training & Guides: A YouTube channel for learning Git and GitHub.
- Git tutorial by RyPress : This is a short and concise tutorial of GIT. I find it more approachable than the official Git documentation.
- Pro Git : The official git documentation.
This section introduces various tools that may become handy while working on the projects.
Database client programs make it easier to interact with tables. The following PostgreSQL clients provides GUI to view and manipulate the database.
- Postico: A free Postgres client for OS X.
- Valentina Studio: A cross-platform database client.
- pgAdmin 3: A PostGreSQL client for both Windows and macOS.
It is possible to use SSH to connect to the remote database. For instance, this is useful if you have to apply a python script against the data stored in the remote database.
ssh -L 63333:localhost:5432 [email protected]
The first number in the -L argument, 63333, is the port number of your end of the tunnel; it can be any unused port. (IANA reserves ports 49152 through 65535 for private use.) The second number, 5432, is the remote end of the tunnel: the port number your server is using. (From the PostgreSQL web page)
The web applications that run on the UMIACS server need mechanisms to auto-restart as the server shuts down/restart periodically, killing the application processes. Two approaches recommended by the UMIACS staff are (i) making a crontab entry and (ii) adding a service to the systemd service manager.
Cron & crontab
- Write a shell script that runs your application (see below)
- Make the script executable:
chmod 755 sidewalk_webpage_runner.sh - Run
crontab -eorEDITOR=/usr/bin/emacs crontab -e - Add the following line:
@reboot /PATH/TO/DIR/sidewalk_webpage_runner.shand save the crontab file - Check if the script has been added to the crontab entries by runing:
crontab -l
#!/bin/bash
# Filename: sidewalk_runner.sh
nohup sidewalk-webpage/bin/sidewalk-webpage -Dhttp.port=9000 &
When you are working with GIS dataset, you have to visualize it at many stages of your research & development cycle (e.g., debugging, analysis). Following tools are what I have found handy while working with GIS dataset.
- QGIS : A comprehensive open source GIS tool. I recommend to install OpenLayers plug-in and Quick OSM.
- geojson.io : A browser based geojson data viewer.
- ogr2ogr : ogr2ogr allows you to convert different file formats
- osm2pgrouting : The application allows you to convert data from the OSM format into the pgRouting format and populate a database. If you are on OS X, you can install it via homebrew (
brew install osm2pgrouting). - pgShapeloader: The tool allows you to load a shapefile into PostgreSQL.
- FOSS4G routing with pgRouting: http://workshop.pgrouting.org/index.html
- Introduction to PostGIS : A comprehensive tutorial for using PostGIS.
- Download OSM file using wget: http://workshop.pgrouting.org/chapters/installation.html (Use the following bounding box parameter for downloading the Washington DC dataset: -76.9,39.0,-77.1,38.8) Or simply download the data from http://download.bbbike.org/osm/bbbike/WashingtonDC/
- Follow the steps to import the OSM data into the PostGIS-enabled database (or use a python script with Imposm ) http://workshop.pgrouting.org/chapters/osm2pgrouting.html
Finish up setting up the development environment by going back to the instructions in the README file in the main Sidewalk webpage repo.