Django 1.4 project template for fast Heroku deployment

About

This is a template for starting Django 1.4 projects and rapidly deploying to heroku. It is opinionated and configured for the things that I like and repetitively end up adding before deployment.

This is a first version, while it has been tested and used it is not promised to be complete or bug free.

Skel Template Features

#####Logical default file tree

• Global assets, fixtures, applib directory.
• Project template and misc directories by default.
• Collects static and media into assets/{static-destination,media} respectively.

#####Deployment best practices by default

• Uses virtualenv and virtualenvwrapper.
• Uses pip and requirements.txt.
• Uses of git.
• Uses environment variables for secrets.
• Includes a .gitignore for the usual junk.

#####Sane settings.py configurations by default

• Timezone set to UTC.
• Discourages storing credentials and secrets in the repository.
• Encourages storing credentials and secrets in environment variables
• Uses env['DJANGO_ENV'] to configure settings for each environment ['PRODUCTION', 'TEST', 'DEV'] in a single settings.py file that gets stored in repo
• Tolerates the use of local_settings.py file, by default only in DEV.

#####Encourages simple separation of apps by type:

• Unmodified 3rd party apps installed to virtualenv site-packages
• Modified 3rd party apps placed in applib/ directory, applib/ added to path
• Apps specific to project to be created in project_name/[app_name] directory

#####Ready for Heroku Deployment

• Includes Procfile, using gunicorn
• Includes storages, s3_folder_storage (allows separate directories for media and static)
• Setup to use sendgrid for email delivery

#####Common apps already installed with reasonable default configuration

• Django admin activated by default.
• allauth
• south
• django-storages for static/media on s3
• django-folder-storages to keep static/media in separate silos

####Other Goodies

• Simple 404 and 500 error templates.
• TODO: bootstrap driven base.html
• Automatically builds a README with installation notes.

####Template

• Twitter Bootstrap based template included
• JQuery & Twitter Bootstrap themed JQUI included
• AllAuth login signup manage links already in header
• JS in footer for faster page loading
• Google analytics and Addthis code in base.html, commented out

###Template blocks cascade

• js > extra-js, css > extra-css, head > extra-head
• any use of an extra-[foo] block should include call to super
• block content is inside of a block canvas allowing for app level layouts while still using common block content idiom without a call to super at the page level
• expects base.html > app-base.html > view.html series of cascades

Prerequisites

• Heroku account, AWS account
• python2 >= 2.7, pip, virtualenv, git

First time system config/installation stuff.

easy_install pip
pip install virtualenv, virtualenvwrapper
brew install postgresql
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
heroku login

Installation

Preparing for a new project

There is some wait time while pip installs requirements and while running the first set of tests, these steps can be delayed until waiting for tasks to complete.

• Gather domain credentials
• Collect AWS Credentials
  1. AWS Management Console > IAM > Users > Create new user > [project_name]
  2. Record credentials
  3. Bottom Pane, Permissions Tab > Attach User Policy > S3 Full access NB: This is a convenience shortcut, for production access should be limited to specific buckets.
• Create s3 bucket
  1. AWS Management Console > S3
  2. Create Bucket [project_name]-logging
  3. Create Bucket [project_name]-assets-production, logging to ibid, prefix s3

Start the new project

First time setups steps

Create Virtualenv

virtualenv --no-site-packages --distribute ~/ve/[project-name]
source ~/ve/[project-name]/bin/activate

Install Django

pip install Django

Start a project.

django-admin.py startproject --template https://github.com/tedtieken/django-project-skel/zipball/master --extension py,md,gitignore,dist [project-name]
cd [project-name]
touch virtenv_is_[project-name]

Install base requirements.

pip install -r requirements.txt

(Optional) Check for newer versions Note that this command isn't instant, it can take a few seconds to start showing output. This is normal.

pip freeze | cut -d = -f 1 | xargs -n 1 pip search | grep -B2 'LATEST:'
pip install [package] --upgrade

Create local .env file

cp sample.env .env && rm sample.env

NB there are other ways to manage dev environment variables, see discussion below.

Validate

foreman run python manage.py validate

Create DB

sudo -u postgres createuser user -P
...
Enter pass
n
y
y
...
sudo -u postgres createdb -O user [project-name]

Local changes script

Syncdb and migrate

foreman run python manage.py syncdb && foreman run python manage.py migrate

Collect static

foreman run python manage.py collectstatic --noinput

Run Tests

foreman run python manage.py test

Fire up local/dev server

foreman start
#... or ...
foreman run python manage.py runserver 0.0.0.0:8000

Deploying

Preparing for first deploy

Create git repo

git init
... 
Initialized empty Git repository in ...
...
git add .
git commit -m "initial commit"

Create heroku app

heroku create [appname]

Add addons: sendgrid for email and postgress for database

heroku addons:add sendgrid:starter
heroku addons:add heroku-postgresql:dev
heroku addons:add pgbackups:auto-month
heroku addons:add newrelic:standard
heroku addons:add scheduler:standard

Promote database to DATABASE_URL

heroku config
...
HEROKU_POSTGRESQL_[color]_URL: postgress://x:[email protected]:XXXX/ABCDEFGHIJ
...
heroku pg:promote HEROKU_POSTGRESQL_[color]

Configure Environment Variables

heroku config:set DJANGO_ENV="PRODUCTION" DJANGO_DEBUG="false" 
heroku config:set AWS_ACCESS_KEY_ID=[KEY] AWS_SECRET_ACCESS_KEY=[KEY] 

Regular Deploy Script

Run All Tests

foreman run python manage.py test

Freeze Pip state

pip freeze > requirements.txt

Commit Changes

git add .
git commit -m "message"

Push repo to heroku

git push heroku master

Sync and migrate the remote DB

heroku run python manage.py syncdb && heroku run python manage.py migrate

Collect remote static If you haven't configured AWS correctly, this is going to break loudly

heroku run python manage.py collectstatic --noinput

Go see your site in action

heroku open
heroku logs --tail

That's it, you're configured and deployed. Now go build something awesome.

Housekeeping

• Login to admin and set site, so emails don't come from example.com

Getting custom domain set up

https://devcenter.heroku.com/articles/custom-domains https://devcenter.heroku.com/articles/avoiding-naked-domains-dns-arecords

If you set up one of heroku's [adjective]-[item]-[number] app names, rename to your domain

heroku apps:rename [domain]

Note that if you change this via the website you'll have to move around the git remotes and do a checkout: https://devcenter.heroku.com/articles/renaming-apps

Add the domain

heroku domains:add www.[domain].com

Setup CNAME for www to [appname].herokuapp.com Setup forwarding for naked domain to www.[domain].com

Wait ~15-60 minutes

Alternatively, the zerigo DNS app is another way to set this up that may be preferred.

Options, considerations, and commands worth discussing

Deploying

Procfile settings

Dev only runserver.

web: python manage.py runserver "0.0.0.0:$PORT"

Standard #1: production setup gunicorn server

python manage.py run_gunicorn 0.0.0.0:$PORT -w 3

Standard #2: newrelic and gunicorn Recommended starting setup https://newrelic.com/docs/python/django-on-heroku-quick-start

web: newrelic-admin run-program python manage.py run_gunicorn -b "0.0.0.0:$PORT" -w 3

Standard #3: newrelic, gunicorn, and celery scheduler/worker https://newrelic.com/docs/python/django-on-heroku-quick-start Note: This setup doesn't include celerybeat, instead use the heroku scheduler.

web: newrelic-admin run-program python manage.py run_gunicorn -b "0.0.0.0:$PORT" -w 3
# worker: newrelic-admin run-program python manage.py celeryd -E --loglevel=INFO

NB: The following Advanced setups use gevent to process requests asynchronously. This can yield substantial performance improvements, but can also make debugging substantially more complicated. They also include both a celery scheduler and worker -- there should never be more than one scheduler instance. I have not used or confirmed that these settings work -- they are here for reference.

Advanced, with newrelic, gunicorn, gevent, and celery worker from https://github.com/seanbrant/django-project-skeleton

web: newrelic-admin run-program gunicorn [project_name].wsgi -w 4 -b 0.0.0.0:$PORT -k gevent --max-requests 250
# scheduler: newrelic-admin run-program python manage.py celeryd -B -E --loglevel=INFO
# worker: newrelic-admin run-program python manage.py celeryd -E --loglevel=INFO

Advanced with newrelic, gunicorn, gevent, celery scheduler and worker https://github.com/rdegges/django-skel/

web: newrelic-admin run-program gunicorn -c gunicorn.py.ini wsgi:application
# scheduler: python manage.py celery worker -B -E --maxtasksperchild=1000
# worker: python manage.py celery worker -E --maxtasksperchild=1000

Working Locally

Option 1: ENV Variables and vanilla manage.py - Not recommended

Set DEV environment variables, use standard django runserver command

export DJANGO_DEBUG=True
export DJANGO_ENV=DEV

python manage.py runserver 0.0.0.0:8000

Not preferred because: 1) less dev-prod parity 2a) environment variables set like this aren't confined to the virtual environment 2b) they aren't maintained between virtenv sessions either (unless you set them in virtenv wrapper) 3) linux environment variables set with export aren't shared between shell sessions

Option 2: Heroku Foreman - Preferred

Use .env file for environment variables, foreman spools up processes from the Procfile

cp sample.env .env
foreman start
...
18:29:37 web.1  | started with pid 29000
18:29:38 web.1  | 2012-12-05 23:29:38 [29002] [INFO] Starting gunicorn 0.16.1
18:29:38 web.1  | 2012-12-05 23:29:38 [29002] [INFO] Listening at: http://0.0.0.0:5000 (29002)
18:29:38 web.1  | 2012-12-05 23:29:38 [29002] [INFO] Using worker: sync
18:29:38 web.1  | 2012-12-05 23:29:38 [29005] [INFO] Booting worker with pid: 29005
...

Preferred because creates closet dev-prod parity. Automatically spools up worker nodes if they are declared in the procfile.

However, sometimes you want the runserver. Run management and other commands via foreman -- given the way settings.py switches, this is required for anything that imports settings.py. Of further advantage, Dev and production mental models are identical.

foreman run python manage.py [command]
foreman run python manage.py runserver 0.0.0.0:8000

Option 3: VirtualenvWrapper PostActivate

http://stackoverflow.com/questions/9554087/setting-an-environment-variable-in-virtualenv

Working with VirtEnv

Put in home directory so VirtualBox symlink doesn't have issues

virtualenv --no-site-packages --distribute ~/ve/[project-name]
source ~/ve/[project-name]/bin/activate
deactivate

For future reference, note where the virtenv is stored

touch virtenv_is_[project-name]

Getting a specific python version http://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv

virtualenv -p /usr/bin/python2.6
... or with wrapper ...
mkvirtualenv -p python2.6 env

Working with PIP

Install from requirements.txt

pip install -r requirements.txt

Freezing to reequirements.txt

pip freeze > requirements.txt

Checking for later versions fo packages

pip freeze | cut -d = -f 1 | xargs -n 1 pip search | grep -B2 'LATEST:'

Upgrading a package

pip install --upgrade [package] 

Uninstalling a package

pip uninstall [package]

Setting up databases

Local Postgres DB

For dev/prod parity it is recommended that postgres be used locally as well as remotely

sudo -u postgres createdb [whatever]

Postgres on Heroku

Adding a db to your plan

heroku addons:add heroku-postgresql:dev

Getting info on your dbs

heroku pg:info

Promoting a db to DATABASE_URL

heroku config
...
HEROKU_POSTGRESQL_[color]_URL: postgress://x:[email protected]:XXXX/ABCDEFGHIJ
...
heroku pg:promote HEROKU_POSTGRESQL_[color]_URL
heroku config
...
HEROKU_POSTGRESQL_[color]_URL: postgress://x:[email protected]:XXXX/ABCDEFGHIJ
DATABASE_URL:                  postgress://x:[email protected]:XXXX/ABCDEFGHIJ
...

Setting Environment Variables

Locally Option 1 -- Not recommended

(see working locally above) Setting and Verifying

export DJANGO_ENV=DEV
echo $DJANGO_ENV

Deleting

unset DJANGO_ENV

Locally Option 2 -- recommended

(see working locally above) Use a .env file and foreman

On Heroku (Production and Staging/Testing)

Seeing environment variables on heroku

heroku config

Setting an environment variable on heroku

heroku config:add DJANGO_ENV=PRODUCTION

Deleting an environment variable on heroku

heroku config:remove DJANGO_ENV

Updating database with [South][south]

1. Make changes to your models
2. $ ./manage.py schemamigration [appname] --auto
3. $ ./manage.py migrate [appname]
4. [commit & push changes to heroku]
5. $ heroku run ./manage.py migrate [appname]

Testing

http://www.tdd-django-tutorial.com

#Frontend

Bootstrap

Getting latest version of bootstrap

cd assets/assets
wget http://twitter.github.com/bootstrap/assets/bootstrap.zip && unzip bootstrap && rm bootstrap.zip 


wget  --no-check-certificate https://github.com/addyosmani/jquery-ui-bootstrap/zipball/v0.23 && unzip v0.23
cp -r addyosmani-jquery-ui-bootstrap-cf2a77b/ jquery-ui-bootstrap

Bootstrap resources

• http://bootsnipp.com
• http://fontcustom.com/

Template Setup

In Base.html remember to

• Set Page Title base
• Set addthis key
• Set google analytics variables, uncomment google analytics section

In file directory

• Add favicons to assets/assets/ico

Notes and Niceties

(psycopg2 is a PostgreSQL adapter for Python. mysql-python is a MySQL adapter for Python)

To make .manage.py executable if you don't want to type python manage.py ... all the time.

chmod +x manage.py

Things done on purpose and why

Secret_key not put into environment variable

I think it is more work and more error prone to keep this as an env variable. If this ever gets out of sync on a node the certs/cookies signed by that node will be incompatible with the rest of the system.

#With fallback -- OK
SECRET_KEY = environ.get('SECRET_KEY', SECRET_KEY)
#Without fallback -- Better
SECRET_KEY = environ.get('SECRET_KEY')
#Just not doing it -- Best

Requirements not split into ENV specific files

Some setups create a requirements file for _dev, _test, _prod, _common, and even go as far as to keep these in a separate folder. This is purposefully not done.

The upsides of a single requirements.txt:

1. It keeps closer dev/prod parity
2. If the packages aren't imported in production, they don't take up RAM
3. No extra steps required after pip freeze > requirements.txt
4. When something goes wrong in staging/test environment that works in dev, removes a debugging step

The downsides and risks of a single requirements.txt

1. Extra time for deploy/cold node startup while the server downloads the additional packages
2. Unused packages take server space
3. Risk of mis-configuration silent failure in produciton

Foreman(ruby) instead of Honcho(python)

As of 12/5/12 Honcho is ~Foreman written in python, less used, and a few features behind. As it is a tool, and I haven't yet really had to dive into the guts, ruby seems fine to me for now.

Configuration/settings samples

CACHES = {
  'default': {
    'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
    #'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
    #'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
    'LOCATION': '127.0.0.1:11211',
  }
}    

[Re]Sources

• Part 1: The Basics — South v0.7 documentation
• Getting Started with Django on Heroku/Cedar | Heroku Dev Center
• Deploying Django to Heroku
• PAAS bakeoff
• Rails like environs with django
• Dev Guide to Django on Heroku
• south
• Django-Skel
• Django-prokect-skel
• Django-project-skel
• https://github.com/caktus/django-project-template
• https://gist.github.com/3266518
• http://www.12factor.net
• https://newrelic.com/docs/python/django-on-heroku-quick-start
• https://newrelic.com/docs/python/python-agent-and-heroku
• https://newrelic.com/docs/python/python-agent-and-gunicorn
• https://github.com/chrisfranklin/django-project-skel
• http://kencochrane.net/blog/2011/11/developers-guide-for-running-django-apps-on-heroku/