Skip to content

edX Developer Stack

Jump to bottom Edit New page
Jim Abramson edited this page Feb 3, 2014 · 94 revisions

The edX devstack is a Vagrant instance designed for local development. The instance:

  • Uses the same system requirements as production. This allows developers to discover and fix system configuration issues early in development.
  • Simplifies certain production settings to make development more convenient. For example, it disables nginx and gunicorn in favor of runserver for Django development.

The devstack instance is designed to run code and tests, but you can do most development in the host environment:

  • Git repositories are shared with the host system, so you can use your preferred text editor/IDE.
  • You can load pages served by the running Vagrant instance.

The devstack configuration has the following components:

  • LMS (student facing website)
  • Studio (course authoring)
  • Forums / elasticsearch / ruby (discussion forums)
  • ORA (Open Response Assessor)

Pre-Requisites

Before installing and using the edX Developer Stack, you should:

Installing the edX Developer Stack

mkdir devstack
cd devstack
curl https://raw.github.com/edx/configuration/master/vagrant/release/devstack/Vagrantfile > Vagrantfile
vagrant up

  • Note: The first time you create the virtual machine (VM) instance, Vagrant will download the base box, which is about 2GB. If you destroy and recreate the VM, Vagrant will re-use the box it downloaded.

  • After the VM is booted up, it mounts the shared folders so they can be accessed by both your host system and the virtual system. When it gets to this point it is asking for your password on the host system.

[default] Exporting NFS shared folders...
Preparing to edit /etc/exports. Administrator privileges will be required...
Password:
  • Once the VM is created, login to the VM via the command
vagrant ssh
  • OPTIONAL [pertains to unix/linux and mac hosts]: To use "preview" from within Studio, you will need to add a line,
192.168.33.10  preview.localhost

to your development machine's /etc/hosts file. For example:

$ cat /etc/hosts
# Host Database
127.0.0.1   localhost
255.255.255.255 broadcasthost
...
$ # if preview.localhost doesn't already exist:
$ sudo bash -c "echo '192.168.33.10  preview.localhost' >> /etc/hosts"
$ cat /etc/hosts
# Host Database
127.0.0.1   localhost
255.255.255.255 broadcasthost
...
192.168.33.10  preview.localhost
  • OPTIONAL - if you wish to customize the location of the source code that gets cloned during devstack provisioning, or if you prefer to work on source trees that already exist on your system, you can set the VAGRANT_MOUNT_BASE env var, which sets the base dir for the edx-platform and cs_comments_service. By default it will be wherever you run vagrant up. Example: 
$ cd $HOME/my-workspace/my-edx-workspace
$ # This is where you have previously 'git cloned' these repositories:
$ ls -F
$ #  You might optionally need to rename edx-ora to ora
cs_comments_service/   edx-platform/   ora/
$ # Now, set the environment for vagrant
$ # (you may want to put this in a file, and 'source' it instead)
$ export VAGRANT_MOUNT_BASE=$PWD

Using the edX devstack

LMS Workflow

  • Within the VM instance, switch to the edxapp account:
sudo su edxapp

This will source the edxapp environment (/edx/app/edxapp/edxapp_env) so that the venv python, rbenv ruby and rake are in your search path. It will also set the current working directory to the edx-platform repository (/edx/app/edxapp/edx-platform).

  • If you are running the developer stack with the standard environment file, lms/envs/devstack.py, then you can use these single commands:

    • Update Python/Ruby requirements, compile Sass/CoffeeScript, and start the server:
rake devstack[lms]
  • Alternately, to start the server without updating requirements or compiling assets (faster):
rake devstack:start[lms]

**NOTE**: The above commands were added to edx-platform on January 7, 2014.  If you see this error message:

rake aborted!
Don't know how to build task 'devstack'

then you need to update your branch with the changes in origin/master.
*Note*, however, that if you are building with any environment other than `devstack`, you
will need to use the commands below.
  • The latest version of devstack has the demo course pre-loaded and dummy accounts, you can login to the website as:
 [email protected] / edx
 [email protected] / edx
 [email protected] /edx
 [email protected] /edx

  • If you are bringing up LMS with your custom configuration (I'll call it devstack_mod, but you name yours as you require):

    • On first server bringup, or when Python or Ruby requirements change:
pip install -r requirements/edx/base.txt
bundle install
  • Compile Sass and CoffeeScript:
rake assets[lms,devstack_mod]

Note: rake -T will show other available commands

  • Start your custom configured server:
./manage.py lms runserver --settings=devstack_mod 0.0.0.0:8000

Note: ./manage.py lms -h --settings=devstack_mod will show other available commands on the server.

  • Open a browser on your host machine and navigate to http://localhost:8000/ to load the LMS. (Vagrant will forward port 8000 to the LMS server running in the VM.)

Studio Workflow

  • Within the VM instance, switch to the edxapp account:
sudo su edxapp

This will source the edxapp environment (/edx/app/edxapp/edxapp_env) so that the venv python, rbenv ruby and rake are in your search path. It will also set the current working directory to the edx-platform repository (/edx/app/edxapp/edx-platform).

Note: the following steps are for running with the standard configuration file. To run Studio with a custom configuration, follow the instructions as for LMS, above, replacing lms with cms in the commands.

  • Update Python/Ruby requirements, compile Sass/CoffeeScript, and start the server:
rake devstack[studio]

There is a faster version of this command that starts the server without updating requirements or compiling assets.   To ONLY start the server:

rake devstack:start[studio]

**NOTE**: The above commands were added to edx-platform on January 7, 2014.  If you see this error message:

rake aborted!
Don't know how to build task 'devstack'

then you need to update your branch with the changes in origin/master.

  • Open a browser on your host machine and navigate to http://localhost:8001/ to load Studio. (Vagrant will forward port 800 to the Studio server running in the VM.)

  • OPTIONAL: you can see what other commands are available on the server (such as creating users, collecting static files, etc) with:

./manage.py cms -h --settings=devstack

Forum Workflow

  • Within the VM instance, switch to the forum account
sudo su forum
  • Update Ruby requirements (Note that if you get a message for entering a password to install the bundled RubyGems to the system, you can safely ctrl-c out of that. The gems will still be installed correctly for the forum user.)
bundle install
  • Start the server
ruby app.rb -p 18080
  • Access the API at localhost:4567 (Vagrant will forward port 4567 to the Forum server running in the VM.)

Running LMS/Studio Tests

  • Within the VM instance, switch to the edxapp account
sudo su edxapp
  • Run the Python unit tests
rake test:python
  • Run the JavaScript unit tests
rake test:js
  • Run the LMS and Studio acceptance tests
rake test:acceptance

See the docs on testing edx-platform for detailed information about writing and running tests.

Updating Application Versions

Use vagrant provision to update the configuration and all repositories to the master branch. NOTE: this will perform a git clean on your repositories, so make sure you have committed any changes you want to save.

Issues / Workarounds

  • If you are using empanada and get a message like this when running vagrant up for the first time, then run vagrant provision. This will reprovision the virtual machine with the latest configuration repo contents, instead of what was on the virtual server when the empanada image was cut.
TASK: [local_dev | create login scripts] ************************************** 
fatal: [localhost] => input file not found at /edx/app/edx_ansible/edx_ansible/playbooks/roles/local_dev/templates/edxapp_bashrc.j2 or /edx/app/edx_ansible/edx_ansible/playbooks/edxapp_bashrc.j2

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/root/vagrant-devstack.retry

localhost                  : ok=172  changed=64   unreachable=1    failed=0

  • If the /edx/app/edx/edx-platform directory in the Vagrant image is empty, it means the provisioning scripts did not complete successfully. Try re-running the provisioning scripts using vagrant provision.

  • If vagrant up shows directories not existing, such as:

/tmp/vagrant-shell: line 1: /edx/app/edx_ansible/venvs/edx_ansible/bin/activate: No such file or directory

then there is possibly something not quite up to date with the base vagrant box (which is assumed to be properly configured, and is provided for you). You can always try to build a current box yourself from the configuration/vagrant/base/devstack directory. This will take longer, and require more setup on your part (such as checking out all the appropriate repositories in advance).

  • If you see this message:
    !! ERROR !!

    The last command did not complete successfully,
    For more details or trying running the
    script again with the -v flag.

check that you are running Vagrant commands in the directory containing the devstack Vagrantfile, NOT from the edx-platform directory. There is an older Vagrantfile in edx-platform that will eventually be deprecated in favor of devstack.

  • If you see this error when running a manage.py command:
[Errno 2] No such file or directory: u'/edx/app/edxapp/log/edx.log'

you need to add --settings=devstack to the command. (The default settings assume a legacy directory structure. Eventually, we may change the default settings to devstack.)

  • If you see this error when attempting to run the server:
pymongo.errors.ConnectionFailure: could not connect to localhost:27017: [Errno 111] Connection refused

Then, from a vagrant user shell, you should sudo rm /edx/var/mongo/mongodb/mongod.lock, and then sudo start mongodb. (You get this error if the Vagrant instance does not shut down properly. Use vagrant halt to safely stop the Vagrant instance.)

  • If you see a database error something like this when running the server:
DatabaseError: (1146, "Table 'edxapp.dark_lang_darklangconfig' doesn't exist")

Then this means that your database is outdated and needs to be migrated. From a vagrant user shell, you should ./manage.py lms syncdb --migrate --settings=devstack, and then do the same thing for Studio with ./manage.py cms syncdb --migrate --settings=devstack.

  • Ubuntu: if you see this message:
* It appears your machine doesn't support NFS, or there is not an
adapter to enable NFS on this machine for Vagrant. Please verify
that `nfsd` is installed on your machine, and try again. If you're
on Windows, NFS isn't supported. If the problem persists, please
contact Vagrant support.

you need to install nfs using this command:

sudo apt-get install nfs-common nfs-kernel-server

  • If you get an error such as NS_ERROR_FAILURE, try upgrading VirtualBox (4.3.2 seems to solve this on OS/X Mavericks).

  • Ubuntu: if you see this message:

exportfs: Warning: ... does not support NFS export.

If you are working on an Ubuntu workstation and your home directory is encrypted you will likely run into NFS problems. To work around the issue set VAGRANT_MOUNT_BASE to a dir outside the encrypted volume.

  • Mac OS X: Sometimes, NFS fails the first time you start Vagrant on Mac OS X. If you see this message:
[default] Mounting NFS shared folders...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
mount -o 'vers=3,udp' 192.168.33.1:'/path/to/devstack/edx-platform' /edx/app/edxapp/edx-platform
Stdout from the command:
Stderr from the command:
stdin: is not a tty
mount.nfs: requested NFS version or transport protocol is not supported

then reload and re-provision the instance using these commands:

vagrant reload
vagrant provision
  • Signing up a new user (activation message problems):

When signing up as a new user, this message is shown in the browser:

 We need to verify your email address

 Almost there! In order to complete your sign up we need you to verify your email address. 
 An activation message and next steps should be waiting for you there.

However, when running the stack locally, there is no email message sent. Rather the message comes out on the console, giving the activation key to use:

 Thank you for signing up for edX Studio! To activate your account, please copy and paste this address 
 into your web browser's address bar:

        http://localhost/activate/0699e17df6024fb382fadfabed100691

 If you didn't request this, you don't need to do anything; you won't receive any more email from us. Please do 
 not reply to this e-mail; if you require assistance, check the help section of the edX web site.

Using the domain address ‘http://localhost/activate/… may not work when you copy this address into the web browser’s address field as instructed. if a ’Server not found’ message comes up in the browser, substitute

 http://127.0.0.1:8001

for localhost. For example:

 http://localhost/activate/a19de8c6330f4213ac3313c147297990

becomes

 http://127.0.0.1:8001/activate/a19de8c6330f4213ac3313c147297990
Add a custom footer
Add a custom sidebar
Clone this wiki locally