Skip to content

edX Developer Stack

Jump to bottom Edit New page
Feanil Patel edited this page Mar 18, 2014 · 94 revisions

The edX devstack is a Vagrant instance designed for local development. The instance:

  • Uses the same system requirements as production. This allows developers to discover and fix system configuration issues early in development.
  • Simplifies certain production settings to make development more convenient. For example, it disables nginx and gunicorn in favor of runserver for Django development.

The devstack instance is designed to run code and tests, but you can do most development in the host environment:

  • Git repositories are shared with the host system, so you can use your preferred text editor/IDE.
  • You can load pages served by the running Vagrant instance.

The devstack configuration has the following components:

  • LMS (student facing website)
  • Studio (course authoring)
  • Forums / elasticsearch / ruby (discussion forums)
  • ORA (Open Response Assessor)

For comparison, look at the Production Stack page.

Pre-Requisites

Before installing and using the edX Developer Stack, you should:

Installing the edX Developer Stack

mkdir devstack
cd devstack
curl https://raw.github.com/edx/configuration/master/vagrant/release/devstack/Vagrantfile > Vagrantfile
vagrant up

  • Note: The first time you create the virtual machine (VM) instance, Vagrant will download the base box, which is about 2GB. If you destroy and recreate the VM, Vagrant will re-use the box it downloaded.

  • After the VM is booted up, it mounts the shared folders so they can be accessed by both your host system and the virtual system. When it gets to this point it is asking for your password on the host system.

[default] Exporting NFS shared folders...
Preparing to edit /etc/exports. Administrator privileges will be required...
Password:
  • Once the VM is created, log in to the VM via the command
vagrant ssh
  • OPTIONAL [pertains to unix/linux and mac hosts]: To use "preview" from within Studio, you will need to add a line,
192.168.33.10  preview.localhost

to your development machine's /etc/hosts file. For example:

$ cat /etc/hosts
# Host Database
127.0.0.1   localhost
255.255.255.255 broadcasthost
...
$ # if preview.localhost doesn't already exist:
$ sudo bash -c "echo '192.168.33.10  preview.localhost' >> /etc/hosts"
$ cat /etc/hosts
# Host Database
127.0.0.1   localhost
255.255.255.255 broadcasthost
...
192.168.33.10  preview.localhost
  • OPTIONAL - if you wish to customize the location of the source code that gets cloned during devstack provisioning, or if you prefer to work on source trees that already exist on your system, you can set the VAGRANT_MOUNT_BASE env var, which sets the base dir for the edx-platform and cs_comments_service. By default it will be wherever you run vagrant up. Example: 
$ cd $HOME/my-workspace/my-edx-workspace
$ # This is where you either want vagrant provisioning to clone,
$ #  or you have existing repositories (although note that vagrant provision will overwrite them):
$ ls -F
$ #  You might optionally need to rename edx-ora to ora
cs_comments_service/   edx-platform/   ora/
$ # Now, set the environment for vagrant
$ # (you may want to put this in a file, and 'source' it instead)
$ export VAGRANT_MOUNT_BASE=$PWD

Using the edX devstack

LMS Workflow

Within the VM instance, switch to the edxapp account:

sudo su edxapp

This will source the edxapp environment (/edx/app/edxapp/edxapp_env) so that the venv python, rbenv ruby and rake are in your search path. It will also set the current working directory to the edx-platform repository (/edx/app/edxapp/edx-platform).

If you are running the developer stack with the standard environment file, lms/envs/devstack.py, then you can use this command:

rake devstack[lms]

There is a faster version of this command that starts the server without updating requirements or compiling assets. To ONLY start the server:

rake devstack:start[lms]

Open a browser on your host machine and navigate to http://localhost:8000/ to load the LMS. (Vagrant will forward port 8000 to the LMS server running in the VM.)

The latest version of devstack has the demo course pre-loaded and dummy accounts, you can log in to the website as:

 [email protected] / edx
 [email protected] / edx
 [email protected] /edx
 [email protected] /edx

If you are bringing up LMS with a custom configuration (I'll call it devstack_mod, but you name yours as you require):

  • Update requirements, and compile Sass and CoffeeScript:
rake lms:gather_assets:devstack_mod

Note: rake -T will show other available commands

  • Start your custom configured server:
./manage.py lms runserver --settings=devstack_mod 0.0.0.0:8000

Note: ./manage.py lms -h --settings=devstack_mod will show other available commands on the server.

Studio Workflow

Within the VM instance, switch to the edxapp account:

sudo su edxapp

This will source the edxapp environment (/edx/app/edxapp/edxapp_env) so that the venv python, rbenv ruby and rake are in your search path. It will also set the current working directory to the edx-platform repository (/edx/app/edxapp/edx-platform).

Note: the following steps are for running with the standard configuration file. To run Studio with a custom configuration, follow the instructions as for LMS, above, replacing lms with cms in the commands.

If you are running the developer stack with the standard environment file, cms/envs/devstack.py, then you can use this command:

rake devstack[studio]

There is a faster version of this command that starts the server without updating requirements or compiling assets. To ONLY start the server:

rake devstack:start[studio]

Open a browser on your host machine and navigate to http://localhost:8001/ to load Studio. (Vagrant will forward port 8001 to the Studio server running in the VM.)

  • OPTIONAL: you can see what other commands are available on the server (such as creating users, collecting static files, etc) with:
./manage.py cms -h --settings=devstack

Forum Workflow

  • Within the VM instance, switch to the forum account
sudo su forum
  • Update Ruby requirements (Note that if you get a message for entering a password to install the bundled RubyGems to the system, you can safely ctrl-c out of that. The gems will still be installed correctly for the forum user.)
bundle install
  • Start the server
ruby app.rb -p 18080
  • Access the API at localhost:4567 (Vagrant will forward port 4567 to the Forum server running in the VM.)

Developing on devstack

See Developing on the edX Developer Stack once you are ready to start developing on devstack.

Running LMS/Studio Tests

  • Within the VM instance, switch to the edxapp account
sudo su edxapp
  • Run the Python unit tests
rake test:python
  • Run the JavaScript unit tests
rake test:js
  • Run the LMS and Studio acceptance tests
rake test:acceptance

See the docs on testing edx-platform for detailed information about writing and running tests.

Updating Application Versions

Use vagrant provision to update the configuration and all repositories to the master branch. NOTE: this will perform a git clean on your repositories, so make sure you have committed any changes you want to save.

Logs

Logs for the running applications can be found in various application specific folders in /edx/var/log.

Issues / Workarounds

  • If the /edx/app/edx/edx-platform directory in the Vagrant image is empty, it means the provisioning scripts did not complete successfully. Try re-running the provisioning scripts using vagrant provision.

  • If vagrant up shows directories not existing, such as:

/tmp/vagrant-shell: line 1: /edx/app/edx_ansible/venvs/edx_ansible/bin/activate: No such file or directory

then there is possibly something not quite up to date with the base vagrant box (which is assumed to be properly configured, and is provided for you). You can always try to build a current box yourself from the configuration/vagrant/base/devstack directory. This will take longer, and require more setup on your part (such as checking out all the appropriate repositories in advance).

  • If you see this message:
    !! ERROR !!

    The last command did not complete successfully,
    For more details or trying running the
    script again with the -v flag.

check that you are running Vagrant commands in the directory containing the devstack Vagrantfile, NOT from the edx-platform directory. There is an older Vagrantfile in edx-platform that will eventually be deprecated in favor of devstack.

  • If you see this error when running a manage.py command:
[Errno 2] No such file or directory: u'/edx/app/edxapp/log/edx.log'

you need to add --settings=devstack to the command. (The default settings assume a legacy directory structure. Eventually, we may change the default settings to devstack.)

  • If you see this error when attempting to run the server:
pymongo.errors.ConnectionFailure: could not connect to localhost:27017: [Errno 111] Connection refused

Then, from a vagrant user shell, you should sudo rm /edx/var/mongo/mongodb/mongod.lock, and then sudo start mongodb. (You get this error if the Vagrant instance does not shut down properly. Use vagrant halt to safely stop the Vagrant instance.)

  • If you see a database error something like this when running the server:
DatabaseError: (1146, "Table 'edxapp.dark_lang_darklangconfig' doesn't exist")

Then this means that your database is outdated and needs to be migrated. From a vagrant user shell, you should ./manage.py lms syncdb --migrate --settings=devstack, and then do the same thing for Studio with ./manage.py cms syncdb --migrate --settings=devstack.

  • Ubuntu: if you see this message:
* It appears your machine doesn't support NFS, or there is not an
adapter to enable NFS on this machine for Vagrant. Please verify
that `nfsd` is installed on your machine, and try again. If you're
on Windows, NFS isn't supported. If the problem persists, please
contact Vagrant support.

you need to install nfs using this command:

sudo apt-get install nfs-common nfs-kernel-server

  • If you get an error such as NS_ERROR_FAILURE, try upgrading VirtualBox (4.3.2 seems to solve this on OS/X Mavericks).

  • Ubuntu: if you see this message:

exportfs: Warning: ... does not support NFS export.

If you are working on an Ubuntu workstation and your home directory is encrypted you will likely run into NFS problems. To work around the issue set VAGRANT_MOUNT_BASE to a dir outside the encrypted volume.

  • Mac OS X: Sometimes, NFS fails the first time you start Vagrant on Mac OS X. If you see this message:
[default] Mounting NFS shared folders...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
mount -o 'vers=3,udp' 192.168.33.1:'/path/to/devstack/edx-platform' /edx/app/edxapp/edx-platform
Stdout from the command:
Stderr from the command:
stdin: is not a tty
mount.nfs: requested NFS version or transport protocol is not supported

then reload and re-provision the instance using these commands:

vagrant reload
vagrant provision
  • Signing up a new user (activation message problems):

When signing up as a new user, this message is shown in the browser:

 We need to verify your email address

 Almost there! In order to complete your sign up we need you to verify your email address. 
 An activation message and next steps should be waiting for you there.

However, when running the stack locally, there is no email message sent. Rather the message comes out on the console, giving the activation key to use:

 Thank you for signing up for edX Studio! To activate your account, please copy and paste this address 
 into your web browser's address bar:

        http://localhost/activate/0699e17df6024fb382fadfabed100691

 If you didn't request this, you don't need to do anything; you won't receive any more email from us. Please do 
 not reply to this e-mail; if you require assistance, check the help section of the edX web site.

Using the domain address ‘http://localhost/activate/… may not work when you copy this address into the web browser’s address field as instructed. if a ’Server not found’ message comes up in the browser, substitute

 http://127.0.0.1:8001

for localhost. For example:

 http://localhost/activate/a19de8c6330f4213ac3313c147297990

becomes

 http://127.0.0.1:8001/activate/a19de8c6330f4213ac3313c147297990
  • Rake: No such file or directory: If you see this error message in the console:
bash: /edx/app/edxapp/edx-platform/bin/rake: No such file or directory

then you probably used git clean, which deletes "binstubs" required to run Rake. To re-create them, run:

bundle install --binstubs

from within the edx-platform directory.

  • Database migrations: If you update the platform code, then when using the application you get an error message in your browser like this:
Exception Type:   DatabaseError
Exception Value:  

(1146, "Table 'edxapp.new_foo' doesn't exist")

Exception Location:   /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/MySQLdb/connections.py in defaulterrorhandler, line 36

there have been database migrations that you need to perform. Note this will not destroy your data. You can use the following rake task:

rake resetdb[devstack]
Add a custom footer
Add a custom sidebar
Clone this wiki locally