Dev_Set up Azure data storage

In order to test uploading and running bundles in CodaLab, you will need to have a Windows Azure storage account and Azure Service Bus. Once you have set up your Azure account, log on to the Azure Portal and follow the steps in this section.

Create storage containers.

1. Log on to the Azure Portal.
2. In the left pane, click Storage.
3. Select your storage account.
4. At the bottom of the dashboard, click Manage Access Keys. Copy your access keys, you'll need them later.
5. At the top of the dashboard page, click Containers.
6. At the bottom of the Containers page click Add.
7. Create a new container named "bundles". Set the Access to "Private".
8. Add another container named "public". Set the Access to "Public Blob".

Add a service bus namespace

1. Install azure-cli
2. From azure-cli sb namespace create <name> <location> where <location> can be "West US"
3. Log on to the Azure Portal.
4. In the left pane, click Service Bus.
5. Select the service bus you just created.
6. At the top of the screen click Queues.
7. Click Create a new queue.
8. Click Quick Create and create a new queue named "compute".
9. Click Create A New Queue.
10. At the bottom of the screen, click New, and create another queue named "response".
11. In the left pane, click Service Bus.
12. At the bottom of the page, click Connection Information.
13. Copy the following connection information:
    • Namespace name
    • Default issuer
    • Default key or
    • Shared Access Key Name
    • Shared Access Key Value
    • Default key

Create a local configuration file

1. Open your local configuration file (local.py). If there is no local.py, save a copy of local_sample.py named local.py in the same directory.

2. In the Azure storage section, enter your Azure account details:

   DEFAULT_FILE_STORAGE = 'codalab.azure_storage.AzureStorage'
   AZURE_ACCOUNT_NAME = "<enter name>"
   AZURE_ACCOUNT_KEY = '<enter key>'
   AZURE_CONTAINER = '<public container name>'
   
   BUNDLE_AZURE_CONTAINER = "<private container name>"
   BUNDLE_AZURE_ACCOUNT_NAME = AZURE_ACCOUNT_NAME
   BUNDLE_AZURE_ACCOUNT_KEY = AZURE_ACCOUNT_KEY
   

3. In the Service Bus section, enter your service bus connection information:

   SBS_NAMESPACE = '<enter the name of your service bus>'
   SBS_ISSUER = 'owner'
   SBS_ACCOUNT_KEY = '<enter value for 'default key'>'
   SBS_SHARED_ACCESS_KEY_NAME = '<shared access key name>' Note:'RootManageSharedAccessKey' is usually the name 
   SBS_SHARED_ACCESS_KEY_VALUE = '<shared access key value>'
   

Important: Do not change the values for DEFAULT_FILE_STORAGE and PRIVATE_FILE_STORAGE, as these parameters contain the name of the Python class which implements the Azure storage back-end for Django.

Create a local compute configuration file

1. Open codalab/codalab/codalabtools/compute/sample.config.

2. Save a copy of sample.config named .codalabconfig in the same directory.

3. Open .codalabconfig.

4. In the compute-worker section, enter the configuration settings for the storage account and the compute queue.

   compute-worker:
       azure-storage:
           account-name: "your account name"
           account-key: "your account key"
       azure-service-bus:
           namespace: "your namespace"
           key: "your secret key"
           shared-access-key-name: "shared access key name" Usually RootManageSharedAccessKey is the name 
           shared-access-key-value: "shared access key value"
           issuer: "owner"
           listen-to: "compute"
       local-root: "D:\\Temp"
   

Enable CORS (Cross-origin resource sharing) on Blob Storage

In order to work with competitions and bundles in your local development environment, you will need to manually enable Cross-origin resource sharing (CORS). CORS is a mechanism that allows many resources on a web page to be requested from another domain outside the domain the resource originated from. Web Browsers commonly apply same origin restriction policy to network requests. CORS relaxes these restrictions allowing domains to give each other permissions for accessing each other's resources.

• Click on your storage account and add the following under CORS section:
• allowed origin = *
• methods = PUT
• headers = *
• max age = 1800 seconds

Note: these are policies for a dev env.