Skip to content

Commit

Permalink
doc: remove confusing information from the Readme. (#177)
Browse files Browse the repository at this point in the history
  • Loading branch information
@nadiaya
nadiaya authored Jun 12, 2020
1 parent 9d24e47 commit 065c4ff
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 181 deletions.
185 changes: 8 additions & 177 deletions README.rst
View file
Original file line number Diff line number Diff line change
@@ -1,191 +1,22 @@
=========================
SageMaker MXNet Container
=========================
================================
SageMaker MXNet Training Toolkit
================================

SageMaker MXNet Container is an open-source library for making Docker images for using MXNet to train models on Amazon SageMaker.
For serving images, see `SageMaker MXNet Serving Container <https://github.com/aws/sagemaker-mxnet-serving-container>`__.
SageMaker MXNet Training Toolkit is an open-source library for using MXNet to train models on Amazon SageMaker.
For inference, see `SageMaker MXNet Inference Toolkit <https://github.com/aws/sagemaker-mxnet-serving-container>`__.
For the Dockerfiles used for building SageMaker MXNet Containers, see `AWS Deep Learning Containers <https://github.com/aws/deep-learning-containers>`__.
For information on running MXNet jobs on Amazon SageMaker, please refer to the `SageMaker Python SDK documentation <https://github.com/aws/sagemaker-python-sdk>`__.

The information in this README is for MXNet versions 1.4.0 and higher.
For versions 0.12.0-1.3.0 (including serving images), see `the previous version of this README <https://github.com/aws/sagemaker-mxnet-container/blob/4f4492ba71ab5210bb0594449d3996f0bc3e5807/README.rst>`__.

-----------------
Table of Contents
-----------------
.. contents::
:local:

Getting Started
---------------

Prerequisites
~~~~~~~~~~~~~

Make sure you have installed all of the following prerequisites on your development machine:

- `Docker <https://www.docker.com/>`__
- For GPU testing: `nvidia-docker2 <https://github.com/NVIDIA/nvidia-docker>`__

Recommended
^^^^^^^^^^^

- A Python environment management tool (e.g. `PyEnv <https://github.com/pyenv/pyenv>`__,
`VirtualEnv <https://virtualenv.pypa.io/en/stable/>`__)

Building Images
---------------

The Dockerfiles in this repository are intended to be used for building Docker images to run training jobs on `Amazon SageMaker <https://aws.amazon.com/documentation/sagemaker/>`__.

The current master branch of this repository contains Dockerfiles and support code for MXNet versions 1.4.0 and higher.
For MXNet version 1.3.0, check out `v2.0.0 of this repository <https://github.com/aws/sagemaker-mxnet-container/releases/tag/v2.0.0>`__.
For MXNet versions 0.12.1-1.2.1, check out `v1.0.0 of this repository <https://github.com/aws/sagemaker-mxnet-container/releases/tag/v1.0.0>`__.

For each supported MXNet version, Dockerfiles can be found for each processor type (i.e. CPU and GPU).
They install the SageMaker-specific support code found in this repository.

Before building these images, you need to have a pip-installable binary of this repository saved locally.
To create the SageMaker MXNet Container Python package:

::

# Create the binary
git clone https://github.com/aws/sagemaker-mxnet-container.git
cd sagemaker-mxnet-container
python setup.py sdist

Once you have those binaries, you can then build the image.

If you are building images for Python 3 with MXNet 1.6.0, the Dockerfiles don't require any build arguments.
You do need to copy the pip-installable binary from above to ``docker/1.6.0/``.

If you are building images for Python 2 or Python 3 with MXNet 1.4.0 or lower, the Dockerfiles expect two build arguments:

- ``py_version``: the Python version (i.e. 2 or 3).
- ``framework_support_installable``: the pip-installable binary created with the command above

The integration tests expect the Docker images to be tagged as ``preprod-mxnet:<tag>``, where ``<tag>`` looks like <mxnet_version>-<processor>-<python_version> (e.g. 1.6.0-cpu-py3).

Example commands for building images:

::

# All build instructions assume you're starting from the root directory of this repository

# MXNet 1.6.0, Python 3, CPU
$ cp dist/sagemaker_mxnet_training*.tar.gz docker/1.6.0/sagemaker_mxnet_training.tar.gz
$ cp -r docker/artifacts/* docker/1.6.0/py3
$ cd docker/1.6.0/py3
$ docker build -t preprod-mxnet:1.6.0-cpu-py3 -f Dockerfile.cpu .

Don't forget the period at the end of the command!

Running the tests
-----------------

Running the tests requires installation of the SageMaker MXNet Container code and its test dependencies.

::

git clone https://github.com/aws/sagemaker-mxnet-container.git
cd sagemaker-mxnet-container
pip install -e .[test]

Alternatively, instead of pip installing the dependencies yourself, you can use `tox <https://tox.readthedocs.io/en/latest>`__.

Tests are defined in `test/ <https://github.com/aws/sagemaker-mxnet-containers/tree/master/test>`__ and include unit and integration tests.
The integration tests include both running the Docker containers locally and running them on SageMaker.
The tests are compatible with only the Docker images built by Dockerfiles in the current branch.

All test instructions should be run from the top level directory

Unit Tests
~~~~~~~~~~

To run unit tests:

::

tox test/unit

Local Integration Tests
~~~~~~~~~~~~~~~~~~~~~~~

Running local integration tests require `Docker <https://www.docker.com/>`__ and `AWS credentials <https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html>`__,
as the integration tests make calls to a couple AWS services.
Local integration tests on GPU require `nvidia-docker2 <https://github.com/NVIDIA/nvidia-docker>`__.
You Docker image must also be built in order to run the tests against it.

Local integration tests use the following pytest arguments:

- ``docker-base-name``: the Docker image's repository. Defaults to 'preprod-mxnet'.
- ``framework-version``: the MXNet version. Defaults to the latest supported version.
- ``py-version``: the Python version. Defaults to '3'.
- ``processor``: CPU or GPU. Defaults to 'cpu'.
- ``tag``: the Docker image's tag. Defaults to <mxnet_version>-<processor>-py<py-version>

To run local integration tests:

::

tox -- test/integration/local --docker-base-name <your_docker_image> \
--tag <your_docker_image_tag> \
--py-version <2_or_3> \
--framework-version <mxnet_version> \
--processor <cpu_or_gpu>

::

# Example
tox -- test/integration/local --docker-base-name preprod-mxnet \
--tag 1.6.0-cpu-py3 \
--py-version 3 \
--framework-version 1.6.0 \
--processor cpu

SageMaker Integration Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~

SageMaker integration tests require your Docker image to be within an `Amazon ECR repository <https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_Console_Repositories.html>`__.
They also require that you have the setup described under "Integration Tests" at https://github.com/aws/sagemaker-python-sdk#running-tests.

SageMaker integration tests use the following pytest arguments:

- ``docker-base-name``: the Docker image's `ECR repository namespace <https://docs.aws.amazon.com/AmazonECR/latest/userguide/Repositories.html>`__.
- ``framework-version``: the MXNet version. Defaults to the latest supported version.
- ``py-version``: the Python version. Defaults to '3'.
- ``processor``: CPU or GPU. Defaults to 'cpu'.
- ``tag``: the Docker image's tag. Defaults to <mxnet_version>-<processor>-py<py-version>
- ``aws-id``: your AWS account ID.
- ``instance-type``: the specified `Amazon SageMaker Instance Type <https://aws.amazon.com/sagemaker/pricing/instance-types/>`__ that the tests will run on.
Defaults to 'ml.c4.xlarge' for CPU and 'ml.p2.xlarge' for GPU.

To run SageMaker integration tests:

::

tox -- test/integration/sagmaker --aws-id <your_aws_id> \
--docker-base-name <your_docker_image> \
--instance-type <amazon_sagemaker_instance_type> \
--tag <your_docker_image_tag> \

::

# Example
tox -- test/integration/sagemaker --aws-id 12345678910 \
--docker-base-name preprod-mxnet \
--instance-type ml.m4.xlarge \
--tag 1.6.0-cpu-py3

Contributing
------------

Please read `CONTRIBUTING.md <https://github.com/aws/sagemaker-mxnet-containers/blob/master/CONTRIBUTING.md>`__
Please read `CONTRIBUTING.md <https://github.com/aws/sagemaker-mxnet-training-toolkit/blob/master/CONTRIBUTING.md>`__
for details on our code of conduct, and the process for submitting pull requests to us.

License
-------

SageMaker MXNet Containers is licensed under the Apache 2.0 License.
SageMaker MXNet Training Toolkit is licensed under the Apache 2.0 License.
It is copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
The license is available at: http://aws.amazon.com/apache2.0/
6 changes: 2 additions & 4 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,8 @@ def read(fname):
return open(os.path.join(os.path.dirname(__file__), fname)).read()


test_dependencies = ['tox', 'pluggy==0.11', 'flake8', 'pytest==4.5.0', 'pytest-cov', 'pytest-xdist',
'mock', 'sagemaker==1.28.1', 'docker-compose', 'boto3==1.10.32', 'six==1.13.0',
'python-dateutil>=2.1,<2.8.1', 'botocore==1.13.32', 'awscli==1.16.278',
'requests_mock==1.7.0']
test_dependencies = ['botocore', 'boto3', 'docker-compose', 'flake8', 'mock', 'pytest>=4.6.0',
'pytest-cov', 'pytest-xdist', 'sagemaker==1.62.0', 'six', 'tox']

if sys.version_info.major > 2:
test_dependencies.append('sagemaker-experiments==0.1.7')
Expand Down

0 comments on commit 065c4ff

Please sign in to comment.