diff --git a/README.rst b/README.rst index 18caf24c..b1e48d0a 100644 --- a/README.rst +++ b/README.rst @@ -1,191 +1,22 @@ -========================= -SageMaker MXNet Container -========================= +================================ +SageMaker MXNet Training Toolkit +================================ -SageMaker MXNet Container is an open-source library for making Docker images for using MXNet to train models on Amazon SageMaker. -For serving images, see `SageMaker MXNet Serving Container `__. +SageMaker MXNet Training Toolkit is an open-source library for using MXNet to train models on Amazon SageMaker. +For inference, see `SageMaker MXNet Inference Toolkit `__. +For the Dockerfiles used for building SageMaker MXNet Containers, see `AWS Deep Learning Containers `__. For information on running MXNet jobs on Amazon SageMaker, please refer to the `SageMaker Python SDK documentation `__. -The information in this README is for MXNet versions 1.4.0 and higher. -For versions 0.12.0-1.3.0 (including serving images), see `the previous version of this README `__. - ------------------ -Table of Contents ------------------ -.. contents:: - :local: - -Getting Started ---------------- - -Prerequisites -~~~~~~~~~~~~~ - -Make sure you have installed all of the following prerequisites on your development machine: - -- `Docker `__ -- For GPU testing: `nvidia-docker2 `__ - -Recommended -^^^^^^^^^^^ - -- A Python environment management tool (e.g. `PyEnv `__, - `VirtualEnv `__) - -Building Images ---------------- - -The Dockerfiles in this repository are intended to be used for building Docker images to run training jobs on `Amazon SageMaker `__. - -The current master branch of this repository contains Dockerfiles and support code for MXNet versions 1.4.0 and higher. -For MXNet version 1.3.0, check out `v2.0.0 of this repository `__. -For MXNet versions 0.12.1-1.2.1, check out `v1.0.0 of this repository `__. - -For each supported MXNet version, Dockerfiles can be found for each processor type (i.e. CPU and GPU). -They install the SageMaker-specific support code found in this repository. - -Before building these images, you need to have a pip-installable binary of this repository saved locally. -To create the SageMaker MXNet Container Python package: - -:: - - # Create the binary - git clone https://github.com/aws/sagemaker-mxnet-container.git - cd sagemaker-mxnet-container - python setup.py sdist - -Once you have those binaries, you can then build the image. - -If you are building images for Python 3 with MXNet 1.6.0, the Dockerfiles don't require any build arguments. -You do need to copy the pip-installable binary from above to ``docker/1.6.0/``. - -If you are building images for Python 2 or Python 3 with MXNet 1.4.0 or lower, the Dockerfiles expect two build arguments: - -- ``py_version``: the Python version (i.e. 2 or 3). -- ``framework_support_installable``: the pip-installable binary created with the command above - -The integration tests expect the Docker images to be tagged as ``preprod-mxnet:``, where ```` looks like -- (e.g. 1.6.0-cpu-py3). - -Example commands for building images: - -:: - - # All build instructions assume you're starting from the root directory of this repository - - # MXNet 1.6.0, Python 3, CPU - $ cp dist/sagemaker_mxnet_training*.tar.gz docker/1.6.0/sagemaker_mxnet_training.tar.gz - $ cp -r docker/artifacts/* docker/1.6.0/py3 - $ cd docker/1.6.0/py3 - $ docker build -t preprod-mxnet:1.6.0-cpu-py3 -f Dockerfile.cpu . - -Don't forget the period at the end of the command! - -Running the tests ------------------ - -Running the tests requires installation of the SageMaker MXNet Container code and its test dependencies. - -:: - - git clone https://github.com/aws/sagemaker-mxnet-container.git - cd sagemaker-mxnet-container - pip install -e .[test] - -Alternatively, instead of pip installing the dependencies yourself, you can use `tox `__. - -Tests are defined in `test/ `__ and include unit and integration tests. -The integration tests include both running the Docker containers locally and running them on SageMaker. -The tests are compatible with only the Docker images built by Dockerfiles in the current branch. - -All test instructions should be run from the top level directory - -Unit Tests -~~~~~~~~~~ - -To run unit tests: - -:: - - tox test/unit - -Local Integration Tests -~~~~~~~~~~~~~~~~~~~~~~~ - -Running local integration tests require `Docker `__ and `AWS credentials `__, -as the integration tests make calls to a couple AWS services. -Local integration tests on GPU require `nvidia-docker2 `__. -You Docker image must also be built in order to run the tests against it. - -Local integration tests use the following pytest arguments: - -- ``docker-base-name``: the Docker image's repository. Defaults to 'preprod-mxnet'. -- ``framework-version``: the MXNet version. Defaults to the latest supported version. -- ``py-version``: the Python version. Defaults to '3'. -- ``processor``: CPU or GPU. Defaults to 'cpu'. -- ``tag``: the Docker image's tag. Defaults to --py - -To run local integration tests: - -:: - - tox -- test/integration/local --docker-base-name \ - --tag \ - --py-version <2_or_3> \ - --framework-version \ - --processor - -:: - - # Example - tox -- test/integration/local --docker-base-name preprod-mxnet \ - --tag 1.6.0-cpu-py3 \ - --py-version 3 \ - --framework-version 1.6.0 \ - --processor cpu - -SageMaker Integration Tests -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -SageMaker integration tests require your Docker image to be within an `Amazon ECR repository `__. -They also require that you have the setup described under "Integration Tests" at https://github.com/aws/sagemaker-python-sdk#running-tests. - -SageMaker integration tests use the following pytest arguments: - -- ``docker-base-name``: the Docker image's `ECR repository namespace `__. -- ``framework-version``: the MXNet version. Defaults to the latest supported version. -- ``py-version``: the Python version. Defaults to '3'. -- ``processor``: CPU or GPU. Defaults to 'cpu'. -- ``tag``: the Docker image's tag. Defaults to --py -- ``aws-id``: your AWS account ID. -- ``instance-type``: the specified `Amazon SageMaker Instance Type `__ that the tests will run on. - Defaults to 'ml.c4.xlarge' for CPU and 'ml.p2.xlarge' for GPU. - -To run SageMaker integration tests: - -:: - - tox -- test/integration/sagmaker --aws-id \ - --docker-base-name \ - --instance-type \ - --tag \ - -:: - - # Example - tox -- test/integration/sagemaker --aws-id 12345678910 \ - --docker-base-name preprod-mxnet \ - --instance-type ml.m4.xlarge \ - --tag 1.6.0-cpu-py3 Contributing ------------ -Please read `CONTRIBUTING.md `__ +Please read `CONTRIBUTING.md `__ for details on our code of conduct, and the process for submitting pull requests to us. License ------- -SageMaker MXNet Containers is licensed under the Apache 2.0 License. +SageMaker MXNet Training Toolkit is licensed under the Apache 2.0 License. It is copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at: http://aws.amazon.com/apache2.0/ diff --git a/setup.py b/setup.py index 6c018893..e72139b2 100644 --- a/setup.py +++ b/setup.py @@ -31,10 +31,8 @@ def read(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() -test_dependencies = ['tox', 'pluggy==0.11', 'flake8', 'pytest==4.5.0', 'pytest-cov', 'pytest-xdist', - 'mock', 'sagemaker==1.28.1', 'docker-compose', 'boto3==1.10.32', 'six==1.13.0', - 'python-dateutil>=2.1,<2.8.1', 'botocore==1.13.32', 'awscli==1.16.278', - 'requests_mock==1.7.0'] +test_dependencies = ['botocore', 'boto3', 'docker-compose', 'flake8', 'mock', 'pytest>=4.6.0', + 'pytest-cov', 'pytest-xdist', 'sagemaker==1.62.0', 'six', 'tox'] if sys.version_info.major > 2: test_dependencies.append('sagemaker-experiments==0.1.7')