=====================================
SageMaker TensorFlow Training Toolkit
=====================================
The SageMaker TensorFlow Training Toolkit is an open source library for making the
TensorFlow framework run on `Amazon SageMaker <https://aws.amazon.com/documentation/sagemaker/>`__.
This repository also contains Dockerfiles which install this library, TensorFlow, and dependencies
for building SageMaker TensorFlow images.
For information on running TensorFlow jobs on SageMaker:
- `SageMaker Python SDK documentation <https://sagemaker.readthedocs.io/en/stable/using_tf.html>`__
- `SageMaker Notebook Examples <https://github.com/awslabs/amazon-sagemaker-examples>`__
Table of Contents
-----------------
#. `Getting Started <#getting-started>`__
#. `Building your Image <#building-your-image>`__
#. `Running the tests <#running-the-tests>`__
Getting Started
---------------
Prerequisites
~~~~~~~~~~~~~
Make sure you have installed all of the following prerequisites on your
development machine:
- `Docker <https://www.docker.com/>`__
For Testing on GPU
^^^^^^^^^^^^^^^^^^
- `Nvidia-Docker <https://github.com/NVIDIA/nvidia-docker>`__
Recommended
^^^^^^^^^^^
- A Python environment management tool. (e.g.
`PyEnv <https://github.com/pyenv/pyenv>`__,
`VirtualEnv <https://virtualenv.pypa.io/en/stable/>`__)
Building your Image
-------------------
`Amazon SageMaker <https://aws.amazon.com/documentation/sagemaker/>`__
utilizes Docker containers to run all training jobs & inference endpoints.
The Docker images are built from the Dockerfiles specified in
`docker/ <https://github.com/aws/sagemaker-tensorflow-containers/tree/master/docker>`__.
The Dockerfiles are grouped based on TensorFlow version and separated
based on Python version and processor type.
The Dockerfiles for TensorFlow 2.0+ are available in the
`tf-2 <https://github.com/aws/sagemaker-tensorflow-container/tree/tf-2>`__ branch.
To build the images, first copy the files under
`docker/build_artifacts/ <https://github.com/aws/sagemaker-tensorflow-container/tree/tf-2/docker/build_artifacts>`__
to the folder container the Dockerfile you wish to build.
::
# Example for building a TF 2.1 image with Python 3
cp docker/build_artifacts/* docker/2.1.0/py3/.
After that, go to the directory containing the Dockerfile you wish to build,
and run ``docker build`` to build the image.
::
# Example for building a TF 2.1 image for CPU with Python 3
cd docker/2.1.0/py3
docker build -t tensorflow-training:2.1.0-cpu-py3 -f Dockerfile.cpu .
Don't forget the period at the end of the ``docker build`` command!
Running the tests
-----------------
Running the tests requires installation of the SageMaker TensorFlow Training Toolkit code and its test
dependencies.
::
git clone https://github.com/aws/sagemaker-tensorflow-container.git
cd sagemaker-tensorflow-container
pip install -e .[test]
Tests are defined in
`test/ <https://github.com/aws/sagemaker-tensorflow-container/tree/master/test>`__
and include unit, integration and functional tests.
Unit Tests
~~~~~~~~~~
If you want to run unit tests, then use:
::
# All test instructions should be run from the top level directory
pytest test/unit
Integration Tests
~~~~~~~~~~~~~~~~~
Running integration tests require `Docker <https://www.docker.com/>`__ and `AWS
credentials <https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html>`__,
as the integration tests make calls to a couple AWS services. The integration and functional
tests require configurations specified within their respective
`conftest.py <https://github.com/aws/sagemaker-tensorflow-containers/blob/master/test/integration/conftest.py>`__.Make sure to update the account-id and region at a minimum.
Integration tests on GPU require `Nvidia-Docker <https://github.com/NVIDIA/nvidia-docker>`__.
Before running integration tests:
#. Build your Docker image.
#. Pass in the correct pytest arguments to run tests against your Docker image.
If you want to run local integration tests, then use:
::
# Required arguments for integration tests are found in test/integ/conftest.py
pytest test/integration --docker-base-name <your_docker_image> \
--tag <your_docker_image_tag> \
--framework-version <tensorflow_version> \
--processor <cpu_or_gpu>
::
# Example
pytest test/integration --docker-base-name preprod-tensorflow \
--tag 1.0 \
--framework-version 1.4.1 \
--processor cpu
Functional Tests
~~~~~~~~~~~~~~~~
Functional tests are removed from the current branch, please see them in older branch `r1.0 <https://github.com/aws/sagemaker-tensorflow-container/tree/r1.0#functional-tests>`__.
Contributing
------------
Please read
`CONTRIBUTING.md <https://github.com/aws/sagemaker-tensorflow-containers/blob/master/CONTRIBUTING.md>`__
for details on our code of conduct, and the process for submitting pull
requests to us.
License
-------
SageMaker TensorFlow Containers is licensed under the Apache 2.0 License. It is copyright 2018
Amazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at:
http://aws.amazon.com/apache2.0/
Raw data
{
"_id": null,
"home_page": "https://github.com/aws/sagemaker-tensorflow-training-toolkit",
"name": "sagemaker-tensorflow-training",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Amazon Web Services",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/23/75/1dcac37fe6c757ed02bf1318e2c86c520aa2e7a9d937b2fe5ca61cc3f80f/sagemaker_tensorflow_training-20.4.1.tar.gz",
"platform": null,
"description": "=====================================\nSageMaker TensorFlow Training Toolkit\n=====================================\n\nThe SageMaker TensorFlow Training Toolkit is an open source library for making the\nTensorFlow framework run on `Amazon SageMaker <https://aws.amazon.com/documentation/sagemaker/>`__.\n\nThis repository also contains Dockerfiles which install this library, TensorFlow, and dependencies\nfor building SageMaker TensorFlow images.\n\nFor information on running TensorFlow jobs on SageMaker:\n\n- `SageMaker Python SDK documentation <https://sagemaker.readthedocs.io/en/stable/using_tf.html>`__\n- `SageMaker Notebook Examples <https://github.com/awslabs/amazon-sagemaker-examples>`__\n\nTable of Contents\n-----------------\n\n#. `Getting Started <#getting-started>`__\n#. `Building your Image <#building-your-image>`__\n#. `Running the tests <#running-the-tests>`__\n\nGetting Started\n---------------\n\nPrerequisites\n~~~~~~~~~~~~~\n\nMake sure you have installed all of the following prerequisites on your\ndevelopment machine:\n\n- `Docker <https://www.docker.com/>`__\n\nFor Testing on GPU\n^^^^^^^^^^^^^^^^^^\n\n- `Nvidia-Docker <https://github.com/NVIDIA/nvidia-docker>`__\n\nRecommended\n^^^^^^^^^^^\n\n- A Python environment management tool. (e.g.\n `PyEnv <https://github.com/pyenv/pyenv>`__,\n `VirtualEnv <https://virtualenv.pypa.io/en/stable/>`__)\n\nBuilding your Image\n-------------------\n\n`Amazon SageMaker <https://aws.amazon.com/documentation/sagemaker/>`__\nutilizes Docker containers to run all training jobs & inference endpoints.\n\nThe Docker images are built from the Dockerfiles specified in\n`docker/ <https://github.com/aws/sagemaker-tensorflow-containers/tree/master/docker>`__.\n\nThe Dockerfiles are grouped based on TensorFlow version and separated\nbased on Python version and processor type.\n\nThe Dockerfiles for TensorFlow 2.0+ are available in the\n`tf-2 <https://github.com/aws/sagemaker-tensorflow-container/tree/tf-2>`__ branch.\n\nTo build the images, first copy the files under\n`docker/build_artifacts/ <https://github.com/aws/sagemaker-tensorflow-container/tree/tf-2/docker/build_artifacts>`__\nto the folder container the Dockerfile you wish to build.\n\n::\n\n # Example for building a TF 2.1 image with Python 3\n cp docker/build_artifacts/* docker/2.1.0/py3/.\n\nAfter that, go to the directory containing the Dockerfile you wish to build,\nand run ``docker build`` to build the image.\n\n::\n\n # Example for building a TF 2.1 image for CPU with Python 3\n cd docker/2.1.0/py3\n docker build -t tensorflow-training:2.1.0-cpu-py3 -f Dockerfile.cpu .\n\nDon't forget the period at the end of the ``docker build`` command!\n\nRunning the tests\n-----------------\n\nRunning the tests requires installation of the SageMaker TensorFlow Training Toolkit code and its test\ndependencies.\n\n::\n\n git clone https://github.com/aws/sagemaker-tensorflow-container.git\n cd sagemaker-tensorflow-container\n pip install -e .[test]\n\nTests are defined in\n`test/ <https://github.com/aws/sagemaker-tensorflow-container/tree/master/test>`__\nand include unit, integration and functional tests.\n\nUnit Tests\n~~~~~~~~~~\n\nIf you want to run unit tests, then use:\n\n::\n\n # All test instructions should be run from the top level directory\n pytest test/unit\n\nIntegration Tests\n~~~~~~~~~~~~~~~~~\n\nRunning integration tests require `Docker <https://www.docker.com/>`__ and `AWS\ncredentials <https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html>`__,\nas the integration tests make calls to a couple AWS services. The integration and functional\ntests require configurations specified within their respective\n`conftest.py <https://github.com/aws/sagemaker-tensorflow-containers/blob/master/test/integration/conftest.py>`__.Make sure to update the account-id and region at a minimum.\n\nIntegration tests on GPU require `Nvidia-Docker <https://github.com/NVIDIA/nvidia-docker>`__.\n\nBefore running integration tests:\n\n#. Build your Docker image.\n#. Pass in the correct pytest arguments to run tests against your Docker image.\n\nIf you want to run local integration tests, then use:\n\n::\n\n # Required arguments for integration tests are found in test/integ/conftest.py\n pytest test/integration --docker-base-name <your_docker_image> \\\n --tag <your_docker_image_tag> \\\n --framework-version <tensorflow_version> \\\n --processor <cpu_or_gpu>\n\n::\n\n # Example\n pytest test/integration --docker-base-name preprod-tensorflow \\\n --tag 1.0 \\\n --framework-version 1.4.1 \\\n --processor cpu\n\nFunctional Tests\n~~~~~~~~~~~~~~~~\n\nFunctional tests are removed from the current branch, please see them in older branch `r1.0 <https://github.com/aws/sagemaker-tensorflow-container/tree/r1.0#functional-tests>`__.\n\nContributing\n------------\n\nPlease read\n`CONTRIBUTING.md <https://github.com/aws/sagemaker-tensorflow-containers/blob/master/CONTRIBUTING.md>`__\nfor details on our code of conduct, and the process for submitting pull\nrequests to us.\n\nLicense\n-------\n\nSageMaker TensorFlow Containers is licensed under the Apache 2.0 License. It is copyright 2018\nAmazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at:\nhttp://aws.amazon.com/apache2.0/\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Open source library for using TensorFlow to train models on on Amazon SageMaker.",
"version": "20.4.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "18c626adc574126083d5913dd1019a4a",
"sha256": "d4c089266bc7e66c128c013901649b1315cd6eba61b11061e1e1bde84b8699e1"
},
"downloads": -1,
"filename": "sagemaker_tensorflow_training-20.4.1.tar.gz",
"has_sig": true,
"md5_digest": "18c626adc574126083d5913dd1019a4a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 13323,
"upload_time": "2022-12-06T00:13:35",
"upload_time_iso_8601": "2022-12-06T00:13:35.798266Z",
"url": "https://files.pythonhosted.org/packages/23/75/1dcac37fe6c757ed02bf1318e2c86c520aa2e7a9d937b2fe5ca61cc3f80f/sagemaker_tensorflow_training-20.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-06 00:13:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "aws",
"github_project": "sagemaker-tensorflow-training-toolkit",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "sagemaker-tensorflow-training"
}
JSON
