.. image:: https://github.com/aws/sagemaker-experiments/raw/main/branding/icon/sagemaker-banner.png
:height: 100px
:alt: SageMaker
:target: https://aws.amazon.com/sagemaker/
**NOTE:** Use the SageMaker `SDK <https://sagemaker.readthedocs.io/en/v2.125.0/experiments/sagemaker.experiments.html>`_ to use SageMaker Experiments. This repository will not be up to date with the latest product improvements. Link to `developer guide <https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html>`_.
================================
SageMaker Experiments Python SDK
================================
.. image:: https://img.shields.io/pypi/v/sagemaker-experiments.svg
:target: https://pypi.python.org/pypi/sagemaker-experiments
:alt: Latest Version
.. image:: https://img.shields.io/pypi/pyversions/sagemaker-experiments.svg
:target: https://pypi.python.org/pypi/sagemaker-experiments
:alt: Supported Python Versions
.. image:: https://img.shields.io/pypi/l/sagemaker-experiments
:target: https://github.com/aws/sagemaker-experiments/blob/main/LICENSE
:alt: License
.. image:: https://img.shields.io/pypi/dm/sagemaker-experiments
:target: https://pypi.python.org/pypi/sagemaker-experiments
:alt: PyPI - Downloads
.. image:: https://codecov.io/gh/aws/sagemaker-experiments/branch/main/graph/badge.svg
:target: https://codecov.io/gh/aws/sagemaker-experiments
:alt: CodeCov
.. image:: https://img.shields.io/pypi/status/sagemaker-experiments
:target: https://github.com/aws/sagemaker-experiments
:alt: PyPI - Status
.. image:: https://img.shields.io/pypi/format/coverage.svg
:target: https://pypi.org/project/coverage/
:alt: Kit format
.. image:: https://img.shields.io/github/workflow/status/aws/sagemaker-experiments/Main
:target: https://github.com/aws/sagemaker-experiments/actions
:alt: GitHub Workflow Status
.. image:: https://img.shields.io/github/stars/aws/sagemaker-experiments.svg?logo=github
:target: https://github.com/aws/sagemaker-experiments/stargazers
:alt: Github stars
.. image:: https://img.shields.io/github/forks/aws/sagemaker-experiments.svg?logo=github
:target: https://github.com/aws/sagemaker-experiments/network/members
:alt: Github forks
.. image:: https://img.shields.io/github/contributors/aws/sagemaker-experiments.svg?logo=github
:target: https://github.com/aws/sagemaker-experiments/graphs/contributors
:alt: Contributors
.. image:: https://img.shields.io/github/search/aws/sagemaker-experiments/sagemaker
:target: https://github.com/aws/sagemaker-experiments
:alt: GitHub search hit counter
.. image:: https://img.shields.io/badge/code_style-black-000000.svg
:target: https://github.com/python/black
:alt: Code style: black
.. image:: https://readthedocs.org/projects/sagemaker-experiments/badge/?version=latest
:target: https://readthedocs.org/projects/sagemaker-experiments/
:alt: Read the Docs - Sagemaker Experiments
.. image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gh/aws/amazon-sagemaker-examples/main?filepath=sagemaker-experiments%2Fmnist-handwritten-digits-classification-experiment.ipynb
Experiment tracking in SageMaker Training Jobs, Processing Jobs, and Notebooks.
Overview
--------
SageMaker Experiments is an AWS service for tracking machine learning Experiments. The SageMaker Experiments Python SDK is a high-level interface to this service that helps you track Experiment information using Python.
Experiment tracking powers the machine learning integrated development environment `Amazon SageMaker Studio <https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio.html>`_.
For detailed API reference please go to: `Read the Docs <https://sagemaker-experiments.readthedocs.io>`_
Concepts
--------
- **Experiment**: A collection of related Trials. Add Trials to an Experiment that you wish to compare together.
- **Trial**: A description of a multi-step machine learning workflow. Each step in the workflow is described by a Trial Component. There is no relationship between Trial Components such as ordering.
- **Trial Component**: A description of a single step in a machine learning workflow. For example data cleaning, feature extraction, model training, model evaluation, etc...
- **Tracker**: A Python context-manager for logging information about a single TrialComponent.
For more information see `Amazon SageMaker Experiments - Organize, Track, and Compare Your Machine Learning Trainings <https://aws.amazon.com/blogs/aws/amazon-sagemaker-experiments-organize-track-and-compare-your-machine-learning-trainings/>`_
Using the SDK
-------------
You can use this SDK to:
- Manage Experiments, Trials, and Trial Components within Python scripts, programs, and notebooks.
- Add tracking information to a SageMaker notebook, allowing you to model your notebook in SageMaker Experiments as a multi-step ML workflow.
- Record experiment information from inside your running SageMaker Training and Processing Jobs.
Installation
------------
.. code-block:: bash
pip install sagemaker-experiments
Examples
--------
.. code-block:: python
import boto3
import pickle, gzip, numpy, json, os
import io
import numpy as np
import sagemaker.amazon.common as smac
import sagemaker
from sagemaker import get_execution_role
from sagemaker import analytics
from smexperiments import experiment
# Specify training container
from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(boto3.Session().region_name, 'linear-learner')
# Load the dataset
s3 = boto3.client("s3")
s3.download_file("sagemaker-sample-files", "datasets/image/MNIST/mnist.pkl.gz", "mnist.pkl.gz")
with gzip.open('mnist.pkl.gz', 'rb') as f:
train_set, valid_set, test_set = pickle.load(f, encoding='latin1')
vectors = np.array([t.tolist() for t in train_set[0]]).astype('float32')
labels = np.where(np.array([t.tolist() for t in train_set[1]]) == 0, 1, 0).astype('float32')
buf = io.BytesIO()
smac.write_numpy_to_dense_tensor(buf, vectors, labels)
buf.seek(0)
key = 'recordio-pb-data'
bucket = sagemaker.session.Session().default_bucket()
prefix = 'sagemaker/DEMO-linear-mnist'
boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf)
s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)
output_location = 's3://{}/{}/output'.format(bucket, prefix)
my_experiment = experiment.Experiment.create(experiment_name='MNIST')
my_trial = my_experiment.create_trial(trial_name='linear-learner')
role = get_execution_role()
sess = sagemaker.Session()
linear = sagemaker.estimator.Estimator(container,
role,
train_instance_count=1,
train_instance_type='ml.c4.xlarge',
output_path=output_location,
sagemaker_session=sess)
linear.set_hyperparameters(feature_dim=784,
predictor_type='binary_classifier',
mini_batch_size=200)
linear.fit(inputs={'train': s3_train_data}, experiment_config={
"ExperimentName": my_experiment.experiment_name,
"TrialName": my_trial.trial_name,
"TrialComponentDisplayName": "MNIST-linear-learner",
},)
trial_component_analytics = analytics.ExperimentAnalytics(experiment_name=my_experiment.experiment_name)
analytic_table = trial_component_analytics.dataframe()
analytic_table
For more examples, check out: `sagemaker-experiments <https://github.com/aws/amazon-sagemaker-examples/tree/master/sagemaker-experiments>`_ in `AWS Labs Amazon SageMaker Examples <https://github.com/aws/amazon-sagemaker-examples>`_.
License
-------
This library is licensed under the Apache 2.0 License.
Running Tests
-------------
**Unit Tests**
.. code-block:: bash
tox tests/unit
**Integration Tests**
To run the integration tests, the following prerequisites must be met:
- AWS account credentials are available in the environment for the boto3 client to use.
- The AWS account has an IAM role with SageMaker permissions.
.. code-block:: bash
tox tests/integ
- Test against different regions
.. code-block:: bash
tox -e py39 -- --region cn-north-1
**Docker Based Integration Tests**
Several integration tests rely on docker to push an image to ECR which is then used for training.
Docker Setup
1. Install docker
2. set aws cred helper in docker config (~/.docker/config.json)
.. code-block:: javascript
# docker config example
{
"stackOrchestrator": "swarm",
"credsStore": "desktop",
"auths": {
"https://index.docker.io/v1/": {}
},
"credHelpers": {
"aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
},
"experimental": "disabled"
}
.. code-block:: bash
# run only docker based tests
tox -e py39 -- tests/integ -m 'docker'
# exclude docker based tests
tox -e py39 -- tests/integ -m 'not docker'
Generate Docs
-------------
.. code-block:: bash
tox -e docs
Raw data
{
"_id": null,
"home_page": "https://github.com/aws/sagemaker-experiment-tracking/",
"name": "sagemaker-experiments",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "ML Amazon AWS AI Tensorflow MXNet",
"author": "Amazon Web Services",
"author_email": "",
"download_url": "",
"platform": null,
"description": ".. image:: https://github.com/aws/sagemaker-experiments/raw/main/branding/icon/sagemaker-banner.png\n :height: 100px\n :alt: SageMaker\n :target: https://aws.amazon.com/sagemaker/\n\n**NOTE:** Use the SageMaker `SDK <https://sagemaker.readthedocs.io/en/v2.125.0/experiments/sagemaker.experiments.html>`_ to use SageMaker Experiments. This repository will not be up to date with the latest product improvements. Link to `developer guide <https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html>`_. \n\n================================\nSageMaker Experiments Python SDK\n================================\n\n.. image:: https://img.shields.io/pypi/v/sagemaker-experiments.svg\n :target: https://pypi.python.org/pypi/sagemaker-experiments\n :alt: Latest Version\n\n.. image:: https://img.shields.io/pypi/pyversions/sagemaker-experiments.svg\n :target: https://pypi.python.org/pypi/sagemaker-experiments\n :alt: Supported Python Versions\n\n.. image:: https://img.shields.io/pypi/l/sagemaker-experiments\n :target: https://github.com/aws/sagemaker-experiments/blob/main/LICENSE\n :alt: License\n\n.. image:: https://img.shields.io/pypi/dm/sagemaker-experiments\n :target: https://pypi.python.org/pypi/sagemaker-experiments\n :alt: PyPI - Downloads\n\n.. image:: https://codecov.io/gh/aws/sagemaker-experiments/branch/main/graph/badge.svg\n :target: https://codecov.io/gh/aws/sagemaker-experiments\n :alt: CodeCov\n\n.. image:: https://img.shields.io/pypi/status/sagemaker-experiments\n :target: https://github.com/aws/sagemaker-experiments\n :alt: PyPI - Status\n\n.. image:: https://img.shields.io/pypi/format/coverage.svg\n :target: https://pypi.org/project/coverage/\n :alt: Kit format\n\n.. image:: https://img.shields.io/github/workflow/status/aws/sagemaker-experiments/Main\n :target: https://github.com/aws/sagemaker-experiments/actions\n :alt: GitHub Workflow Status\n\n.. image:: https://img.shields.io/github/stars/aws/sagemaker-experiments.svg?logo=github\n :target: https://github.com/aws/sagemaker-experiments/stargazers\n :alt: Github stars\n\n.. image:: https://img.shields.io/github/forks/aws/sagemaker-experiments.svg?logo=github\n :target: https://github.com/aws/sagemaker-experiments/network/members\n :alt: Github forks\n\n.. image:: https://img.shields.io/github/contributors/aws/sagemaker-experiments.svg?logo=github\n :target: https://github.com/aws/sagemaker-experiments/graphs/contributors\n :alt: Contributors\n\n.. image:: https://img.shields.io/github/search/aws/sagemaker-experiments/sagemaker\n :target: https://github.com/aws/sagemaker-experiments\n :alt: GitHub search hit counter\n\n.. image:: https://img.shields.io/badge/code_style-black-000000.svg\n :target: https://github.com/python/black\n :alt: Code style: black\n\n.. image:: https://readthedocs.org/projects/sagemaker-experiments/badge/?version=latest\n :target: https://readthedocs.org/projects/sagemaker-experiments/\n :alt: Read the Docs - Sagemaker Experiments\n\n.. image:: https://mybinder.org/badge_logo.svg\n :target: https://mybinder.org/v2/gh/aws/amazon-sagemaker-examples/main?filepath=sagemaker-experiments%2Fmnist-handwritten-digits-classification-experiment.ipynb\n\n\n\nExperiment tracking in SageMaker Training Jobs, Processing Jobs, and Notebooks.\n\nOverview\n--------\nSageMaker Experiments is an AWS service for tracking machine learning Experiments. The SageMaker Experiments Python SDK is a high-level interface to this service that helps you track Experiment information using Python.\n\nExperiment tracking powers the machine learning integrated development environment `Amazon SageMaker Studio <https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio.html>`_.\n\nFor detailed API reference please go to: `Read the Docs <https://sagemaker-experiments.readthedocs.io>`_\n\nConcepts\n--------\n\n- **Experiment**: A collection of related Trials. Add Trials to an Experiment that you wish to compare together.\n- **Trial**: A description of a multi-step machine learning workflow. Each step in the workflow is described by a Trial Component. There is no relationship between Trial Components such as ordering.\n- **Trial Component**: A description of a single step in a machine learning workflow. For example data cleaning, feature extraction, model training, model evaluation, etc...\n- **Tracker**: A Python context-manager for logging information about a single TrialComponent.\n\nFor more information see `Amazon SageMaker Experiments - Organize, Track, and Compare Your Machine Learning Trainings <https://aws.amazon.com/blogs/aws/amazon-sagemaker-experiments-organize-track-and-compare-your-machine-learning-trainings/>`_\n\nUsing the SDK\n-------------\nYou can use this SDK to:\n\n- Manage Experiments, Trials, and Trial Components within Python scripts, programs, and notebooks.\n- Add tracking information to a SageMaker notebook, allowing you to model your notebook in SageMaker Experiments as a multi-step ML workflow.\n- Record experiment information from inside your running SageMaker Training and Processing Jobs.\n\nInstallation\n------------\n\n.. code-block:: bash\n\n pip install sagemaker-experiments\n\nExamples\n--------\n\n.. code-block:: python\n\n import boto3\n import pickle, gzip, numpy, json, os\n import io\n import numpy as np\n import sagemaker.amazon.common as smac\n import sagemaker\n from sagemaker import get_execution_role\n from sagemaker import analytics\n from smexperiments import experiment\n\n # Specify training container\n from sagemaker.amazon.amazon_estimator import get_image_uri\n container = get_image_uri(boto3.Session().region_name, 'linear-learner')\n\n # Load the dataset\n s3 = boto3.client(\"s3\")\n s3.download_file(\"sagemaker-sample-files\", \"datasets/image/MNIST/mnist.pkl.gz\", \"mnist.pkl.gz\")\n with gzip.open('mnist.pkl.gz', 'rb') as f:\n train_set, valid_set, test_set = pickle.load(f, encoding='latin1')\n\n vectors = np.array([t.tolist() for t in train_set[0]]).astype('float32')\n labels = np.where(np.array([t.tolist() for t in train_set[1]]) == 0, 1, 0).astype('float32')\n\n buf = io.BytesIO()\n smac.write_numpy_to_dense_tensor(buf, vectors, labels)\n buf.seek(0)\n\n key = 'recordio-pb-data'\n bucket = sagemaker.session.Session().default_bucket()\n prefix = 'sagemaker/DEMO-linear-mnist'\n boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf)\n s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)\n output_location = 's3://{}/{}/output'.format(bucket, prefix)\n\n my_experiment = experiment.Experiment.create(experiment_name='MNIST')\n my_trial = my_experiment.create_trial(trial_name='linear-learner')\n\n role = get_execution_role()\n sess = sagemaker.Session()\n\n linear = sagemaker.estimator.Estimator(container,\n role, \n train_instance_count=1, \n train_instance_type='ml.c4.xlarge',\n output_path=output_location,\n sagemaker_session=sess)\n linear.set_hyperparameters(feature_dim=784,\n predictor_type='binary_classifier',\n mini_batch_size=200)\n\n linear.fit(inputs={'train': s3_train_data}, experiment_config={\n \"ExperimentName\": my_experiment.experiment_name,\n \"TrialName\": my_trial.trial_name,\n \"TrialComponentDisplayName\": \"MNIST-linear-learner\",\n },)\n \n trial_component_analytics = analytics.ExperimentAnalytics(experiment_name=my_experiment.experiment_name)\n\n analytic_table = trial_component_analytics.dataframe()\n analytic_table\n\nFor more examples, check out: `sagemaker-experiments <https://github.com/aws/amazon-sagemaker-examples/tree/master/sagemaker-experiments>`_ in `AWS Labs Amazon SageMaker Examples <https://github.com/aws/amazon-sagemaker-examples>`_.\n\nLicense\n-------\nThis library is licensed under the Apache 2.0 License.\n\nRunning Tests\n-------------\n\n**Unit Tests**\n\n.. code-block:: bash\n\n tox tests/unit\n\n**Integration Tests**\n\nTo run the integration tests, the following prerequisites must be met:\n\n- AWS account credentials are available in the environment for the boto3 client to use.\n- The AWS account has an IAM role with SageMaker permissions.\n\n.. code-block:: bash\n\n tox tests/integ\n\n- Test against different regions\n\n.. code-block:: bash\n\n tox -e py39 -- --region cn-north-1\n \n**Docker Based Integration Tests**\n\nSeveral integration tests rely on docker to push an image to ECR which is then used for training.\n\nDocker Setup\n\n1. Install docker\n2. set aws cred helper in docker config (~/.docker/config.json)\n\n.. code-block:: javascript\n\n # docker config example\n {\n \"stackOrchestrator\": \"swarm\",\n \"credsStore\": \"desktop\",\n \"auths\": {\n \"https://index.docker.io/v1/\": {}\n },\n \"credHelpers\": {\n \"aws_account_id.dkr.ecr.region.amazonaws.com\": \"ecr-login\"\n },\n \"experimental\": \"disabled\"\n }\n\n\n.. code-block:: bash\n\n # run only docker based tests\n tox -e py39 -- tests/integ -m 'docker'\n \n # exclude docker based tests\n tox -e py39 -- tests/integ -m 'not docker'\n\n\n\nGenerate Docs\n-------------\n\n.. code-block:: bash\n\n tox -e docs\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Open source library for Experiment Tracking in SageMaker Jobs and Notebooks",
"version": "0.1.45",
"project_urls": {
"Homepage": "https://github.com/aws/sagemaker-experiment-tracking/"
},
"split_keywords": [
"ml",
"amazon",
"aws",
"ai",
"tensorflow",
"mxnet"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2b2b47d105bbcc328c58b1a23948c3fd9b86930d10b33d220d20c9819e75c41b",
"md5": "d1380c23982dc6f13414d9338702459d",
"sha256": "0fd3eb683882e7dfa25d46da5849a920a2d8850895fd18e4ea790d4d00f8a99d"
},
"downloads": -1,
"filename": "sagemaker_experiments-0.1.45-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d1380c23982dc6f13414d9338702459d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 42713,
"upload_time": "2023-05-17T17:31:33",
"upload_time_iso_8601": "2023-05-17T17:31:33.087037Z",
"url": "https://files.pythonhosted.org/packages/2b/2b/47d105bbcc328c58b1a23948c3fd9b86930d10b33d220d20c9819e75c41b/sagemaker_experiments-0.1.45-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-17 17:31:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aws",
"github_project": "sagemaker-experiment-tracking",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "sagemaker-experiments"
}
JSON
