| Name | jaynes JSON |
| Version |
0.9.8
JSON |
| download |
| home_page | https://github.com/episodeyang/jaynes |
| Summary | A tool for running python code with runner on aws |
| upload_time | 2023-10-19 19:15:16 |
| maintainer | |
| docs_url | None |
| author | Ge Yang |
| requires_python | |
| license | Copyright 2022 Ge Yang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
jaynes
logging
debug
debugging
timer
timeit
decorator
stopwatch
tic
toc
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Jaynes, A Utility for training ML models on AWS, GCE, SLURM, with or without docker
===================================================================================
|Downloads|
Overview
--------
The reality of ML training in universities is that we use what ever
hardware we are given (for free). This means that we might have a few
beefy GPU machines, an HPC cluster, plus some GCE/AWS credits that we
get through grants. `Jaynes <https://github.com/episodeyang/jaynes>`__
is a well-designed python package that makes running across these
inhomogenous hardward resources a pleasure.
**install** (requires unix operating system.)
.. code-block:: bash
pip install jaynes
The best way to get started with jaynes is to take a look at one of the
example projects in the
`[geyang/jaynes-starter-kit] <https://github.com/geyang/jaynes-starter-kit>`__.
For a rough idea, here is how to use jaynes to launch a training
function:
To run **locally**:
.. code-block:: python
import jaynes
def training(arg_1, key_arg=None, key_arg_2=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.config(mode="local", arg_1=10, key_arg=0.3)
jaynes.run(training)
jaynes.listen()
We recommend setting up a ``main`` training function with the following
sinature:
.. code-block:: python
from params_proto import ParamsProto
class Args(ParamsProto):
seed = 100
lr = 3e-4
# ...
def main(**deps):
from ml_logger import logger
Args._update(deps)
logger.log_params(Args=vars(Args))
# ... your main training steps
This way you can call the main fn directly for local debugging, but
launch it as an entry point at scale.
Setup
-----
Jaynes has gone through a large number of iterations. This version
incorporates best practices we learned from other open-source
communities. You can specify a ``jaynes.yml`` config file (`copy one
from our sample project to get started! <example_projects>`__) for the
type of hosts (ssh/docker/singularity) and launchers
(ssh/ec2/gce/slurm), so that none of those settings need to appear in
your ML python script. When called from python, Jaynes automatically
traverses the file tree to find the root of the project, the same way as
git.
For example, to run your code-block on a remote computer via ssh:
.. code-block:: yaml
# your_project/jaynes.yml
version: 0
verbose: true
run: # this is specific to each launch, and is dynamically overwritten in-memory
mounts:
- !mounts.S3Code
s3_prefix: s3://ge-bair/jaynes-debug
local_path: .
host_path: /home/ubuntu/
container_path: /Users/geyang/learning-to-learn
pypath: true
excludes: "--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'"
compress: true
runner:
!runners.Docker
name: # not implemented yet
image: "episodeyang/super-expert"
startup: "yes | pip install jaynes ml-logger -q"
work_directory: "{mounts[0].container_path}"
ipc: host
host:
envs: "LANG=utf-8"
pre_launch: "pip install jaynes ml-logger -q"
launch:
type: ssh
ip: <your ip address>
username: ubuntu
pem: ~/.ssh/your_rsa_key
In python (your code-block):
.. code-block:: python
# your_project/launch.py
import jaynes
def training(arg_1, key_arg=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.run(training)
Using Modes
-----------
A lot of times you want to setup a different run **modes** so it is easy
to switch between them during development.
.. code-block:: yaml
# your_project/jaynes.yml
version: 0
mounts: # mount configurations Available keys: NOW, UUID,
- !mounts.S3Code &code-block_mount
s3_prefix: s3://ge-bair/jaynes-debug
local_path: .
host_path: /home/ubuntu/jaynes-mounts/{NOW:%Y-%m-%d}/{NOW:%H%M%S.%f}
# container_path: /Users/geyang/learning-to-learn
pypath: true
excludes: "--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'"
compress: true
hosts:
hodor: &hodor
ip: <your ip address>
username: ubuntu
pem: ~/.ssh/incrementium-berkeley
runners:
- !runners.Docker &ssh_docker
name: "some-job" # only for docker
image: "episodeyang/super-expert"
startup: yes | pip install jaynes ml-logger -q
envs: "LANG=utf-8"
pypath: "{mounts[0].container_path}"
launch_directory: "{mounts[0].container_path}"
ipc: host
use_gpu: false
modes: # todo: add support to modes.
hodor:
mounts:
- *code-block_mount
runner: *ssh_docker
launch:
type: ssh
<<: *hodor
now run in python
.. code-block:: python
# your_project/launch.py
import jaynes
def training(arg_1, key_arg=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.config(mode="hodor")
jaynes.run(training)
ToDos
-----
- ☐ more documentation
- ☐ singularity support
- ☐ GCE support
- ☐ support using non-s3 code-block repo.
Done
~~~~
- ☒ get the initial template to work
Installation
------------
.. code-block:: bash
pip install jaynes
Usage (**Show me the Mo-NAY!! :moneybag::money_with_wings:**)
-------------------------------------------------------------
Check out the `test_projects <example_projects>`__ folder for projects
that you can run.
To Develop
----------
.. code-block:: bash
git clone https://github.com/episodeyang/jaynes.git
cd jaynes
make dev
To test, run
.. code-block:: bash
make test
This ``make dev`` command should build the wheel and install it in your
current python environment. Take a look at the
`https://github.com/episodeyang/jaynes/blob/master/Makefile <https://github.com/episodeyang/jaynes/blob/master/Makefile>`__ for details.
**To publish**, first update the version number, then do:
.. code-block:: bash
make publish
Acknowledgements
----------------
This code-block is inspired by @justinfu’s
`doodad <https://github.com/justinjfu/doodad>`__, which is in turn built
on top of Peter Chen’s script.
This code-block is written from scratch to allow a more permissible
open-source license (BSD). Go bears :bear: !!
.. |Downloads| image:: http://pepy.tech/badge/jaynes
:target: http://pepy.tech/project/jaynes
Raw data
{
"_id": null,
"home_page": "https://github.com/episodeyang/jaynes",
"name": "jaynes",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "jaynes,logging,DEBUG,debugging,timer,timeit,decorator,stopwatch,tic,toc",
"author": "Ge Yang",
"author_email": "yangge1987@gmail.com",
"download_url": "",
"platform": null,
"description": "Jaynes, A Utility for training ML models on AWS, GCE, SLURM, with or without docker \n===================================================================================\n\n|Downloads|\n\nOverview\n--------\n\nThe reality of ML training in universities is that we use what ever\nhardware we are given (for free). This means that we might have a few\nbeefy GPU machines, an HPC cluster, plus some GCE/AWS credits that we\nget through grants. `Jaynes <https://github.com/episodeyang/jaynes>`__\nis a well-designed python package that makes running across these\ninhomogenous hardward resources a pleasure.\n\n**install** (requires unix operating system.)\n\n.. code-block:: bash\n\n pip install jaynes\n\nThe best way to get started with jaynes is to take a look at one of the\nexample projects in the\n`[geyang/jaynes-starter-kit] <https://github.com/geyang/jaynes-starter-kit>`__.\nFor a rough idea, here is how to use jaynes to launch a training\nfunction:\n\nTo run **locally**:\n\n.. code-block:: python\n\n import jaynes\n\n def training(arg_1, key_arg=None, key_arg_2=None):\n print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n jaynes.config(mode=\"local\", arg_1=10, key_arg=0.3)\n jaynes.run(training)\n jaynes.listen()\n\nWe recommend setting up a ``main`` training function with the following\nsinature:\n\n.. code-block:: python\n\n from params_proto import ParamsProto\n\n class Args(ParamsProto):\n seed = 100\n lr = 3e-4\n # ...\n \n def main(**deps):\n from ml_logger import logger\n \n Args._update(deps)\n logger.log_params(Args=vars(Args))\n \n # ... your main training steps\n\nThis way you can call the main fn directly for local debugging, but\nlaunch it as an entry point at scale.\n\nSetup\n-----\n\nJaynes has gone through a large number of iterations. This version\nincorporates best practices we learned from other open-source\ncommunities. You can specify a ``jaynes.yml`` config file (`copy one\nfrom our sample project to get started! <example_projects>`__) for the\ntype of hosts (ssh/docker/singularity) and launchers\n(ssh/ec2/gce/slurm), so that none of those settings need to appear in\nyour ML python script. When called from python, Jaynes automatically\ntraverses the file tree to find the root of the project, the same way as\ngit.\n\nFor example, to run your code-block on a remote computer via ssh:\n\n.. code-block:: yaml\n\n # your_project/jaynes.yml\n version: 0\n verbose: true\n run: # this is specific to each launch, and is dynamically overwritten in-memory\n mounts:\n - !mounts.S3Code\n s3_prefix: s3://ge-bair/jaynes-debug\n local_path: .\n host_path: /home/ubuntu/\n container_path: /Users/geyang/learning-to-learn\n pypath: true\n excludes: \"--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'\"\n compress: true\n runner:\n !runners.Docker\n name: # not implemented yet\n image: \"episodeyang/super-expert\"\n startup: \"yes | pip install jaynes ml-logger -q\"\n work_directory: \"{mounts[0].container_path}\"\n ipc: host\n host:\n envs: \"LANG=utf-8\"\n pre_launch: \"pip install jaynes ml-logger -q\"\n launch:\n type: ssh\n ip: <your ip address>\n username: ubuntu\n pem: ~/.ssh/your_rsa_key\n\nIn python (your code-block):\n\n.. code-block:: python\n\n # your_project/launch.py\n import jaynes\n\n def training(arg_1, key_arg=None):\n print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n jaynes.run(training)\n\nUsing Modes\n-----------\n\nA lot of times you want to setup a different run **modes** so it is easy\nto switch between them during development.\n\n.. code-block:: yaml\n\n # your_project/jaynes.yml\n version: 0\n mounts: # mount configurations Available keys: NOW, UUID,\n - !mounts.S3Code &code-block_mount\n s3_prefix: s3://ge-bair/jaynes-debug\n local_path: .\n host_path: /home/ubuntu/jaynes-mounts/{NOW:%Y-%m-%d}/{NOW:%H%M%S.%f}\n # container_path: /Users/geyang/learning-to-learn\n pypath: true\n excludes: \"--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'\"\n compress: true\n hosts:\n hodor: &hodor\n ip: <your ip address>\n username: ubuntu\n pem: ~/.ssh/incrementium-berkeley\n runners:\n - !runners.Docker &ssh_docker\n name: \"some-job\" # only for docker\n image: \"episodeyang/super-expert\"\n startup: yes | pip install jaynes ml-logger -q\n envs: \"LANG=utf-8\"\n pypath: \"{mounts[0].container_path}\"\n launch_directory: \"{mounts[0].container_path}\"\n ipc: host\n use_gpu: false\n modes: # todo: add support to modes.\n hodor:\n mounts:\n - *code-block_mount\n runner: *ssh_docker\n launch:\n type: ssh\n <<: *hodor\n\nnow run in python\n\n.. code-block:: python\n\n # your_project/launch.py\n import jaynes\n\n def training(arg_1, key_arg=None):\n print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n jaynes.config(mode=\"hodor\")\n jaynes.run(training)\n\nToDos\n-----\n\n- \u2610 more documentation\n- \u2610 singularity support\n- \u2610 GCE support\n- \u2610 support using non-s3 code-block repo.\n\nDone\n~~~~\n\n- \u2612 get the initial template to work\n\nInstallation\n------------\n\n.. code-block:: bash\n\n pip install jaynes\n\nUsage (**Show me the Mo-NAY!! :moneybag::money_with_wings:**)\n-------------------------------------------------------------\n\nCheck out the `test_projects <example_projects>`__ folder for projects\nthat you can run.\n\nTo Develop\n----------\n\n.. code-block:: bash\n\n git clone https://github.com/episodeyang/jaynes.git\n cd jaynes\n make dev\n\nTo test, run\n\n.. code-block:: bash\n\n make test\n\nThis ``make dev`` command should build the wheel and install it in your\ncurrent python environment. Take a look at the\n`https://github.com/episodeyang/jaynes/blob/master/Makefile <https://github.com/episodeyang/jaynes/blob/master/Makefile>`__ for details.\n\n**To publish**, first update the version number, then do:\n\n.. code-block:: bash\n\n make publish\n\nAcknowledgements\n----------------\n\nThis code-block is inspired by @justinfu\u2019s\n`doodad <https://github.com/justinjfu/doodad>`__, which is in turn built\non top of Peter Chen\u2019s script.\n\nThis code-block is written from scratch to allow a more permissible\nopen-source license (BSD). Go bears :bear: !!\n\n.. |Downloads| image:: http://pepy.tech/badge/jaynes\n :target: http://pepy.tech/project/jaynes\n",
"bugtrack_url": null,
"license": "Copyright 2022 Ge Yang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A tool for running python code with runner on aws",
"version": "0.9.8",
"project_urls": {
"Homepage": "https://github.com/episodeyang/jaynes"
},
"split_keywords": [
"jaynes",
"logging",
"debug",
"debugging",
"timer",
"timeit",
"decorator",
"stopwatch",
"tic",
"toc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a2dafc30e8cea39f8f1b0c57379aebce24715c520b688c6041443d06d14c212a",
"md5": "148008bedcc9ff85f5c84c6078e2d932",
"sha256": "4635c154c93d5fbdbc759b4131eab154c425be89612eed0cdffefc4af52c8951"
},
"downloads": -1,
"filename": "jaynes-0.9.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "148008bedcc9ff85f5c84c6078e2d932",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 40143,
"upload_time": "2023-10-19T19:15:16",
"upload_time_iso_8601": "2023-10-19T19:15:16.097864Z",
"url": "https://files.pythonhosted.org/packages/a2/da/fc30e8cea39f8f1b0c57379aebce24715c520b688c6041443d06d14c212a/jaynes-0.9.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-19 19:15:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "episodeyang",
"github_project": "jaynes",
"github_not_found": true,
"lcname": "jaynes"
}
