``params-proto``, Modern Hyper Parameter Management for Machine Learning
========================================================================
- 2022/07/04:
- Move ``neo_proto`` to top-level, move older ``params_proto`` to
``v1`` namespace.
- Implement nested update via `global
prefix <https://github.com/geyang/params_proto/blob/master/test_params_proto/test_neo_proto.py#L278>`__.
No relative update via ``**kwargs``, yet
- Fix ``to_value`` bug in Flag
- 2021/06/16: Proto now supports using environment variables as
default.
- 2021/06/13: 5 month into my postdoc at MIT, add
``sweep.save("sweep.jsonl")`` to dump the sweep into a ``jsonl`` file
for large scale experiments on AWS.
- 2019/12/09: Just finished my DeepMind Internship. Now
``params-proto`` contain a new proto implementation, and a
complementary hyperparameter search library! See ``neo_proto`` and
``neo_hyper``.
- 2019/06/11: Now supports ``tab-completion`` at the command line!
- 2018/11/08: Now supports both python ``3.52`` as well as ``3.6``!
:bangbang::star:
What is “Experiment Parameter Hell”?
------------------------------------
“Experiemnt Parameter Hell” occurs when you have more than twenty
parameters for your ML project that are all defined as string/function
parameters with ``click`` or ``argparse``. Sometimes these parameters
are defined in a launch script and passes through five layers of
function calls during an experiment.
Your Python IDEs work very hard on static code-block analysis to intelligently
make you more productive, and the “parameter hell” breaks all of that.
Step 1: Declarative Pattern to the Rescue!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For this reason, you want to avoid using dictionaries or opaque
``argparse`` definitions as much as possible. Instead, you want to write
those declaratively, so that your IDE can actually help you navigate
through those layers of function calls. The hyper-parameter library,
``params_proto`` makes this easy, by integrating python namespaces (a
bare python class) with ``argparse``, so that on the python side you get
auto-completion, and from the command line you can pass in changes.
**Installation**
First let’s install ``params-proto`` and its supporting module
``waterbear``
.. code-block:: bash
pip install params-proto waterbear
Then to declare your hyperparameters, you can write the following in a
``your_project/soft_ac/config.py`` file:
.. code-block:: python
import sys
from params_proto.proto import ParamsProto, Flag, Proto, PrefixProto
# this is the first config schema
class Args(PrefixProto):
"""Soft-actor Critic Implementation with SOTA Performance
"""
debug = True if "pydevd" in sys.modules else False
cuda = Flag("cuda tend to be slower.")
seed = 42
env_name = "FetchReach-v1"
n_workers = 1 if debug else 12
v_lr = 1e-3
pi_lr = 1e-3
n_initial_rollouts = 0 if debug else 100
n_test_rollouts = 15
demo_length = 20
clip_inputs = Flag()
normalize_inputs = Flag()
# this is the second schema
class LfGR(PrefixProto):
# reporting
use_lfgr = True
start = 0 if Args.debug else 10
store_interval = 10
visualization_interval = 10
Step 2: Sweeping Hyper-parameters :fire:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Then you an sweep the hyperparameter via the following declarative
pattern:
.. code-block:: python
from rl import main, Args
from params_proto.hyper import Sweep
if __name__ == '__main__':
from lp_analysis import instr
with Sweep(Args, LfGR) as sweep:
# override the default
Args.pi_lr = 3e-3
Args.clip_inputs = True # this was a flag
# override the second config object
LfGR.visualization_interval = 40
# product between the zipped and the seed
with sweep.product:
# similar to python zip, unpacks a list of values.
with sweep.zip:
Args.env_name = ['FetchReach-v1', 'FetchPush-v1', 'FetchPickAndPlace-v1', 'FetchSlide-v1']
Args.n_epochs = [4, 12, 12, 20]
Args.n_workers = [5, 150, 200, 500]
# the seed is sweeped at last
Args.seed = [100, 200, 300, 400, 500, 600]
# You can save the sweep into a `jsonl` file
sweep.save('sweep.jsonl')
for i, deps in sweep.items():
thunk = instr(main, deps, _job_postfix=f"{Args.env_name}")
print(deps)
and it should print out a list of dictionaries that looks like:
.. code-block:: bash
{Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: "FetchReach-v1", ... Args.seed: 100}
{Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: "FetchReach-v1", ... Args.seed: 200}
{Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: "FetchReach-v1", ... Args.seed: 300}
...
## Where Can I find Documentation?
Look at the specification file at
`https://github.com/episodeyang/params_proto/blob/master/test_params_proto/\*.py <test_params_proto>`__ , which is part of the
integrated test. These scripts contains the most comprehensive set of
usage patters!!
The new version has a ``neo_`` prefix. We will deprecate the older
(non-neo) version in a few month.
Writing documentation as uhm…, man page?
----------------------------------------
``Params-Proto`` exposes your argument namespace’s doc string as the
usage note. For users of your code-block, there is no better help than the one
that comes with the script itself!
With ``params-proto``, your help is only one ``-h`` away :)
And **Your code-block becomes the documentation.**
Tab-completion for your script!
-------------------------------
``params_proto`` uses ``argparse`` together with ``argcomplete``, which
enables command line autocomplete on tabs! To enable run
.. code-block:: python
pip install params-proto
# then:
activate-global-python-argcomplete
For details, see ```argcomplete``\ ’s
documentation <https://github.com/kislyuk/argcomplete#installation>`__.
Why Use Params_Proto Instead of Click or Argparse?
--------------------------------------------------
Because this declarative, singleton pattern allows you to:
Place all of the arguments under a namespace that can be statically
checked.
so that your IDE can:
1. Find usage of each argument
2. jump from *anywhere* in your code-block base to the declaration of that
argument
3. refactor your argument name **in the entire code-block base** automatically
``Params_proto`` is the declarative way to write command line arguments,
and is the way to go for ML projects.
How to override when calling from python
----------------------------------------
It is very easy to over-ride the parameters when you call your function:
have most of your training code-block **directly** reference the parser
namespace (your configuration namespace really), and just monkey patch
the attribute.
``params-proto`` works very well with the clound ML launch tool
`jaynes <https://github.com/episodeyang/jaynes>`__. Take a look at the
automagic awesomeness of
`jaynes <https://github.com/episodeyang/jaynes>`__:)
To Develop And Contribute
-------------------------
.. code-block:: bash
git clone https://github.com/episodeyang/params_proto.git
cd params_proto
make dev
To test, run the following under both python ``3.52`` and ``3.6``.
.. code-block:: bash
make test
This ``make dev`` command should build the wheel and install it in your
current python environment. Take a look at the
`https://github.com/episodeyang/params_proto/blob/master/Makefile <https://github.com/episodeyang/params_proto/blob/master/Makefile>`__ for details.
**To publish**, first update the version number, then do:
.. code-block:: bash
make publish
Raw data
{
"_id": null,
"home_page": "https://github.com/episodeyang/params_proto",
"name": "params-proto",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "params_proto, decorator, argparse, argcomplete, auto-completion, autocomplete, shell arguments, argument parser",
"author": "Ge Yang",
"author_email": "ge.ike.yang@gmail.com",
"download_url": null,
"platform": null,
"description": "``params-proto``, Modern Hyper Parameter Management for Machine Learning\n========================================================================\n\n- 2022/07/04:\n\n - Move ``neo_proto`` to top-level, move older ``params_proto`` to\n ``v1`` namespace.\n - Implement nested update via `global\n prefix <https://github.com/geyang/params_proto/blob/master/test_params_proto/test_neo_proto.py#L278>`__.\n No relative update via ``**kwargs``, yet\n - Fix ``to_value`` bug in Flag\n\n- 2021/06/16: Proto now supports using environment variables as\n default.\n- 2021/06/13: 5 month into my postdoc at MIT, add\n ``sweep.save(\"sweep.jsonl\")`` to dump the sweep into a ``jsonl`` file\n for large scale experiments on AWS.\n- 2019/12/09: Just finished my DeepMind Internship. Now\n ``params-proto`` contain a new proto implementation, and a\n complementary hyperparameter search library! See ``neo_proto`` and\n ``neo_hyper``.\n- 2019/06/11: Now supports ``tab-completion`` at the command line!\n- 2018/11/08: Now supports both python ``3.52`` as well as ``3.6``!\n :bangbang::star:\n\nWhat is \u201cExperiment Parameter Hell\u201d?\n------------------------------------\n\n\u201cExperiemnt Parameter Hell\u201d occurs when you have more than twenty\nparameters for your ML project that are all defined as string/function\nparameters with ``click`` or ``argparse``. Sometimes these parameters\nare defined in a launch script and passes through five layers of\nfunction calls during an experiment.\n\nYour Python IDEs work very hard on static code-block analysis to intelligently\nmake you more productive, and the \u201cparameter hell\u201d breaks all of that.\n\nStep 1: Declarative Pattern to the Rescue!\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this reason, you want to avoid using dictionaries or opaque\n``argparse`` definitions as much as possible. Instead, you want to write\nthose declaratively, so that your IDE can actually help you navigate\nthrough those layers of function calls. The hyper-parameter library,\n``params_proto`` makes this easy, by integrating python namespaces (a\nbare python class) with ``argparse``, so that on the python side you get\nauto-completion, and from the command line you can pass in changes.\n\n**Installation**\n\nFirst let\u2019s install ``params-proto`` and its supporting module\n``waterbear``\n\n.. code-block:: bash\n\n pip install params-proto waterbear\n\nThen to declare your hyperparameters, you can write the following in a\n``your_project/soft_ac/config.py`` file:\n\n.. code-block:: python\n\n import sys\n from params_proto.proto import ParamsProto, Flag, Proto, PrefixProto\n\n\n # this is the first config schema\n class Args(PrefixProto):\n \"\"\"Soft-actor Critic Implementation with SOTA Performance\n \"\"\"\n\n debug = True if \"pydevd\" in sys.modules else False\n\n cuda = Flag(\"cuda tend to be slower.\")\n seed = 42\n env_name = \"FetchReach-v1\"\n n_workers = 1 if debug else 12\n v_lr = 1e-3\n pi_lr = 1e-3\n n_initial_rollouts = 0 if debug else 100\n n_test_rollouts = 15\n demo_length = 20\n clip_inputs = Flag()\n normalize_inputs = Flag()\n\n\n # this is the second schema\n class LfGR(PrefixProto):\n # reporting\n use_lfgr = True\n start = 0 if Args.debug else 10\n store_interval = 10\n visualization_interval = 10\n\nStep 2: Sweeping Hyper-parameters :fire:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThen you an sweep the hyperparameter via the following declarative\npattern:\n\n.. code-block:: python\n\n from rl import main, Args\n from params_proto.hyper import Sweep\n\n if __name__ == '__main__':\n from lp_analysis import instr\n\n with Sweep(Args, LfGR) as sweep:\n # override the default\n Args.pi_lr = 3e-3\n Args.clip_inputs = True # this was a flag\n\n # override the second config object\n LfGR.visualization_interval = 40\n\n # product between the zipped and the seed\n with sweep.product:\n # similar to python zip, unpacks a list of values.\n with sweep.zip:\n Args.env_name = ['FetchReach-v1', 'FetchPush-v1', 'FetchPickAndPlace-v1', 'FetchSlide-v1']\n Args.n_epochs = [4, 12, 12, 20]\n Args.n_workers = [5, 150, 200, 500]\n\n # the seed is sweeped at last\n Args.seed = [100, 200, 300, 400, 500, 600]\n\n # You can save the sweep into a `jsonl` file\n sweep.save('sweep.jsonl')\n\n for i, deps in sweep.items():\n thunk = instr(main, deps, _job_postfix=f\"{Args.env_name}\")\n print(deps)\n\nand it should print out a list of dictionaries that looks like:\n\n.. code-block:: bash\n\n {Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: \"FetchReach-v1\", ... Args.seed: 100}\n {Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: \"FetchReach-v1\", ... Args.seed: 200}\n {Args.pi_lr: 3e-3, Args.clip_inputs: True, LfGR.visualization_interval: 40, Args.env_name: \"FetchReach-v1\", ... Args.seed: 300}\n ...\n\n## Where Can I find Documentation?\n\nLook at the specification file at\n`https://github.com/episodeyang/params_proto/blob/master/test_params_proto/\\*.py <test_params_proto>`__ , which is part of the\nintegrated test. These scripts contains the most comprehensive set of\nusage patters!!\n\nThe new version has a ``neo_`` prefix. We will deprecate the older\n(non-neo) version in a few month.\n\nWriting documentation as uhm\u2026, man page?\n----------------------------------------\n\n``Params-Proto`` exposes your argument namespace\u2019s doc string as the\nusage note. For users of your code-block, there is no better help than the one\nthat comes with the script itself!\n\n With ``params-proto``, your help is only one ``-h`` away :)\n\nAnd **Your code-block becomes the documentation.**\n\nTab-completion for your script!\n-------------------------------\n\n``params_proto`` uses ``argparse`` together with ``argcomplete``, which\nenables command line autocomplete on tabs! To enable run\n\n.. code-block:: python\n\n pip install params-proto\n # then:\n activate-global-python-argcomplete\n\nFor details, see ```argcomplete``\\ \u2019s\ndocumentation <https://github.com/kislyuk/argcomplete#installation>`__.\n\nWhy Use Params_Proto Instead of Click or Argparse?\n--------------------------------------------------\n\nBecause this declarative, singleton pattern allows you to:\n\n Place all of the arguments under a namespace that can be statically\n checked.\n\nso that your IDE can:\n\n1. Find usage of each argument\n2. jump from *anywhere* in your code-block base to the declaration of that\n argument\n3. refactor your argument name **in the entire code-block base** automatically\n\n``Params_proto`` is the declarative way to write command line arguments,\nand is the way to go for ML projects.\n\nHow to override when calling from python\n----------------------------------------\n\nIt is very easy to over-ride the parameters when you call your function:\nhave most of your training code-block **directly** reference the parser\nnamespace (your configuration namespace really), and just monkey patch\nthe attribute.\n\n``params-proto`` works very well with the clound ML launch tool\n`jaynes <https://github.com/episodeyang/jaynes>`__. Take a look at the\nautomagic awesomeness of\n`jaynes <https://github.com/episodeyang/jaynes>`__:)\n\nTo Develop And Contribute\n-------------------------\n\n.. code-block:: bash\n\n git clone https://github.com/episodeyang/params_proto.git\n cd params_proto\n make dev\n\nTo test, run the following under both python ``3.52`` and ``3.6``.\n\n.. code-block:: bash\n\n make test\n\nThis ``make dev`` command should build the wheel and install it in your\ncurrent python environment. Take a look at the\n`https://github.com/episodeyang/params_proto/blob/master/Makefile <https://github.com/episodeyang/params_proto/blob/master/Makefile>`__ for details.\n\n**To publish**, first update the version number, then do:\n\n.. code-block:: bash\n\n make publish\n",
"bugtrack_url": null,
"license": null,
"summary": "Modern Hyper Parameter Management for Machine Learning",
"version": "2.12.1",
"project_urls": {
"Homepage": "https://github.com/episodeyang/params_proto"
},
"split_keywords": [
"params_proto",
" decorator",
" argparse",
" argcomplete",
" auto-completion",
" autocomplete",
" shell arguments",
" argument parser"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5b021f9ded4df5d939381f62960253b5a4584e85e9533b527b640d9d8510a8d3",
"md5": "2b6f2d32a08b68cbbf2daba03b55a678",
"sha256": "ca4f08ce7bb7ca46b9db52a1935842dba0720cc100d62ff77026ec5237e0046b"
},
"downloads": -1,
"filename": "params_proto-2.12.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2b6f2d32a08b68cbbf2daba03b55a678",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 21516,
"upload_time": "2024-04-20T20:01:37",
"upload_time_iso_8601": "2024-04-20T20:01:37.599477Z",
"url": "https://files.pythonhosted.org/packages/5b/02/1f9ded4df5d939381f62960253b5a4584e85e9533b527b640d9d8510a8d3/params_proto-2.12.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-20 20:01:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "episodeyang",
"github_project": "params_proto",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "params-proto"
}
JSON
