# ML Collections
ML Collections is a library of Python Collections designed for ML use cases.
[](https://ml-collections.readthedocs.io/en/latest/?badge=latest)
[](https://badge.fury.io/py/ml-collections)
[](https://github.com/google/ml_collections/actions?query=workflow%3A%22Python+package%22)
## ConfigDict
The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
structures with dot access to nested elements. Together, they are supposed to be
used as a main way of expressing configurations of experiments and models.
This document describes example usage of `ConfigDict`, `FrozenConfigDict`,
`FieldReference`.
### Features
* Dot-based access to fields.
* Locking mechanism to prevent spelling mistakes.
* Lazy computation.
* FrozenConfigDict() class which is immutable and hashable.
* Type safety.
* "Did you mean" functionality.
* Human readable printing (with valid references and cycles), using valid YAML
format.
* Fields can be passed as keyword arguments using the `**` operator.
* There are two exceptions to the strong type-safety of the ConfigDict. `int`
values can be passed in to fields of type `float`. In such a case, the value
is type-converted to a `float` before being stored. Similarly, all string
types (including Unicode strings) can be stored in fields of type `str` or
`unicode`.
### Basic Usage
```python
import ml_collections
cfg = ml_collections.ConfigDict()
cfg.float_field = 12.6
cfg.integer_field = 123
cfg.another_integer_field = 234
cfg.nested = ml_collections.ConfigDict()
cfg.nested.string_field = 'tom'
print(cfg.integer_field) # Prints 123.
print(cfg['integer_field']) # Prints 123 as well.
try:
cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.
except TypeError as e:
print(e)
cfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.
cfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.
print(cfg)
```
### FrozenConfigDict
A `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:
```python
import ml_collections
initial_dictionary = {
'int': 1,
'list': [1, 2],
'tuple': (1, 2, 3),
'set': {1, 2, 3, 4},
'dict_tuple_list': {'tuple_list': ([1, 2], 3)}
}
cfg = ml_collections.ConfigDict(initial_dictionary)
frozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)
print(frozen_dict.tuple) # Prints tuple (1, 2, 3)
print(frozen_dict.list) # Prints tuple (1, 2)
print(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}
print(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)
frozen_cfg = ml_collections.FrozenConfigDict(cfg)
print(frozen_cfg == frozen_dict) # True
print(hash(frozen_cfg) == hash(frozen_dict)) # True
try:
frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.
except AttributeError as e:
print(e)
# Converting between `FrozenConfigDict` and `ConfigDict`:
thawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)
print(thawed_frozen_cfg == cfg) # True
frozen_cfg_to_cfg = frozen_dict.as_configdict()
print(frozen_cfg_to_cfg == cfg) # True
```
### FieldReferences and placeholders
A `FieldReference` is useful for having multiple fields use the same value. It
can also be used for [lazy computation](#lazy-computation).
You can use `placeholder()` as a shortcut to create a `FieldReference` (field)
with a `None` default value. This is useful if a program uses optional
configuration fields.
```python
import ml_collections
from ml_collections.config_dict import config_dict
placeholder = ml_collections.FieldReference(0)
cfg = ml_collections.ConfigDict()
cfg.placeholder = placeholder
cfg.optional = config_dict.placeholder(int)
cfg.nested = ml_collections.ConfigDict()
cfg.nested.placeholder = placeholder
try:
cfg.optional = 'tom' # Raises Type error as this field is an integer.
except TypeError as e:
print(e)
cfg.optional = 1555 # Works fine.
cfg.placeholder = 1 # Changes the value of both placeholder and
# nested.placeholder fields.
print(cfg)
```
Note that the indirection provided by `FieldReference`s will be lost if accessed
through a `ConfigDict`.
```python
import ml_collections
placeholder = ml_collections.FieldReference(0)
cfg.field1 = placeholder
cfg.field2 = placeholder # This field will be tied to cfg.field1.
cfg.field3 = cfg.field1 # This will just be an int field initialized to 0.
```
### Lazy computation
Using a `FieldReference` in a standard operation (addition, subtraction,
multiplication, etc...) will return another `FieldReference` that points to the
original's value. You can use `FieldReference.get()` to execute the operations
and get the reference's computed value, and `FieldReference.set()` to change the
original reference's value.
```python
import ml_collections
ref = ml_collections.FieldReference(1)
print(ref.get()) # Prints 1
add_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten
add_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer
print(add_ten) # Prints 11
print(add_ten_lazy.get()) # Prints 11 because ref's value is 1
# Addition is lazily computed for FieldReferences so changing ref will change
# the value that is used to compute add_ten.
ref.set(5)
print(add_ten) # Prints 11
print(add_ten_lazy.get()) # Prints 15 because ref's value is 5
```
If a `FieldReference` has `None` as its original value, or any operation has an
argument of `None`, then the lazy computation will evaluate to `None`.
We can also use fields in a `ConfigDict` in lazy computation. In this case a
field will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.
```python
import ml_collections
config = ml_collections.ConfigDict()
config.reference_field = ml_collections.FieldReference(1)
config.integer_field = 2
config.float_field = 2.5
# No lazy evaluatuations because we didn't use get_ref()
config.no_lazy = config.integer_field * config.float_field
# This will lazily evaluate ONLY config.integer_field
config.lazy_integer = config.get_ref('integer_field') * config.float_field
# This will lazily evaluate ONLY config.float_field
config.lazy_float = config.integer_field * config.get_ref('float_field')
# This will lazily evaluate BOTH config.integer_field and config.float_Field
config.lazy_both = (config.get_ref('integer_field') *
config.get_ref('float_field'))
config.integer_field = 3
print(config.no_lazy) # Prints 5.0 - It uses integer_field's original value
print(config.lazy_integer) # Prints 7.5
config.float_field = 3.5
print(config.lazy_float) # Prints 7.0
print(config.lazy_both) # Prints 10.5
```
#### Changing lazily computed values
Lazily computed values in a ConfigDict can be overridden in the same way as
regular values. The reference to the `FieldReference` used for the lazy
computation will be lost and all computations downstream in the reference graph
will use the new value.
```python
import ml_collections
config = ml_collections.ConfigDict()
config.reference = 1
config.reference_0 = config.get_ref('reference') + 10
config.reference_1 = config.get_ref('reference') + 20
config.reference_1_0 = config.get_ref('reference_1') + 100
print(config.reference) # Prints 1.
print(config.reference_0) # Prints 11.
print(config.reference_1) # Prints 21.
print(config.reference_1_0) # Prints 121.
config.reference_1 = 30
print(config.reference) # Prints 1 (unchanged).
print(config.reference_0) # Prints 11 (unchanged).
print(config.reference_1) # Prints 30.
print(config.reference_1_0) # Prints 130.
```
#### Cycles
You cannot create cycles using references. Fortunately
[the only way](#changing-lazily-computed-values) to create a cycle is by
assigning a computed field to one that *is not* the result of computation. This
is forbidden:
```python
import ml_collections
from ml_collections.config_dict import config_dict
config = ml_collections.ConfigDict()
config.integer_field = 1
config.bigger_integer_field = config.get_ref('integer_field') + 10
try:
# Raises a MutabilityError because setting config.integer_field would
# cause a cycle.
config.integer_field = config.get_ref('bigger_integer_field') + 2
except config_dict.MutabilityError as e:
print(e)
```
### Advanced usage
Here are some more advanced examples showing lazy computation with different
operators and data types.
```python
import ml_collections
config = ml_collections.ConfigDict()
config.float_field = 12.6
config.integer_field = 123
config.list_field = [0, 1, 2]
config.float_multiply_field = config.get_ref('float_field') * 3
print(config.float_multiply_field) # Prints 37.8
config.float_field = 10.0
print(config.float_multiply_field) # Prints 30.0
config.longer_list_field = config.get_ref('list_field') + [3, 4, 5]
print(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]
config.list_field = [-1]
print(config.longer_list_field) # Prints [-1, 3, 4, 5]
# Both operands can be references
config.ref_subtraction = (
config.get_ref('float_field') - config.get_ref('integer_field'))
print(config.ref_subtraction) # Prints -113.0
config.integer_field = 10
print(config.ref_subtraction) # Prints 0.0
```
### Equality checking
You can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`
and `FrozenConfigDict` objects.
```python
import ml_collections
dict_1 = {'list': [1, 2]}
dict_2 = {'list': (1, 2)}
cfg_1 = ml_collections.ConfigDict(dict_1)
frozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)
frozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)
# True because FrozenConfigDict converts lists to tuples
print(frozen_cfg_1.items() == frozen_cfg_2.items())
# False because == distinguishes the underlying difference
print(frozen_cfg_1 == frozen_cfg_2)
# False because == distinguishes these types
print(frozen_cfg_1 == cfg_1)
# But eq_as_configdict() treats both as ConfigDict, so these are True:
print(frozen_cfg_1.eq_as_configdict(cfg_1))
print(cfg_1.eq_as_configdict(frozen_cfg_1))
```
### Equality checking with lazy computation
Equality checks see if the computed values are the same. Equality is satisfied
if two sets of computations are different as long as they result in the same
value.
```python
import ml_collections
cfg_1 = ml_collections.ConfigDict()
cfg_1.a = 1
cfg_1.b = cfg_1.get_ref('a') + 2
cfg_2 = ml_collections.ConfigDict()
cfg_2.a = 1
cfg_2.b = cfg_2.get_ref('a') * 3
# True because all computed values are the same
print(cfg_1 == cfg_2)
```
### Locking and copying
Here is an example with `lock()` and `deepcopy()`:
```python
import copy
import ml_collections
cfg = ml_collections.ConfigDict()
cfg.integer_field = 123
# Locking prohibits the addition and deletion of new fields but allows
# modification of existing values.
cfg.lock()
try:
cfg.integer_field = 124 # Raises AttributeError and suggests valid field.
except AttributeError as e:
print(e)
with cfg.unlocked():
cfg.integer_field = 1555 # Works fine too.
# Get a copy of the config dict.
new_cfg = copy.deepcopy(cfg)
new_cfg.integer_field = -123 # Works fine.
print(cfg)
```
### Dictionary attributes and initialization
```python
import ml_collections
referenced_dict = {'inner_float': 3.14}
d = {
'referenced_dict_1': referenced_dict,
'referenced_dict_2': referenced_dict,
'list_containing_dict': [{'key': 'value'}],
}
# We can initialize on a dictionary
cfg = ml_collections.ConfigDict(d)
# Reference structure is preserved
print(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True
# And the dict attributes have been converted to ConfigDict
print(type(cfg.referenced_dict_1)) # ConfigDict
# However, the initialization does not look inside of lists, so dicts inside
# lists are not converted to ConfigDict
print(type(cfg.list_containing_dict[0])) # dict
```
### More Examples
For more examples, take a look at
[`ml_collections/config_dict/examples/`](https://github.com/google/ml_collections/tree/master/ml_collections/config_dict/examples)
For examples and gotchas specifically about initializing a ConfigDict, see
[`ml_collections/config_dict/examples/config_dict_initialization.py`](https://github.com/google/ml_collections/blob/master/ml_collections/config_dict/examples/config_dict_initialization.py).
## Config Flags
This library adds flag definitions to `absl.flags` to handle config files. It
does not wrap `absl.flags` so if using any standard flag definitions alongside
config file flags, users must also import `absl.flags`.
Currently, this module adds two new flag types, namely `DEFINE_config_file`
which accepts a path to a Python file that generates a configuration, and
`DEFINE_config_dict` which accepts a configuration directly. Configurations are
dict-like structures (see [ConfigDict](#configdict)) whose nested elements
can be overridden using special command-line flags. See the examples below
for more details.
### Usage
Use `ml_collections.config_flags` alongside `absl.flags`. For
example:
`script.py`:
```python
from absl import app
from absl import flags
from ml_collections.config_flags import config_flags
FLAGS = flags.FLAGS
config_flags.DEFINE_config_file('my_config')
def main(_):
print(FLAGS.my_config)
if __name__ == '__main__':
app.run(main)
```
`config.py`:
```python
# Note that this is a valid Python script.
# get_config() can return an arbitrary dict-like object. However, it is advised
# to use ml_collections.ConfigDict.
# See ml_collections/config_dict/examples/config_dict_basic.py
import ml_collections
def get_config():
config = ml_collections.ConfigDict()
config.field1 = 1
config.field2 = 'tom'
config.nested = ml_collections.ConfigDict()
config.nested.field = 2.23
config.tuple = (1, 2, 3)
return config
```
Now, after running:
```bash
python script.py --my_config=config.py \
--my_config.field1=8 \
--my_config.nested.field=2.1 \
--my_config.tuple='(1, 2, (1, 2))'
```
we get:
```
field1: 8
field2: tom
nested:
field: 2.1
tuple: !!python/tuple
- 1
- 2
- !!python/tuple
- 1
- 2
```
Usage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main
difference is the configuration is defined in `script.py` instead of in a
separate file.
`script.py`:
```python
from absl import app
from absl import flags
import ml_collections
from ml_collections.config_flags import config_flags
config = ml_collections.ConfigDict()
config.field1 = 1
config.field2 = 'tom'
config.nested = ml_collections.ConfigDict()
config.nested.field = 2.23
config.tuple = (1, 2, 3)
FLAGS = flags.FLAGS
config_flags.DEFINE_config_dict('my_config', config)
def main(_):
print(FLAGS.my_config)
if __name__ == '__main__':
app.run()
```
`config_file` flags are compatible with the command-line flag syntax. All the
following options are supported for non-boolean values in configurations:
* `-(-)config.field=value`
* `-(-)config.field value`
Options for boolean values are slightly different:
* `-(-)config.boolean_field`: set boolean value to True.
* `-(-)noconfig.boolean_field`: set boolean value to False.
* `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or
`False`.
Note that `-(-)config.boolean_field value` is not supported.
### Parameterising the get_config() function
It's sometimes useful to be able to pass parameters into `get_config`, and
change what is returned based on this configuration. One example is if you are
grid searching over parameters which have a different hierarchical structure -
the flag needs to be present in the resulting ConfigDict. It would be possible
to include the union of all possible leaf values in your ConfigDict,
but this produces a confusing config result as you have to remember which
parameters will actually have an effect and which won't.
A better system is to pass some configuration, indicating which structure of
ConfigDict should be returned. An example is the following config file:
```python
import ml_collections
def get_config(config_string):
possible_structures = {
'linear': ml_collections.ConfigDict({
'model_constructor': 'snt.Linear',
'model_config': ml_collections.ConfigDict({
'output_size': 42,
}),
'lstm': ml_collections.ConfigDict({
'model_constructor': 'snt.LSTM',
'model_config': ml_collections.ConfigDict({
'hidden_size': 108,
})
})
}
return possible_structures[config_string]
```
The value of `config_string` will be anything that is to the right of the first
colon in the config file path, if one exists. If no colon exists, no value is
passed to `get_config` (producing a TypeError if `get_config` expects a value.)
The above example can be run like:
```bash
python script.py -- --config=path_to_config.py:linear \
--config.model_config.output_size=256
```
or like:
```bash
python script.py -- --config=path_to_config.py:lstm \
--config.model_config.hidden_size=512
```
### Additional features
* Loads any valid python script which defines `get_config()` function
returning any python object.
* Automatic locking of the loaded object, if the loaded object defines a
callable `.lock()` method.
* Supports command-line overriding of arbitrarily nested values in dict-like
objects (with key/attribute based getters/setters) of the following types:
* `types.IntType` (integer)
* `types.FloatType` (float)
* `types.BooleanType` (bool)
* `types.StringType` (string)
* `types.TupleType` (tuple)
* Overriding is type safe.
* Overriding of `TupleType` can be done by passing in the `tuple` as a string
(see the example in the [Usage](#usage) section).
* The overriding `tuple` object can be of a different size and have different
types than the original. Nested tuples are also supported.
## Authors
* Sergio Gómez Colmenarejo - sergomez@google.com
* Wojciech Marian Czarnecki - lejlot@google.com
* Nicholas Watters
* Mohit Reddy - mohitreddy@google.com
Raw data
{
"_id": null,
"home_page": "https://github.com/google/ml_collections",
"name": "ml-collections",
"maintainer": "",
"docs_url": null,
"requires_python": ">=2.6",
"maintainer_email": "",
"keywords": "",
"author": "ML Collections Authors",
"author_email": "ml-collections@google.com",
"download_url": "https://files.pythonhosted.org/packages/aa/ea/853aa32dfa1006d3eb43384712f35b8f2d6f0a757b8c779d40c29e3e8515/ml_collections-0.1.1.tar.gz",
"platform": "",
"description": "# ML Collections\n\nML Collections is a library of Python Collections designed for ML use cases.\n\n[](https://ml-collections.readthedocs.io/en/latest/?badge=latest)\n[](https://badge.fury.io/py/ml-collections)\n[](https://github.com/google/ml_collections/actions?query=workflow%3A%22Python+package%22)\n\n## ConfigDict\n\nThe two classes called `ConfigDict` and `FrozenConfigDict` are \"dict-like\" data\nstructures with dot access to nested elements. Together, they are supposed to be\nused as a main way of expressing configurations of experiments and models.\n\nThis document describes example usage of `ConfigDict`, `FrozenConfigDict`,\n`FieldReference`.\n\n### Features\n\n* Dot-based access to fields.\n* Locking mechanism to prevent spelling mistakes.\n* Lazy computation.\n* FrozenConfigDict() class which is immutable and hashable.\n* Type safety.\n* \"Did you mean\" functionality.\n* Human readable printing (with valid references and cycles), using valid YAML\n format.\n* Fields can be passed as keyword arguments using the `**` operator.\n* There are two exceptions to the strong type-safety of the ConfigDict. `int`\n values can be passed in to fields of type `float`. In such a case, the value\n is type-converted to a `float` before being stored. Similarly, all string\n types (including Unicode strings) can be stored in fields of type `str` or\n `unicode`.\n\n### Basic Usage\n\n```python\nimport ml_collections\n\ncfg = ml_collections.ConfigDict()\ncfg.float_field = 12.6\ncfg.integer_field = 123\ncfg.another_integer_field = 234\ncfg.nested = ml_collections.ConfigDict()\ncfg.nested.string_field = 'tom'\n\nprint(cfg.integer_field) # Prints 123.\nprint(cfg['integer_field']) # Prints 123 as well.\n\ntry:\n cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.\nexcept TypeError as e:\n print(e)\n\ncfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.\ncfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.\n\nprint(cfg)\n```\n\n### FrozenConfigDict\n\nA `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:\n\n```python\nimport ml_collections\n\ninitial_dictionary = {\n 'int': 1,\n 'list': [1, 2],\n 'tuple': (1, 2, 3),\n 'set': {1, 2, 3, 4},\n 'dict_tuple_list': {'tuple_list': ([1, 2], 3)}\n}\n\ncfg = ml_collections.ConfigDict(initial_dictionary)\nfrozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)\n\nprint(frozen_dict.tuple) # Prints tuple (1, 2, 3)\nprint(frozen_dict.list) # Prints tuple (1, 2)\nprint(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}\nprint(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)\n\nfrozen_cfg = ml_collections.FrozenConfigDict(cfg)\nprint(frozen_cfg == frozen_dict) # True\nprint(hash(frozen_cfg) == hash(frozen_dict)) # True\n\ntry:\n frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.\nexcept AttributeError as e:\n print(e)\n\n# Converting between `FrozenConfigDict` and `ConfigDict`:\nthawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)\nprint(thawed_frozen_cfg == cfg) # True\nfrozen_cfg_to_cfg = frozen_dict.as_configdict()\nprint(frozen_cfg_to_cfg == cfg) # True\n```\n\n### FieldReferences and placeholders\n\nA `FieldReference` is useful for having multiple fields use the same value. It\ncan also be used for [lazy computation](#lazy-computation).\n\nYou can use `placeholder()` as a shortcut to create a `FieldReference` (field)\nwith a `None` default value. This is useful if a program uses optional\nconfiguration fields.\n\n```python\nimport ml_collections\nfrom ml_collections.config_dict import config_dict\n\nplaceholder = ml_collections.FieldReference(0)\ncfg = ml_collections.ConfigDict()\ncfg.placeholder = placeholder\ncfg.optional = config_dict.placeholder(int)\ncfg.nested = ml_collections.ConfigDict()\ncfg.nested.placeholder = placeholder\n\ntry:\n cfg.optional = 'tom' # Raises Type error as this field is an integer.\nexcept TypeError as e:\n print(e)\n\ncfg.optional = 1555 # Works fine.\ncfg.placeholder = 1 # Changes the value of both placeholder and\n # nested.placeholder fields.\n\nprint(cfg)\n```\n\nNote that the indirection provided by `FieldReference`s will be lost if accessed\nthrough a `ConfigDict`.\n\n```python\nimport ml_collections\n\nplaceholder = ml_collections.FieldReference(0)\ncfg.field1 = placeholder\ncfg.field2 = placeholder # This field will be tied to cfg.field1.\ncfg.field3 = cfg.field1 # This will just be an int field initialized to 0.\n```\n\n### Lazy computation\n\nUsing a `FieldReference` in a standard operation (addition, subtraction,\nmultiplication, etc...) will return another `FieldReference` that points to the\noriginal's value. You can use `FieldReference.get()` to execute the operations\nand get the reference's computed value, and `FieldReference.set()` to change the\noriginal reference's value.\n\n```python\nimport ml_collections\n\nref = ml_collections.FieldReference(1)\nprint(ref.get()) # Prints 1\n\nadd_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten\nadd_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer\n\nprint(add_ten) # Prints 11\nprint(add_ten_lazy.get()) # Prints 11 because ref's value is 1\n\n# Addition is lazily computed for FieldReferences so changing ref will change\n# the value that is used to compute add_ten.\nref.set(5)\nprint(add_ten) # Prints 11\nprint(add_ten_lazy.get()) # Prints 15 because ref's value is 5\n```\n\nIf a `FieldReference` has `None` as its original value, or any operation has an\nargument of `None`, then the lazy computation will evaluate to `None`.\n\nWe can also use fields in a `ConfigDict` in lazy computation. In this case a\nfield will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.\n\n```python\nimport ml_collections\n\nconfig = ml_collections.ConfigDict()\nconfig.reference_field = ml_collections.FieldReference(1)\nconfig.integer_field = 2\nconfig.float_field = 2.5\n\n# No lazy evaluatuations because we didn't use get_ref()\nconfig.no_lazy = config.integer_field * config.float_field\n\n# This will lazily evaluate ONLY config.integer_field\nconfig.lazy_integer = config.get_ref('integer_field') * config.float_field\n\n# This will lazily evaluate ONLY config.float_field\nconfig.lazy_float = config.integer_field * config.get_ref('float_field')\n\n# This will lazily evaluate BOTH config.integer_field and config.float_Field\nconfig.lazy_both = (config.get_ref('integer_field') *\n config.get_ref('float_field'))\n\nconfig.integer_field = 3\nprint(config.no_lazy) # Prints 5.0 - It uses integer_field's original value\n\nprint(config.lazy_integer) # Prints 7.5\n\nconfig.float_field = 3.5\nprint(config.lazy_float) # Prints 7.0\nprint(config.lazy_both) # Prints 10.5\n```\n\n#### Changing lazily computed values\n\nLazily computed values in a ConfigDict can be overridden in the same way as\nregular values. The reference to the `FieldReference` used for the lazy\ncomputation will be lost and all computations downstream in the reference graph\nwill use the new value.\n\n```python\nimport ml_collections\n\nconfig = ml_collections.ConfigDict()\nconfig.reference = 1\nconfig.reference_0 = config.get_ref('reference') + 10\nconfig.reference_1 = config.get_ref('reference') + 20\nconfig.reference_1_0 = config.get_ref('reference_1') + 100\n\nprint(config.reference) # Prints 1.\nprint(config.reference_0) # Prints 11.\nprint(config.reference_1) # Prints 21.\nprint(config.reference_1_0) # Prints 121.\n\nconfig.reference_1 = 30\n\nprint(config.reference) # Prints 1 (unchanged).\nprint(config.reference_0) # Prints 11 (unchanged).\nprint(config.reference_1) # Prints 30.\nprint(config.reference_1_0) # Prints 130.\n```\n\n#### Cycles\n\nYou cannot create cycles using references. Fortunately\n[the only way](#changing-lazily-computed-values) to create a cycle is by\nassigning a computed field to one that *is not* the result of computation. This\nis forbidden:\n\n```python\nimport ml_collections\nfrom ml_collections.config_dict import config_dict\n\nconfig = ml_collections.ConfigDict()\nconfig.integer_field = 1\nconfig.bigger_integer_field = config.get_ref('integer_field') + 10\n\ntry:\n # Raises a MutabilityError because setting config.integer_field would\n # cause a cycle.\n config.integer_field = config.get_ref('bigger_integer_field') + 2\nexcept config_dict.MutabilityError as e:\n print(e)\n```\n\n### Advanced usage\n\nHere are some more advanced examples showing lazy computation with different\noperators and data types.\n\n```python\nimport ml_collections\n\nconfig = ml_collections.ConfigDict()\nconfig.float_field = 12.6\nconfig.integer_field = 123\nconfig.list_field = [0, 1, 2]\n\nconfig.float_multiply_field = config.get_ref('float_field') * 3\nprint(config.float_multiply_field) # Prints 37.8\n\nconfig.float_field = 10.0\nprint(config.float_multiply_field) # Prints 30.0\n\nconfig.longer_list_field = config.get_ref('list_field') + [3, 4, 5]\nprint(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]\n\nconfig.list_field = [-1]\nprint(config.longer_list_field) # Prints [-1, 3, 4, 5]\n\n# Both operands can be references\nconfig.ref_subtraction = (\n config.get_ref('float_field') - config.get_ref('integer_field'))\nprint(config.ref_subtraction) # Prints -113.0\n\nconfig.integer_field = 10\nprint(config.ref_subtraction) # Prints 0.0\n```\n\n### Equality checking\n\nYou can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`\nand `FrozenConfigDict` objects.\n\n```python\nimport ml_collections\n\ndict_1 = {'list': [1, 2]}\ndict_2 = {'list': (1, 2)}\ncfg_1 = ml_collections.ConfigDict(dict_1)\nfrozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)\nfrozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)\n\n# True because FrozenConfigDict converts lists to tuples\nprint(frozen_cfg_1.items() == frozen_cfg_2.items())\n# False because == distinguishes the underlying difference\nprint(frozen_cfg_1 == frozen_cfg_2)\n\n# False because == distinguishes these types\nprint(frozen_cfg_1 == cfg_1)\n# But eq_as_configdict() treats both as ConfigDict, so these are True:\nprint(frozen_cfg_1.eq_as_configdict(cfg_1))\nprint(cfg_1.eq_as_configdict(frozen_cfg_1))\n```\n\n### Equality checking with lazy computation\n\nEquality checks see if the computed values are the same. Equality is satisfied\nif two sets of computations are different as long as they result in the same\nvalue.\n\n```python\nimport ml_collections\n\ncfg_1 = ml_collections.ConfigDict()\ncfg_1.a = 1\ncfg_1.b = cfg_1.get_ref('a') + 2\n\ncfg_2 = ml_collections.ConfigDict()\ncfg_2.a = 1\ncfg_2.b = cfg_2.get_ref('a') * 3\n\n# True because all computed values are the same\nprint(cfg_1 == cfg_2)\n```\n\n### Locking and copying\n\nHere is an example with `lock()` and `deepcopy()`:\n\n```python\nimport copy\nimport ml_collections\n\ncfg = ml_collections.ConfigDict()\ncfg.integer_field = 123\n\n# Locking prohibits the addition and deletion of new fields but allows\n# modification of existing values.\ncfg.lock()\ntry:\n cfg.integer_field = 124 # Raises AttributeError and suggests valid field.\nexcept AttributeError as e:\n print(e)\nwith cfg.unlocked():\n cfg.integer_field = 1555 # Works fine too.\n\n# Get a copy of the config dict.\nnew_cfg = copy.deepcopy(cfg)\nnew_cfg.integer_field = -123 # Works fine.\n\nprint(cfg)\n```\n\n### Dictionary attributes and initialization\n\n```python\nimport ml_collections\n\nreferenced_dict = {'inner_float': 3.14}\nd = {\n 'referenced_dict_1': referenced_dict,\n 'referenced_dict_2': referenced_dict,\n 'list_containing_dict': [{'key': 'value'}],\n}\n\n# We can initialize on a dictionary\ncfg = ml_collections.ConfigDict(d)\n\n# Reference structure is preserved\nprint(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True\n\n# And the dict attributes have been converted to ConfigDict\nprint(type(cfg.referenced_dict_1)) # ConfigDict\n\n# However, the initialization does not look inside of lists, so dicts inside\n# lists are not converted to ConfigDict\nprint(type(cfg.list_containing_dict[0])) # dict\n```\n\n### More Examples\n\nFor more examples, take a look at\n[`ml_collections/config_dict/examples/`](https://github.com/google/ml_collections/tree/master/ml_collections/config_dict/examples)\n\nFor examples and gotchas specifically about initializing a ConfigDict, see\n[`ml_collections/config_dict/examples/config_dict_initialization.py`](https://github.com/google/ml_collections/blob/master/ml_collections/config_dict/examples/config_dict_initialization.py).\n\n## Config Flags\n\nThis library adds flag definitions to `absl.flags` to handle config files. It\ndoes not wrap `absl.flags` so if using any standard flag definitions alongside\nconfig file flags, users must also import `absl.flags`.\n\nCurrently, this module adds two new flag types, namely `DEFINE_config_file`\nwhich accepts a path to a Python file that generates a configuration, and\n`DEFINE_config_dict` which accepts a configuration directly. Configurations are\ndict-like structures (see [ConfigDict](#configdict)) whose nested elements\ncan be overridden using special command-line flags. See the examples below\nfor more details.\n\n### Usage\n\nUse `ml_collections.config_flags` alongside `absl.flags`. For\nexample:\n\n`script.py`:\n\n```python\nfrom absl import app\nfrom absl import flags\n\nfrom ml_collections.config_flags import config_flags\n\nFLAGS = flags.FLAGS\nconfig_flags.DEFINE_config_file('my_config')\n\ndef main(_):\n print(FLAGS.my_config)\n\nif __name__ == '__main__':\n app.run(main)\n```\n\n`config.py`:\n\n```python\n# Note that this is a valid Python script.\n# get_config() can return an arbitrary dict-like object. However, it is advised\n# to use ml_collections.ConfigDict.\n# See ml_collections/config_dict/examples/config_dict_basic.py\n\nimport ml_collections\n\ndef get_config():\n config = ml_collections.ConfigDict()\n config.field1 = 1\n config.field2 = 'tom'\n config.nested = ml_collections.ConfigDict()\n config.nested.field = 2.23\n config.tuple = (1, 2, 3)\n return config\n```\n\nNow, after running:\n\n```bash\npython script.py --my_config=config.py \\\n --my_config.field1=8 \\\n --my_config.nested.field=2.1 \\\n --my_config.tuple='(1, 2, (1, 2))'\n```\n\nwe get:\n\n```\nfield1: 8\nfield2: tom\nnested:\n field: 2.1\ntuple: !!python/tuple\n- 1\n- 2\n- !!python/tuple\n - 1\n - 2\n```\n\nUsage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main\ndifference is the configuration is defined in `script.py` instead of in a\nseparate file.\n\n`script.py`:\n\n```python\nfrom absl import app\nfrom absl import flags\n\nimport ml_collections\nfrom ml_collections.config_flags import config_flags\n\nconfig = ml_collections.ConfigDict()\nconfig.field1 = 1\nconfig.field2 = 'tom'\nconfig.nested = ml_collections.ConfigDict()\nconfig.nested.field = 2.23\nconfig.tuple = (1, 2, 3)\n\nFLAGS = flags.FLAGS\nconfig_flags.DEFINE_config_dict('my_config', config)\n\ndef main(_):\n print(FLAGS.my_config)\n\nif __name__ == '__main__':\n app.run()\n```\n\n`config_file` flags are compatible with the command-line flag syntax. All the\nfollowing options are supported for non-boolean values in configurations:\n\n* `-(-)config.field=value`\n* `-(-)config.field value`\n\nOptions for boolean values are slightly different:\n\n* `-(-)config.boolean_field`: set boolean value to True.\n* `-(-)noconfig.boolean_field`: set boolean value to False.\n* `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or\n `False`.\n\nNote that `-(-)config.boolean_field value` is not supported.\n\n### Parameterising the get_config() function\n\nIt's sometimes useful to be able to pass parameters into `get_config`, and\nchange what is returned based on this configuration. One example is if you are\ngrid searching over parameters which have a different hierarchical structure -\nthe flag needs to be present in the resulting ConfigDict. It would be possible\nto include the union of all possible leaf values in your ConfigDict,\nbut this produces a confusing config result as you have to remember which\nparameters will actually have an effect and which won't.\n\nA better system is to pass some configuration, indicating which structure of\nConfigDict should be returned. An example is the following config file:\n\n```python\nimport ml_collections\n\ndef get_config(config_string):\n possible_structures = {\n 'linear': ml_collections.ConfigDict({\n 'model_constructor': 'snt.Linear',\n 'model_config': ml_collections.ConfigDict({\n 'output_size': 42,\n }),\n 'lstm': ml_collections.ConfigDict({\n 'model_constructor': 'snt.LSTM',\n 'model_config': ml_collections.ConfigDict({\n 'hidden_size': 108,\n })\n })\n }\n\n return possible_structures[config_string]\n```\n\nThe value of `config_string` will be anything that is to the right of the first\ncolon in the config file path, if one exists. If no colon exists, no value is\npassed to `get_config` (producing a TypeError if `get_config` expects a value.)\n\nThe above example can be run like:\n\n```bash\npython script.py -- --config=path_to_config.py:linear \\\n --config.model_config.output_size=256\n```\n\nor like:\n\n```bash\npython script.py -- --config=path_to_config.py:lstm \\\n --config.model_config.hidden_size=512\n```\n\n### Additional features\n\n* Loads any valid python script which defines `get_config()` function\n returning any python object.\n* Automatic locking of the loaded object, if the loaded object defines a\n callable `.lock()` method.\n* Supports command-line overriding of arbitrarily nested values in dict-like\n objects (with key/attribute based getters/setters) of the following types:\n * `types.IntType` (integer)\n * `types.FloatType` (float)\n * `types.BooleanType` (bool)\n * `types.StringType` (string)\n * `types.TupleType` (tuple)\n* Overriding is type safe.\n* Overriding of `TupleType` can be done by passing in the `tuple` as a string\n (see the example in the [Usage](#usage) section).\n* The overriding `tuple` object can be of a different size and have different\n types than the original. Nested tuples are also supported.\n\n## Authors\n* Sergio G\u00f3mez Colmenarejo - sergomez@google.com\n* Wojciech Marian Czarnecki - lejlot@google.com\n* Nicholas Watters\n* Mohit Reddy - mohitreddy@google.com\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "ML Collections is a library of Python collections designed for ML usecases.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/google/ml_collections"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "aaea853aa32dfa1006d3eb43384712f35b8f2d6f0a757b8c779d40c29e3e8515",
"md5": "28028080e395d496e8267db1a429f5a4",
"sha256": "3fefcc72ec433aa1e5d32307a3e474bbb67f405be814ea52a2166bfc9dbe68cc"
},
"downloads": -1,
"filename": "ml_collections-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "28028080e395d496e8267db1a429f5a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.6",
"size": 77915,
"upload_time": "2022-01-27T12:39:20",
"upload_time_iso_8601": "2022-01-27T12:39:20.918689Z",
"url": "https://files.pythonhosted.org/packages/aa/ea/853aa32dfa1006d3eb43384712f35b8f2d6f0a757b8c779d40c29e3e8515/ml_collections-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-01-27 12:39:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "google",
"github_project": "ml_collections",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "ml-collections"
}
JSON
