# Flytekit OmegaConf Plugin
Flytekit python natively supports serialization of many data types for exchanging information between tasks.
The Flytekit OmegaConf Plugin extends these by the `DictConfig` type from the
[OmegaConf package](https://omegaconf.readthedocs.io/) as well as related types
that are being used by the [hydra package](https://hydra.cc/) for configuration management.
## Task example
```
from dataclasses import dataclass
import flytekitplugins.omegaconf # noqa F401
from flytekit import task, workflow
from omegaconf import DictConfig
@dataclass
class MySimpleConf:
_target_: str = "lightning_module.MyEncoderModule"
learning_rate: float = 0.0001
@task
def my_task(cfg: DictConfig) -> None:
print(f"Doing things with {cfg.learning_rate=}")
@workflow
def pipeline(cfg: DictConfig) -> None:
my_task(cfg=cfg)
if __name__ == "__main__":
from omegaconf import OmegaConf
cfg = OmegaConf.structured(MySimpleConf)
pipeline(cfg=cfg)
```
## Transformer configuration
The transformer can be set to one of three modes:
`Dataclass` - This mode should be used with a StructuredConfig and will reconstruct the config from the matching dataclass
during deserialisation in order to make typing information from the dataclass and continued validation thereof available.
This requires the dataclass definition to be available via python import in the Flyte execution environment in which
objects are (de-)serialised.
`DictConfig` - This mode will deserialize the config into a DictConfig object. In particular, dataclasses are translated
into DictConfig objects and only primitive types are being checked. The definition of underlying dataclasses for
structured configs is only required during the initial serialization for this mode.
`Auto` - This mode will try to deserialize according to the Dataclass mode and fall back to the DictConfig mode if the
dataclass definition is not available. This is the default mode.
You can set the transformer mode globally or for the current context only the following ways:
```python
from flytekitplugins.omegaconf import set_transformer_mode, set_local_transformer_mode, OmegaConfTransformerMode
# Set the global transformer mode using the new function
set_transformer_mode(OmegaConfTransformerMode.DictConfig)
# You can also the mode for the current context only
with set_local_transformer_mode(OmegaConfTransformerMode.Dataclass):
# This will use the Dataclass mode
pass
```
```note
Since the DictConfig is flattened and keys transformed into dot notation, the keys of the DictConfig must not contain
dots.
```
Raw data
{
"_id": null,
"home_page": "https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-omegaconf",
"name": "flytekitplugins-omegaconf",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "flyteorg",
"author_email": "admin@flyte.org",
"download_url": "https://files.pythonhosted.org/packages/58/9d/66e858909022a543930978c38fc1c0a14f024ed09b8f48ff1f4aed4b4c88/flytekitplugins_omegaconf-1.14.3.tar.gz",
"platform": null,
"description": "# Flytekit OmegaConf Plugin\n\nFlytekit python natively supports serialization of many data types for exchanging information between tasks.\nThe Flytekit OmegaConf Plugin extends these by the `DictConfig` type from the\n[OmegaConf package](https://omegaconf.readthedocs.io/) as well as related types\nthat are being used by the [hydra package](https://hydra.cc/) for configuration management.\n\n## Task example\n```\nfrom dataclasses import dataclass\nimport flytekitplugins.omegaconf # noqa F401\nfrom flytekit import task, workflow\nfrom omegaconf import DictConfig\n\n@dataclass\nclass MySimpleConf:\n _target_: str = \"lightning_module.MyEncoderModule\"\n learning_rate: float = 0.0001\n\n@task\ndef my_task(cfg: DictConfig) -> None:\n print(f\"Doing things with {cfg.learning_rate=}\")\n\n\n@workflow\ndef pipeline(cfg: DictConfig) -> None:\n my_task(cfg=cfg)\n\n\nif __name__ == \"__main__\":\n from omegaconf import OmegaConf\n\n cfg = OmegaConf.structured(MySimpleConf)\n pipeline(cfg=cfg)\n```\n\n## Transformer configuration\n\nThe transformer can be set to one of three modes:\n\n`Dataclass` - This mode should be used with a StructuredConfig and will reconstruct the config from the matching dataclass\nduring deserialisation in order to make typing information from the dataclass and continued validation thereof available.\nThis requires the dataclass definition to be available via python import in the Flyte execution environment in which\nobjects are (de-)serialised.\n\n`DictConfig` - This mode will deserialize the config into a DictConfig object. In particular, dataclasses are translated\ninto DictConfig objects and only primitive types are being checked. The definition of underlying dataclasses for\nstructured configs is only required during the initial serialization for this mode.\n\n`Auto` - This mode will try to deserialize according to the Dataclass mode and fall back to the DictConfig mode if the\ndataclass definition is not available. This is the default mode.\n\nYou can set the transformer mode globally or for the current context only the following ways:\n```python\nfrom flytekitplugins.omegaconf import set_transformer_mode, set_local_transformer_mode, OmegaConfTransformerMode\n\n# Set the global transformer mode using the new function\nset_transformer_mode(OmegaConfTransformerMode.DictConfig)\n\n# You can also the mode for the current context only\nwith set_local_transformer_mode(OmegaConfTransformerMode.Dataclass):\n # This will use the Dataclass mode\n pass\n```\n\n```note\nSince the DictConfig is flattened and keys transformed into dot notation, the keys of the DictConfig must not contain\ndots.\n```\n",
"bugtrack_url": null,
"license": "apache2",
"summary": "OmegaConf plugin for Flytekit",
"version": "1.14.3",
"project_urls": {
"Homepage": "https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-omegaconf"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fd48a41623e4a4efb947b17d120567dd9c684956f638e50e12c088e5a6702fa2",
"md5": "13ebf2a2bbf2343accb1cc914a6908c8",
"sha256": "50cfb961ec14bba89c70ea65efee87ac13c64b5a7a9732a427eafef793eec62d"
},
"downloads": -1,
"filename": "flytekitplugins_omegaconf-1.14.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "13ebf2a2bbf2343accb1cc914a6908c8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 10636,
"upload_time": "2024-12-26T22:43:50",
"upload_time_iso_8601": "2024-12-26T22:43:50.271486Z",
"url": "https://files.pythonhosted.org/packages/fd/48/a41623e4a4efb947b17d120567dd9c684956f638e50e12c088e5a6702fa2/flytekitplugins_omegaconf-1.14.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "589d66e858909022a543930978c38fc1c0a14f024ed09b8f48ff1f4aed4b4c88",
"md5": "e98ad522359d6ddaaeaca82dad3ec543",
"sha256": "dd52fb4af8313e53aaa7888d3bf12bc0b6ec64d90750fe1de6dc917a0d98803a"
},
"downloads": -1,
"filename": "flytekitplugins_omegaconf-1.14.3.tar.gz",
"has_sig": false,
"md5_digest": "e98ad522359d6ddaaeaca82dad3ec543",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 12060,
"upload_time": "2024-12-26T22:44:43",
"upload_time_iso_8601": "2024-12-26T22:44:43.971012Z",
"url": "https://files.pythonhosted.org/packages/58/9d/66e858909022a543930978c38fc1c0a14f024ed09b8f48ff1f4aed4b4c88/flytekitplugins_omegaconf-1.14.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-26 22:44:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "flyteorg",
"github_project": "flytekit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "flytekitplugins-omegaconf"
}
JSON
