# CAEP
Configuration library that supports loading configuration from ini, environment variables
and arguments into a [pydantic](https://docs.pydantic.dev/) schema.
With the pydantic schema you will have a fully typed configuration object that is parsed
at load time.
# Change log
## 1.0.0
Support for pydantic 2.x. It is advised to migrate models with these changes:
### Use `min_length` instead of `min_size`
Pydantic has builtin support for size of list, dictionaries and sets using `min_length` so you should change
```python
intlist: List[int] = Field(description="Space separated list of ints", min_size=1)
```
to
```python
intlist: List[int] = Field(description="Space separated list of ints", min_length=1)
```
### Migrate `split` and `kv_split` to `json_schema_extra`
Do not use `split` and `kv_split` directly on the field, but put them in a dictionary `json_schema_extra`. E.g. change
```python
intlist: List[int] = Field(description="Space separated list of ints", split=" ")
```
to
```python
intlist: List[int] = Field(
description="Space separated list of ints", json_schema_extra={"split": " "}
)
```
and change
```python
dict_int: Dict[str, int] = Field(
description="Int Dict split by slash and dash", split="-", kv_split="/"
)
```
to
```python
dict_int: Dict[str, int] = Field(
description="Int Dict split by slash and dash",
json_schema_extra={"split": "-", "kv_split": "/"},
)
```
### Migrate `root_validator` to `model_validator`
`root_validator` are stil supported, but it is advised to migrate to `model_validator`. Example using helper function `raise_if_some_and_not_all`:
```python
@model_validator(mode="after") # type: ignore
def check_arguments(cls, m: "ExampleConfig") -> "ExampleConfig":
"""If one argument is set, they should all be set"""
caep.raise_if_some_and_not_all(
m.__dict__, ["username", "password", "parent_id"]
)
return m
```
# Example
```python
#!/usr/bin/env python3
from typing import List
from pydantic import BaseModel, Field
import caep
class Config(BaseModel):
text: str = Field(description="Required String Argument")
number: int = Field(default=1, description="Integer with default value")
switch: bool = Field(description="Boolean with default value")
intlist: List[int] = Field(description="Space separated list of ints", json_schema_extra={"split": " "})
# Config/section options below will only be used if loading configuration
# from ini file (under ~/.config)
config = caep.load(
Config,
"CAEP Example",
"caep", # Find .ini file under ~/.config/caep
"caep.ini", # Find .ini file name caep.ini
"section", # Load settings from [section] (default to [DEFAULT]
)
print(config)
```
Sample output with a `intlist` read from environment and `switch` from command line:
```bash
$ export INTLIST="1 2 3"
$ ./example.py --text "My value" --switch
text='My value' number=1 switch=True intlist=[1, 2, 3]
```
# Load config without ini support
Specifying configuration location, name and section is optional and can be skipped if you
do not want to support loading ini files from `$XDG_CONFIG_HOME`:
```python
# Only load arguments from environment and command line
config = caep.load(
Config,
"CAEP Example",
)
```
With the code above you can still specify a ini file with `--config <ini-file>`, and use
environment variables and command line arguments.
# Pydantic field types
Pydantic fields should be defined using `Field` and include the `description` parameter
to specify help text for the commandline.
Unless the `Field` has a `default` value, it is a required field that needs to be
specified in the environment, configuration file or on the command line.
Many of the types described in [https://docs.pydantic.dev/usage/types/](https://docs.pydantic.dev/usage/types/)
should be supported, but not all of them are tested. However, nested schemas
are *not* supported.
Tested types:
### `str`
Standard string argument.
### `int`
Values parsed as integer.
### `float`
Value parsed as float.
### `pathlib.Path`
Value parsed as Path.
### `ipaddress.IPv4Address`
Values parsed and validated as IPv4Address.
### `ipaddress.IPv4Network`
Values parsed and validated as IPv4Network.
### `bool`
Value parsed as booleans. Booleans will default to False, if no default value is set.
Examples:
| Field | Input | Configuration |
| - | - | - |
| `enable: bool = Field(description="Enable")` | <NOT SET> | False |
| `enable: bool = Field(value=False, description="Enable")` | `yes` | True |
| `enable: bool = Field(value=False, description="Enable")` | `true` | True |
| `disable: bool = Field(value=True, description="Disable")` | <NOT SET> | True |
| `disable: bool = Field(value=True, description="Disable")` | `yes` | False |
| `disable: bool = Field(value=True, description="Disable")` | `true` | False |
### `List[str]` (`list[str]` for python >= 3.9)
List of strings, split by specified character (default = comma, argument=`split`).
Some examples:
| Field | Input | Configuration |
| - | - | - |
| `List[int] = Field(description="Ints", json_schema_extra={"split": " "})` | `1 2` | [1, 2] |
| `List[str] = Field(description="Strs")` | `ab,bc` | ["ab", "bc"] |
The argument `min_length` (pydantic builtin) can be used to specify the minimum size of the list:
| Field | Input | Configuration |
| - | - | - |
| `List[str] = Field(description="Strs", min_length=1)` | `` | Raises ValidationError |
### `Set[str]` (`set[str]` for python >= 3.9)
Set, split by specified character (default = comma, argument=`split`).
Some examples:
| Field | Input | Configuration |
| - | - | - |
| `Set[int] = Field(description="Ints", json_schema_extra={"split": " "})` | `1 2 2` | {1, 2} |
| `Set[str] = Field(description="Strs")` | `ab,ab,xy` | {"ab", "xy"} |
The argument `min_length` can be used to specify the minimum size of the set:
| Field | Input | Configuration |
| - | - | - |
| `Set[str] = Field(description="Strs", min_length=1)` | `` | Raises ValidationError |
### `Dict[str, <TYPE>]` (`dict[str, <TYPE>]` for python >= 3.9)
Dictioray of strings, split by specified character (default = comma, argument=`split` for
splitting items and colon for splitting key/value).
Some examples:
| Field | Input | Configuration |
| - | - | - |
| `Dict[str, str] = Field(description="Dict")` | `x:a,y:b` | {"x": "a", "y": "b"} |
| `Dict[str, int] = Field(description="Dict of ints")` | `a b c:1, d e f:2` | {"a b c": 1, "d e f": 2} |
The argument `min_length` can be used to specify the minimum numer of keys in the dictionary:
| Field | Input | Configuration |
| - | - | - |
| `Dict[str, str] = Field(description="Strs", min_length=1)` | `` | Raises ValidationError |
# Configuration
Arguments are parsed in two phases. First, it will look for the optional argument `--config`
which can be used to specify an alternative location for the ini file. If not `--config` argument
is given it will look for an optional ini file in the following locations
(`~/.config has presedence`) *if* `config_id` and `config_name` is specified:
- `~/.config/<CONFIG_ID>/<CONFIG_FILE_NAME>` (or directory specified by `$XDG_CONFIG_HOME`)
- `/etc/<CONFIG_FILE_NAME>`
The ini file can contain a `[DEFAULT]` section that will be used for all configurations.
In addition it can have a section that corresponds with `<SECTION_NAME>` (if specified) that for
specific configuration, that will over override config from `[DEFAULT]`
# Environment variables
The configuration step will also look for environment variables in uppercase and
with `-` replaced with `_`. For the example below it will lookup the following environment
variables:
- $NUMBER
- $BOOL
- $STR_ARG
The configuration presedence are (from lowest to highest):
* argparse default
* ini file
* environment variable
* command line argument
## Validation
## XDG
Helper functions to use [XDG Base Directories](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) are included in `caep.xdg`:
It will look up `XDG` environment variables like `$XDG_CONFIG_HOME` and use
defaults if not specified.
### `get_xdg_dir`
Generic function to get a `XDG` directory.
The following example with will return a path object to ~/.config/myprog
(if `$XDG_CONFIG_HOME` is not set) and create the directoy if it does not
exist.
```python
get_xdg_dir("myprog", "XDG_CONFIG_HOME", ".config", True)
```
### `get_config_dir`
Shortcut for `get_xdg_dir("CONFIG")`.
### `get_cache_dir`
Shortcut for `get_xdg_dir("CACHE")`.
## CAEP Legacy usage
Prior to version `0.1.0` the recommend usage was to add parser objects manually. This is
still supported, but with this approach you will not get the validation from pydantic:
```python
>>> import caep
>>> import argparse
>>> parser = argparse.ArgumentParser("test argparse")
>>> parser.add_argument('--number', type=int, default=1)
>>> parser.add_argument('--bool', action='store_true')
>>> parser.add_argument('--str-arg')
>>> args = caep.config.handle_args(parser, <CONFIG_ID>, <CONFIG_FILE_NAME>, <SECTION_NAME>)
```
# Helper Functions
## raise_if_some_and_not_all
Raise ArgumentError if some of the specified entries in the dictionary has non
false values but not all
```python
class ExampleConfig(BaseModel):
username: Optional[str] = Field(description="Username")
password: Optional[str] = Field(description="Password")
parent_id: Optional[str] = Field(description="Parent ID")
@model_validator(mode="after") # type: ignore
def check_arguments(cls, m: "ExampleConfig") -> "ExampleConfig":
"""If one argument is set, they should all be set"""
caep.raise_if_some_and_not_all(
m.__dict__, ["username", "password", "parent_id"]
)
return m
```
## script_name
Return first external module that called this function, directly, or indirectly
Raw data
{
"_id": null,
"home_page": "https://github.com/mnemonic-no/caep",
"name": "caep",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6, <4",
"maintainer_email": "",
"keywords": "mnemonic",
"author": "mnemonic AS",
"author_email": "opensource@mnemonic.no",
"download_url": "https://files.pythonhosted.org/packages/05/69/b8f2217e9bb0d6ac6a5de64b90731c8aa43b735a7501722250ae6e1cb169/caep-1.0.0.tar.gz",
"platform": null,
"description": "# CAEP\n\nConfiguration library that supports loading configuration from ini, environment variables\nand arguments into a [pydantic](https://docs.pydantic.dev/) schema.\n\nWith the pydantic schema you will have a fully typed configuration object that is parsed\nat load time.\n\n# Change log\n\n## 1.0.0\n\nSupport for pydantic 2.x. It is advised to migrate models with these changes:\n\n### Use `min_length` instead of `min_size`\n\nPydantic has builtin support for size of list, dictionaries and sets using `min_length` so you should change\n```python\nintlist: List[int] = Field(description=\"Space separated list of ints\", min_size=1)\n```\n\nto\n\n```python\nintlist: List[int] = Field(description=\"Space separated list of ints\", min_length=1)\n```\n\n### Migrate `split` and `kv_split` to `json_schema_extra`\n\nDo not use `split` and `kv_split` directly on the field, but put them in a dictionary `json_schema_extra`. E.g. change\n\n```python\nintlist: List[int] = Field(description=\"Space separated list of ints\", split=\" \")\n```\n\nto\n\n```python\nintlist: List[int] = Field(\n description=\"Space separated list of ints\", json_schema_extra={\"split\": \" \"}\n)\n```\n\nand change\n\n```python\ndict_int: Dict[str, int] = Field(\n description=\"Int Dict split by slash and dash\", split=\"-\", kv_split=\"/\"\n)\n```\n\nto\n\n```python\ndict_int: Dict[str, int] = Field(\n description=\"Int Dict split by slash and dash\",\n json_schema_extra={\"split\": \"-\", \"kv_split\": \"/\"},\n)\n```\n\n### Migrate `root_validator` to `model_validator`\n\n`root_validator` are stil supported, but it is advised to migrate to `model_validator`. Example using helper function `raise_if_some_and_not_all`:\n\n\n```python\n @model_validator(mode=\"after\") # type: ignore\n def check_arguments(cls, m: \"ExampleConfig\") -> \"ExampleConfig\":\n \"\"\"If one argument is set, they should all be set\"\"\"\n\n caep.raise_if_some_and_not_all(\n m.__dict__, [\"username\", \"password\", \"parent_id\"]\n )\n\n return m\n```\n\n# Example\n\n```python\n#!/usr/bin/env python3\nfrom typing import List\n\nfrom pydantic import BaseModel, Field\n\nimport caep\n\n\nclass Config(BaseModel):\n\n text: str = Field(description=\"Required String Argument\")\n number: int = Field(default=1, description=\"Integer with default value\")\n switch: bool = Field(description=\"Boolean with default value\")\n intlist: List[int] = Field(description=\"Space separated list of ints\", json_schema_extra={\"split\": \" \"})\n\n\n# Config/section options below will only be used if loading configuration\n# from ini file (under ~/.config)\nconfig = caep.load(\n Config,\n \"CAEP Example\",\n \"caep\", # Find .ini file under ~/.config/caep\n \"caep.ini\", # Find .ini file name caep.ini\n \"section\", # Load settings from [section] (default to [DEFAULT]\n)\n\nprint(config)\n```\n\nSample output with a `intlist` read from environment and `switch` from command line:\n\n```bash\n$ export INTLIST=\"1 2 3\"\n$ ./example.py --text \"My value\" --switch\ntext='My value' number=1 switch=True intlist=[1, 2, 3]\n```\n\n# Load config without ini support\n\nSpecifying configuration location, name and section is optional and can be skipped if you\ndo not want to support loading ini files from `$XDG_CONFIG_HOME`:\n\n```python\n# Only load arguments from environment and command line\nconfig = caep.load(\n Config,\n \"CAEP Example\",\n)\n```\n\nWith the code above you can still specify a ini file with `--config <ini-file>`, and use\nenvironment variables and command line arguments.\n\n# Pydantic field types\n\nPydantic fields should be defined using `Field` and include the `description` parameter\nto specify help text for the commandline.\n\nUnless the `Field` has a `default` value, it is a required field that needs to be\nspecified in the environment, configuration file or on the command line.\n\nMany of the types described in [https://docs.pydantic.dev/usage/types/](https://docs.pydantic.dev/usage/types/)\nshould be supported, but not all of them are tested. However, nested schemas\nare *not* supported.\n\nTested types:\n\n### `str`\n\nStandard string argument.\n\n### `int`\n\nValues parsed as integer.\n\n### `float`\n\nValue parsed as float.\n\n### `pathlib.Path`\n\nValue parsed as Path.\n\n### `ipaddress.IPv4Address`\n\nValues parsed and validated as IPv4Address.\n\n### `ipaddress.IPv4Network`\n\nValues parsed and validated as IPv4Network.\n\n### `bool`\n\nValue parsed as booleans. Booleans will default to False, if no default value is set.\nExamples:\n\n\n| Field | Input | Configuration |\n| - | - | - |\n| `enable: bool = Field(description=\"Enable\")` | <NOT SET> | False |\n| `enable: bool = Field(value=False, description=\"Enable\")` | `yes` | True |\n| `enable: bool = Field(value=False, description=\"Enable\")` | `true` | True |\n| `disable: bool = Field(value=True, description=\"Disable\")` | <NOT SET> | True |\n| `disable: bool = Field(value=True, description=\"Disable\")` | `yes` | False |\n| `disable: bool = Field(value=True, description=\"Disable\")` | `true` | False |\n\n### `List[str]` (`list[str]` for python >= 3.9)\n\nList of strings, split by specified character (default = comma, argument=`split`).\n\nSome examples:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `List[int] = Field(description=\"Ints\", json_schema_extra={\"split\": \" \"})` | `1 2` | [1, 2] |\n| `List[str] = Field(description=\"Strs\")` | `ab,bc` | [\"ab\", \"bc\"] |\n\nThe argument `min_length` (pydantic builtin) can be used to specify the minimum size of the list:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `List[str] = Field(description=\"Strs\", min_length=1)` | `` | Raises ValidationError |\n\n### `Set[str]` (`set[str]` for python >= 3.9)\n\nSet, split by specified character (default = comma, argument=`split`).\n\nSome examples:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `Set[int] = Field(description=\"Ints\", json_schema_extra={\"split\": \" \"})` | `1 2 2` | {1, 2} |\n| `Set[str] = Field(description=\"Strs\")` | `ab,ab,xy` | {\"ab\", \"xy\"} |\n\nThe argument `min_length` can be used to specify the minimum size of the set:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `Set[str] = Field(description=\"Strs\", min_length=1)` | `` | Raises ValidationError |\n\n\n### `Dict[str, <TYPE>]` (`dict[str, <TYPE>]` for python >= 3.9)\n\nDictioray of strings, split by specified character (default = comma, argument=`split` for\nsplitting items and colon for splitting key/value).\n\nSome examples:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `Dict[str, str] = Field(description=\"Dict\")` | `x:a,y:b` | {\"x\": \"a\", \"y\": \"b\"} |\n| `Dict[str, int] = Field(description=\"Dict of ints\")` | `a b c:1, d e f:2` | {\"a b c\": 1, \"d e f\": 2} |\n\nThe argument `min_length` can be used to specify the minimum numer of keys in the dictionary:\n\n| Field | Input | Configuration |\n| - | - | - |\n| `Dict[str, str] = Field(description=\"Strs\", min_length=1)` | `` | Raises ValidationError |\n\n\n# Configuration\n\nArguments are parsed in two phases. First, it will look for the optional argument `--config`\nwhich can be used to specify an alternative location for the ini file. If not `--config` argument\nis given it will look for an optional ini file in the following locations\n(`~/.config has presedence`) *if* `config_id` and `config_name` is specified:\n\n- `~/.config/<CONFIG_ID>/<CONFIG_FILE_NAME>` (or directory specified by `$XDG_CONFIG_HOME`)\n- `/etc/<CONFIG_FILE_NAME>`\n\nThe ini file can contain a `[DEFAULT]` section that will be used for all configurations.\nIn addition it can have a section that corresponds with `<SECTION_NAME>` (if specified) that for\nspecific configuration, that will over override config from `[DEFAULT]`\n\n# Environment variables\n\nThe configuration step will also look for environment variables in uppercase and\nwith `-` replaced with `_`. For the example below it will lookup the following environment\nvariables:\n\n- $NUMBER\n- $BOOL\n- $STR_ARG\n\nThe configuration presedence are (from lowest to highest):\n* argparse default\n* ini file\n* environment variable\n* command line argument\n\n\n## Validation\n\n## XDG\n\nHelper functions to use [XDG Base Directories](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) are included in `caep.xdg`:\n\nIt will look up `XDG` environment variables like `$XDG_CONFIG_HOME` and use\ndefaults if not specified.\n\n### `get_xdg_dir`\n\nGeneric function to get a `XDG` directory.\n\nThe following example with will return a path object to ~/.config/myprog\n(if `$XDG_CONFIG_HOME` is not set) and create the directoy if it does not\nexist.\n\n```python\nget_xdg_dir(\"myprog\", \"XDG_CONFIG_HOME\", \".config\", True)\n```\n\n### `get_config_dir`\n\nShortcut for `get_xdg_dir(\"CONFIG\")`.\n\n### `get_cache_dir`\n\nShortcut for `get_xdg_dir(\"CACHE\")`.\n\n## CAEP Legacy usage\n\nPrior to version `0.1.0` the recommend usage was to add parser objects manually. This is\nstill supported, but with this approach you will not get the validation from pydantic:\n\n```python\n>>> import caep\n>>> import argparse\n>>> parser = argparse.ArgumentParser(\"test argparse\")\n>>> parser.add_argument('--number', type=int, default=1)\n>>> parser.add_argument('--bool', action='store_true')\n>>> parser.add_argument('--str-arg')\n>>> args = caep.config.handle_args(parser, <CONFIG_ID>, <CONFIG_FILE_NAME>, <SECTION_NAME>)\n```\n\n# Helper Functions\n\n## raise_if_some_and_not_all\nRaise ArgumentError if some of the specified entries in the dictionary has non\nfalse values but not all\n\n```python\nclass ExampleConfig(BaseModel):\n username: Optional[str] = Field(description=\"Username\")\n password: Optional[str] = Field(description=\"Password\")\n parent_id: Optional[str] = Field(description=\"Parent ID\")\n\n @model_validator(mode=\"after\") # type: ignore\n def check_arguments(cls, m: \"ExampleConfig\") -> \"ExampleConfig\":\n \"\"\"If one argument is set, they should all be set\"\"\"\n\n caep.raise_if_some_and_not_all(\n m.__dict__, [\"username\", \"password\", \"parent_id\"]\n )\n\n return m\n```\n\n## script_name\n Return first external module that called this function, directly, or indirectly\n",
"bugtrack_url": null,
"license": "ISC",
"summary": "Config Argument Env Parser (CAEP)",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/mnemonic-no/caep"
},
"split_keywords": [
"mnemonic"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0569b8f2217e9bb0d6ac6a5de64b90731c8aa43b735a7501722250ae6e1cb169",
"md5": "d620ee8e604e2489ed37b478e7ed90c6",
"sha256": "0fac25963b09708618f27bb231baa1f3be773bddca1a1a85b1d3c36105fd956a"
},
"downloads": -1,
"filename": "caep-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "d620ee8e604e2489ed37b478e7ed90c6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6, <4",
"size": 18277,
"upload_time": "2023-08-24T19:13:45",
"upload_time_iso_8601": "2023-08-24T19:13:45.659611Z",
"url": "https://files.pythonhosted.org/packages/05/69/b8f2217e9bb0d6ac6a5de64b90731c8aa43b735a7501722250ae6e1cb169/caep-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-24 19:13:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mnemonic-no",
"github_project": "caep",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "caep"
}
JSON
