# Colchian
In Greek mythology, the Colchian Dragon guarded the Golden Fleece. Jason was sent on a quest to obtain the Golden Fleece, to prove himself worthy as king - but the Colchian Dragon was the final obstacle (after many others) stopping Jason from obtaining the fleece.
The `colchian` package contains the `Colchian` class, which can be used to validate .json documents, or rather the Python `dict` resulting from loading a .json file with `json.load()`.
Colchian was developed with validation of .json configurations in mind, specifically those provided by the Conffu (https://pypi.org/project/conffu/) package, but will work for any reasonably sized .json or .yaml file (no testing was performed for large documents, nor has the code been optimised for performance).
However, Colchian can be used to validate or correct any dictionary with mostly json-like structure and content.
## Installation
Colchian is available from PyPI:
```commandline
pip install colchian
```
## Usage
A very simple example:
```python
from colchian import Colchian
from json import load, dumps
type_dict = {
"an integer": int,
"some strings": [str]
}
with open('my.json') as f:
data = load(f)
try:
valid_data = Colchian.validated(data, type_dict)
print(f'Valid:\n{dumps(valid_data)}')
except Colchian.ValidationError as e:
print(f'Something is wrong: {e}')
```
A valid `my.json`:
```json
{
"an integer": 42,
"some strings": ["vastly", "hugely", "mind-bogglingly", "big"]
}
```
To use Colchian:
- create a dictionary that defines the structure and types of a valid data dictionary (from a .json document);
- call `Colchian.validated()` with the data and the type dictionary;
- the result will be the same data, with some data casted to the appropriate type, if `strict=False` is passed to the `.validated()` method;
- a Colchian.ValidationError exception will be raised if the data is not valid according top the type_dict.
A few more tricks:
- use the Python `typing` module to use special types like `Union`, `Any`, `List` or `Optional`;
- use wildcards (keys starting with `*`) to define elements that may appear with any name (and repeatedly);
- instead of `typing.Union`, you can also use a tuple to indicate multiple options (for example `(int, float)` for a number field);
- assign a function as a type to perform custom validation or transformation on elements.
These two type dictionaries function identically:
```python
type_dict1 = {
'words': [str],
'secret': (bool, None)
}
type_dict2 = {
'words': List[str],
'secret': Optional[bool]
}
```
When setting functions as a type, make sure they match this signature:
```python
def func(x: Any, *args, strict: bool, keys: List[str]) -> Any:
pass
```
That is, the function should expect the value to validate as the first positional parameter, followed by any extra parameters you may define in the type dict, with two required keyword parameters, `strict` (which will tell your function if the validation should be strict) and `keys` (which will tell your function where the value sits in the .json document).
To keep things easily organised, you should probably define such functions as methods on child of the Colchian class. For example:
```python
from typing import List
from colchian import Colchian
class MyColchian(Colchian):
@staticmethod
def fizzbuzz(xs: List[int], fizz: str, buzz: str, strict: bool, keys: List[str]):
return [fizz*(i % 3 == 0)+buzz*(i % 5 == 0) or i for i in xs]
data = {
'xs': range(15)
}
type_dict = {
'xs': (MyColchian.fizzbuzz, 'fizz', 'buzz')
}
print(MyColchian.validated(data, type_dict))
```
You can pass extra parameters to your functions like this:
```python
type_dict = {
'xs': (MyColchian.dt_str, '%Y-%m-%d %H:%M')
}
```
The validation will interpret the rest of the tuple as positional parameters after the value, instead of type options.
Note: this means that you can't pass two functions as the only options for a type:
```python
# causes problems:
type_dict = {
'xs': (MyColchian.method1, MyColchian.method2)
}
```
This will cause values for key `'xs'` to be validated by calling `MyColchian.method1(value, MyColchian.method2, strict, keys)`, instead of trying each function separately as you might expect. If you need this functionality, you can instead do this:
```python
# no problem:
type_dict = {
'xs': ((MyColchian.method1,), (MyColchian.method2,))
}
```
Similarly, if you want to validate an optional value against a function or method, you should use `(None, function_name)` instead of `(function_name, None)`, as that would result in a call to `function_name(value, None, strict, keys)`.
An additional example is available in [typed_configuration.py](example/typed_configuration.py). Try that code and experiment with breaking [example_configuration.json](example/example_configuration.json) in interesting ways.
### Wildcards
If you define wildcards in a type dictionary, elements that don't match required keys will be matched against them. If there are multiple wildcards, Colchian will try them all. For example:
```python
type_dict = {
'*:1': {
'type': 'car',
'wheels': 4,
'engine': float,
'electric': typing.Optional[bool]
},
'*:2': {
'type': 'bicycle',
'wheels': 2,
'electric': bool
}
}
```
Makes this `vehicles.json` a valid file:
```json
{
"Peugeot 208": {
"type": "car",
"wheels": 4,
"engine": 1.4,
"electric": false
},
"Batavus Socorro": {
"type": "bicycle",
"wheels": 2,
"electric": false
},
"T-Ford": {
"type": "car",
"wheels": 4,
"engine": 2.9
}
}
```
Note that you can't use the key `'*'` twice, which is why the wildcards are distinguished as `'*:1'` and `'*:2'`.
# Dictionary constructor
When you call `Colchian.validate()` with some dictionary, it constructs a new instance of that dictionary and all of its parts. If you pass it some subclass of a dictionary (as for example, a Conffu DictConfig), you may want to pass specific parameters to its constructor.
```python
from colchian import Colchian
class MyDict(dict):
# your dict (class or instance) may have some attribute that's important to you
important = True
def __init__(self, *args, some_param=True, **kwargs):
super().__init__(*args, **kwargs)
# and possibly, you want it set as the object is constructed
self.some_attribute = some_param
def create_my_dict(md):
# create a new object of the same type as md, passing parameters to its constructor matching those of md
result = type(md)(some_param=md.some_param)
# set other attributes to match the passed object md
result.important = md.important
# return the newly created object
return result
# create a MyDict with specific settings
some_dict = MyDict({'a': 1}, some_param=False)
some_dict.important = False
# tell colchian to use a factory function when creating new instances of MyDict, instead of just the constructor
Colchian.type_factories[MyDict] = create_my_dict
# validated_dict will have settings matching some_dict
validated_dict = Colchian.validated(some_dict, {'a': int})
```
Note: this behaviour also means that Colchian will reconstruct lists - it does not check `type_factories` when doing so. It is assumed Colchian is primarily applied to dictionaries derived from .json documents, and only accounts for data types it would encounter in them.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/jaapvandervelde/colchian",
"name": "colchian",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "json,validator",
"author": "BMT Commercial Australia Pty Ltd, Jaap van der Velde",
"author_email": "jaap.vandervelde@bmtglobal.com",
"download_url": "https://files.pythonhosted.org/packages/7d/ad/8478b1d358fbb1c1ea70f5a62a544f01ee811fe8f71acd047134cbca0cd5/colchian-0.2.1.tar.gz",
"platform": null,
"description": "# Colchian\r\n\r\nIn Greek mythology, the Colchian Dragon guarded the Golden Fleece. Jason was sent on a quest to obtain the Golden Fleece, to prove himself worthy as king - but the Colchian Dragon was the final obstacle (after many others) stopping Jason from obtaining the fleece.\r\n\r\nThe `colchian` package contains the `Colchian` class, which can be used to validate .json documents, or rather the Python `dict` resulting from loading a .json file with `json.load()`.\r\n\r\nColchian was developed with validation of .json configurations in mind, specifically those provided by the Conffu (https://pypi.org/project/conffu/) package, but will work for any reasonably sized .json or .yaml file (no testing was performed for large documents, nor has the code been optimised for performance).\r\n\r\nHowever, Colchian can be used to validate or correct any dictionary with mostly json-like structure and content. \r\n\r\n## Installation\r\n\r\nColchian is available from PyPI:\r\n```commandline\r\npip install colchian\r\n```\r\n\r\n## Usage\r\n\r\nA very simple example:\r\n```python\r\nfrom colchian import Colchian\r\nfrom json import load, dumps\r\n\r\ntype_dict = {\r\n \"an integer\": int,\r\n \"some strings\": [str]\r\n}\r\n\r\nwith open('my.json') as f:\r\n data = load(f)\r\n try:\r\n valid_data = Colchian.validated(data, type_dict)\r\n print(f'Valid:\\n{dumps(valid_data)}')\r\n except Colchian.ValidationError as e:\r\n print(f'Something is wrong: {e}')\r\n```\r\nA valid `my.json`:\r\n```json\r\n{\r\n \"an integer\": 42,\r\n \"some strings\": [\"vastly\", \"hugely\", \"mind-bogglingly\", \"big\"]\r\n}\r\n```\r\n\r\nTo use Colchian:\r\n- create a dictionary that defines the structure and types of a valid data dictionary (from a .json document);\r\n- call `Colchian.validated()` with the data and the type dictionary;\r\n- the result will be the same data, with some data casted to the appropriate type, if `strict=False` is passed to the `.validated()` method;\r\n- a Colchian.ValidationError exception will be raised if the data is not valid according top the type_dict.\r\n\r\nA few more tricks:\r\n- use the Python `typing` module to use special types like `Union`, `Any`, `List` or `Optional`;\r\n- use wildcards (keys starting with `*`) to define elements that may appear with any name (and repeatedly);\r\n- instead of `typing.Union`, you can also use a tuple to indicate multiple options (for example `(int, float)` for a number field);\r\n- assign a function as a type to perform custom validation or transformation on elements.\r\n\r\nThese two type dictionaries function identically:\r\n```python\r\ntype_dict1 = {\r\n 'words': [str],\r\n 'secret': (bool, None)\r\n}\r\n\r\ntype_dict2 = {\r\n 'words': List[str],\r\n 'secret': Optional[bool]\r\n}\r\n```\r\n\r\nWhen setting functions as a type, make sure they match this signature:\r\n```python\r\ndef func(x: Any, *args, strict: bool, keys: List[str]) -> Any:\r\n pass\r\n```\r\nThat is, the function should expect the value to validate as the first positional parameter, followed by any extra parameters you may define in the type dict, with two required keyword parameters, `strict` (which will tell your function if the validation should be strict) and `keys` (which will tell your function where the value sits in the .json document).\r\n\r\nTo keep things easily organised, you should probably define such functions as methods on child of the Colchian class. For example:\r\n```python\r\nfrom typing import List\r\nfrom colchian import Colchian\r\n\r\nclass MyColchian(Colchian):\r\n @staticmethod\r\n def fizzbuzz(xs: List[int], fizz: str, buzz: str, strict: bool, keys: List[str]):\r\n return [fizz*(i % 3 == 0)+buzz*(i % 5 == 0) or i for i in xs]\r\n\r\ndata = {\r\n 'xs': range(15)\r\n}\r\n\r\ntype_dict = {\r\n 'xs': (MyColchian.fizzbuzz, 'fizz', 'buzz')\r\n}\r\n\r\nprint(MyColchian.validated(data, type_dict))\r\n```\r\n\r\nYou can pass extra parameters to your functions like this:\r\n```python\r\ntype_dict = {\r\n 'xs': (MyColchian.dt_str, '%Y-%m-%d %H:%M')\r\n}\r\n```\r\nThe validation will interpret the rest of the tuple as positional parameters after the value, instead of type options.\r\n\r\nNote: this means that you can't pass two functions as the only options for a type:\r\n```python\r\n# causes problems:\r\ntype_dict = {\r\n 'xs': (MyColchian.method1, MyColchian.method2)\r\n}\r\n```\r\nThis will cause values for key `'xs'` to be validated by calling `MyColchian.method1(value, MyColchian.method2, strict, keys)`, instead of trying each function separately as you might expect. If you need this functionality, you can instead do this:\r\n```python\r\n# no problem:\r\ntype_dict = {\r\n 'xs': ((MyColchian.method1,), (MyColchian.method2,))\r\n}\r\n```\r\nSimilarly, if you want to validate an optional value against a function or method, you should use `(None, function_name)` instead of `(function_name, None)`, as that would result in a call to `function_name(value, None, strict, keys)`.\r\n\r\nAn additional example is available in [typed_configuration.py](example/typed_configuration.py). Try that code and experiment with breaking [example_configuration.json](example/example_configuration.json) in interesting ways.\r\n\r\n### Wildcards\r\n\r\nIf you define wildcards in a type dictionary, elements that don't match required keys will be matched against them. If there are multiple wildcards, Colchian will try them all. For example:\r\n```python\r\ntype_dict = {\r\n '*:1': {\r\n 'type': 'car',\r\n 'wheels': 4,\r\n 'engine': float,\r\n 'electric': typing.Optional[bool]\r\n },\r\n '*:2': {\r\n 'type': 'bicycle',\r\n 'wheels': 2,\r\n 'electric': bool\r\n }\r\n}\r\n```\r\nMakes this `vehicles.json` a valid file:\r\n```json\r\n{\r\n \"Peugeot 208\": {\r\n \"type\": \"car\",\r\n \"wheels\": 4,\r\n \"engine\": 1.4,\r\n \"electric\": false\r\n },\r\n \"Batavus Socorro\": {\r\n \"type\": \"bicycle\",\r\n \"wheels\": 2,\r\n \"electric\": false\r\n },\r\n \"T-Ford\": {\r\n \"type\": \"car\",\r\n \"wheels\": 4,\r\n \"engine\": 2.9\r\n }\r\n}\r\n```\r\nNote that you can't use the key `'*'` twice, which is why the wildcards are distinguished as `'*:1'` and `'*:2'`.\r\n\r\n# Dictionary constructor\r\n\r\nWhen you call `Colchian.validate()` with some dictionary, it constructs a new instance of that dictionary and all of its parts. If you pass it some subclass of a dictionary (as for example, a Conffu DictConfig), you may want to pass specific parameters to its constructor.\r\n\r\n```python\r\nfrom colchian import Colchian\r\n\r\n\r\nclass MyDict(dict):\r\n # your dict (class or instance) may have some attribute that's important to you\r\n important = True\r\n\r\n def __init__(self, *args, some_param=True, **kwargs):\r\n super().__init__(*args, **kwargs)\r\n # and possibly, you want it set as the object is constructed\r\n self.some_attribute = some_param\r\n\r\n\r\ndef create_my_dict(md):\r\n # create a new object of the same type as md, passing parameters to its constructor matching those of md\r\n result = type(md)(some_param=md.some_param)\r\n # set other attributes to match the passed object md \r\n result.important = md.important\r\n # return the newly created object\r\n return result\r\n\r\n\r\n# create a MyDict with specific settings\r\nsome_dict = MyDict({'a': 1}, some_param=False)\r\nsome_dict.important = False\r\n# tell colchian to use a factory function when creating new instances of MyDict, instead of just the constructor\r\nColchian.type_factories[MyDict] = create_my_dict\r\n# validated_dict will have settings matching some_dict\r\nvalidated_dict = Colchian.validated(some_dict, {'a': int})\r\n```\r\n\r\nNote: this behaviour also means that Colchian will reconstruct lists - it does not check `type_factories` when doing so. It is assumed Colchian is primarily applied to dictionaries derived from .json documents, and only accounts for data types it would encounter in them.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Validate json/yaml documents using a Python dictionary defining keys and types.",
"version": "0.2.1",
"split_keywords": [
"json",
"validator"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7dad8478b1d358fbb1c1ea70f5a62a544f01ee811fe8f71acd047134cbca0cd5",
"md5": "c8947453b815cdfd94a6531ea0c4df99",
"sha256": "e437f39b90b17b097db761339bac8650b61931127e7a236b7c663b404ba3acd9"
},
"downloads": -1,
"filename": "colchian-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "c8947453b815cdfd94a6531ea0c4df99",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12244,
"upload_time": "2023-04-26T03:57:54",
"upload_time_iso_8601": "2023-04-26T03:57:54.287934Z",
"url": "https://files.pythonhosted.org/packages/7d/ad/8478b1d358fbb1c1ea70f5a62a544f01ee811fe8f71acd047134cbca0cd5/colchian-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-26 03:57:54",
"github": false,
"gitlab": true,
"bitbucket": false,
"gitlab_user": "jaapvandervelde",
"gitlab_project": "colchian",
"lcname": "colchian"
}
JSON