| Name | phx-filters JSON |
| Version |
3.4.0
JSON |
| download |
| home_page | |
| Summary | Validation and data pipelines made easy! |
| upload_time | 2023-10-06 02:18:31 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | MIT License Copyright (c) 2016 EFL Global Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
data processing
pipeline
transformation
validation
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
|
| coveralls test coverage |
No coveralls.
|
.. image:: https://github.com/todofixthis/filters/actions/workflows/build.yml/badge.svg
:target: https://github.com/todofixthis/filters/actions/workflows/build.yml
.. image:: https://readthedocs.org/projects/filters/badge/?version=latest
:target: http://filters.readthedocs.io/
Filters
=======
The Filters library provides an easy and readable way to create complex
data validation and processing pipelines, including:
- Validating complex JSON structures in API requests or config files.
- Parsing timestamps and converting to UTC.
- Converting Unicode strings to NFC, normalizing line endings and removing
unprintable characters.
- Decoding Base64, including URL-safe variants.
And much more!
The output from one filter can be piped into the input of another, enabling you
to chain filters together to quickly and easily create complex data schemas and
pipelines.
Examples
--------
Validate a latitude position and round to manageable precision:
.. code-block:: python
(
f.Required |
f.Decimal |
f.Min(Decimal(-90)) |
f.Max(Decimal(90)) |
f.Round(to_nearest='0.000001')
).apply('-12.0431842')
Parse an incoming value as a datetime, convert to UTC and strip tzinfo:
.. code-block:: python
f.Datetime(naive=True).apply('2015-04-08T15:11:22-05:00')
Convert every value in an iterable (e.g., list) to unicode and strip
leading/trailing whitespace.
This also applies `Unicode normalization`_, strips unprintable characters and
normalizes line endings automatically.
.. code-block:: python
f.FilterRepeater(f.Unicode | f.Strip).apply([
b'\xe2\x99\xaa ',
b'\xe2\x94\x8f(\xc2\xb0.\xc2\xb0)\xe2\x94\x9b ',
b'\xe2\x94\x97(\xc2\xb0.\xc2\xb0)\xe2\x94\x93 ',
b'\xe2\x99\xaa ',
])
Parse a JSON string and check that it has correct structure:
.. code-block:: python
(
f.JsonDecode |
f.FilterMapper(
{
'birthday': f.Date,
'gender': f.CaseFold | f.Choice(choices={'f', 'm', 'n'}),
'utcOffset':
f.Decimal |
f.Min(Decimal('-15')) |
f.Max(Decimal('+15')) |
f.Round(to_nearest='0.25'),
},
allow_extra_keys = False,
allow_missing_keys = False,
)
).apply('{"birthday":"1879-03-14", "gender":"M", "utcOffset":"1"}')
Requirements
------------
Filters is known to be compatible with the following Python versions:
- 3.12
- 3.11
- 3.10
.. note::
I'm only one person, so to keep from getting overwhelmed, I'm only committing
to supporting the 3 most recent versions of Python. Filters may work in
versions not listed here — there just won't be any test coverage to prove it
😇
Installation
------------
Install the latest stable version via pip::
pip install phx-filters
.. important::
Make sure to install `phx-filters`, **not** `filters`. I created the latter
at a previous job years ago, and after I left they never touched that project
again and stopped responding to my emails — so in the end I had to fork it 🤷
Extensions
~~~~~~~~~~
The following extensions are available:
- `Django Filters`_: Adds filters designed to work with Django applications.
To install::
pip install phx-filters[django]
- `ISO Filters`_: Adds filters for interpreting standard codes and identifiers.
To install::
pip install phx-filters[iso]
.. tip::
To install multiple extensions, separate them with commas, e.g.::
pip install phx-filters[django,iso]
Running Unit Tests
------------------
Install the package with the ``test-runner`` extra to set up the necessary
dependencies, and then you can run the tests with the ``tox`` command::
pip install -e .[test-runner]
tox -p
To run tests in the current virtualenv::
python -m unittest
Documentation
-------------
Documentation is available on `ReadTheDocs`_.
If you are installing from source (see above), you can also build the
documentation locally:
#. Install extra dependencies (you only have to do this once)::
pip install '.[docs-builder]'
#. Switch to the ``docs`` directory::
cd docs
#. Build the documentation::
make html
Releases
--------
Steps to build releases are based on `Packaging Python Projects Tutorial`_
.. important::
Make sure to build releases off of the ``main`` branch, and check that all
changes from ``develop`` have been merged before creating the release!
1. Build the Project
~~~~~~~~~~~~~~~~~~~~
#. Install extra dependencies (you only have to do this once)::
pip install -e '.[build-system]'
#. Delete artefacts from previous builds, if applicable::
rm dist/*
#. Run the build::
python -m build
#. The build artefacts will be located in the ``dist`` directory at the top
level of the project.
2. Upload to PyPI
~~~~~~~~~~~~~~~~~
#. `Create a PyPI API token`_ (you only have to do this once).
#. Increment the version number in ``pyproject.toml``.
#. Check that the build artefacts are valid, and fix any errors that it finds::
python -m twine check dist/*
#. Upload build artefacts to PyPI::
python -m twine upload dist/*
3. Create GitHub Release
~~~~~~~~~~~~~~~~~~~~~~~~
#. Create a tag and push to GitHub::
git tag <version>
git push
``<version>`` must match the updated version number in ``pyproject.toml``.
#. Go to the `Releases page for the repo`_.
#. Click ``Draft a new release``.
#. Select the tag that you created in step 1.
#. Specify the title of the release (e.g., ``Filters v1.2.3``).
#. Write a description for the release. Make sure to include:
- Credit for code contributed by community members.
- Significant functionality that was added/changed/removed.
- Any backwards-incompatible changes and/or migration instructions.
- SHA256 hashes of the build artefacts.
#. GPG-sign the description for the release (ASCII-armoured).
#. Attach the build artefacts to the release.
#. Click ``Publish release``.
.. _Create a PyPI API token: https://pypi.org/manage/account/token/
.. _Django Filters: https://pypi.python.org/pypi/phx-filters-django
.. _ISO Filters: https://pypi.python.org/pypi/phx-filters-iso
.. _Packaging Python Projects Tutorial: https://packaging.python.org/en/latest/tutorials/packaging-projects/
.. _ReadTheDocs: https://filters.readthedocs.io/
.. _Releases page for the repo: https://github.com/todofixthis/filters/releases
.. _tox: https://tox.readthedocs.io/
.. _Unicode normalization: https://en.wikipedia.org/wiki/Unicode_equivalence
Raw data
{
"_id": null,
"home_page": "",
"name": "phx-filters",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "data processing,pipeline,transformation,validation",
"author": "",
"author_email": "Phoenix Zerin <phx@phx.nz>",
"download_url": "https://files.pythonhosted.org/packages/1e/9e/b10a99567c6478a693c534d04630a415f9c0cc9002aab32fdfc0c000a756/phx-filters-3.4.0.tar.gz",
"platform": null,
"description": ".. image:: https://github.com/todofixthis/filters/actions/workflows/build.yml/badge.svg\n :target: https://github.com/todofixthis/filters/actions/workflows/build.yml\n.. image:: https://readthedocs.org/projects/filters/badge/?version=latest\n :target: http://filters.readthedocs.io/\n\nFilters\n=======\nThe Filters library provides an easy and readable way to create complex\ndata validation and processing pipelines, including:\n\n- Validating complex JSON structures in API requests or config files.\n- Parsing timestamps and converting to UTC.\n- Converting Unicode strings to NFC, normalizing line endings and removing\n unprintable characters.\n- Decoding Base64, including URL-safe variants.\n\nAnd much more!\n\nThe output from one filter can be piped into the input of another, enabling you\nto chain filters together to quickly and easily create complex data schemas and\npipelines.\n\n\nExamples\n--------\nValidate a latitude position and round to manageable precision:\n\n.. code-block:: python\n\n (\n f.Required |\n f.Decimal |\n f.Min(Decimal(-90)) |\n f.Max(Decimal(90)) |\n f.Round(to_nearest='0.000001')\n ).apply('-12.0431842')\n\nParse an incoming value as a datetime, convert to UTC and strip tzinfo:\n\n.. code-block:: python\n\n f.Datetime(naive=True).apply('2015-04-08T15:11:22-05:00')\n\nConvert every value in an iterable (e.g., list) to unicode and strip\nleading/trailing whitespace.\nThis also applies `Unicode normalization`_, strips unprintable characters and\nnormalizes line endings automatically.\n\n.. code-block:: python\n\n f.FilterRepeater(f.Unicode | f.Strip).apply([\n b'\\xe2\\x99\\xaa ',\n b'\\xe2\\x94\\x8f(\\xc2\\xb0.\\xc2\\xb0)\\xe2\\x94\\x9b ',\n b'\\xe2\\x94\\x97(\\xc2\\xb0.\\xc2\\xb0)\\xe2\\x94\\x93 ',\n b'\\xe2\\x99\\xaa ',\n ])\n\nParse a JSON string and check that it has correct structure:\n\n.. code-block:: python\n\n (\n f.JsonDecode |\n f.FilterMapper(\n {\n 'birthday': f.Date,\n 'gender': f.CaseFold | f.Choice(choices={'f', 'm', 'n'}),\n\n 'utcOffset':\n f.Decimal |\n f.Min(Decimal('-15')) |\n f.Max(Decimal('+15')) |\n f.Round(to_nearest='0.25'),\n },\n\n allow_extra_keys = False,\n allow_missing_keys = False,\n )\n ).apply('{\"birthday\":\"1879-03-14\", \"gender\":\"M\", \"utcOffset\":\"1\"}')\n\n\nRequirements\n------------\nFilters is known to be compatible with the following Python versions:\n\n- 3.12\n- 3.11\n- 3.10\n\n.. note::\n I'm only one person, so to keep from getting overwhelmed, I'm only committing\n to supporting the 3 most recent versions of Python. Filters may work in\n versions not listed here \u2014 there just won't be any test coverage to prove it\n \ud83d\ude07\n\nInstallation\n------------\nInstall the latest stable version via pip::\n\n pip install phx-filters\n\n\n.. important::\n Make sure to install `phx-filters`, **not** `filters`. I created the latter\n at a previous job years ago, and after I left they never touched that project\n again and stopped responding to my emails \u2014 so in the end I had to fork it \ud83e\udd37\n\nExtensions\n~~~~~~~~~~\nThe following extensions are available:\n\n- `Django Filters`_: Adds filters designed to work with Django applications.\n To install::\n\n pip install phx-filters[django]\n\n- `ISO Filters`_: Adds filters for interpreting standard codes and identifiers.\n To install::\n\n pip install phx-filters[iso]\n\n.. tip::\n To install multiple extensions, separate them with commas, e.g.::\n\n pip install phx-filters[django,iso]\n\nRunning Unit Tests\n------------------\nInstall the package with the ``test-runner`` extra to set up the necessary\ndependencies, and then you can run the tests with the ``tox`` command::\n\n pip install -e .[test-runner]\n tox -p\n\nTo run tests in the current virtualenv::\n\n python -m unittest\n\nDocumentation\n-------------\nDocumentation is available on `ReadTheDocs`_.\n\nIf you are installing from source (see above), you can also build the\ndocumentation locally:\n\n#. Install extra dependencies (you only have to do this once)::\n\n pip install '.[docs-builder]'\n\n#. Switch to the ``docs`` directory::\n\n cd docs\n\n#. Build the documentation::\n\n make html\n\n\nReleases\n--------\nSteps to build releases are based on `Packaging Python Projects Tutorial`_\n\n.. important::\n\n Make sure to build releases off of the ``main`` branch, and check that all\n changes from ``develop`` have been merged before creating the release!\n\n1. Build the Project\n~~~~~~~~~~~~~~~~~~~~\n#. Install extra dependencies (you only have to do this once)::\n\n pip install -e '.[build-system]'\n\n#. Delete artefacts from previous builds, if applicable::\n\n rm dist/*\n\n#. Run the build::\n\n python -m build\n\n#. The build artefacts will be located in the ``dist`` directory at the top\n level of the project.\n\n2. Upload to PyPI\n~~~~~~~~~~~~~~~~~\n#. `Create a PyPI API token`_ (you only have to do this once).\n#. Increment the version number in ``pyproject.toml``.\n#. Check that the build artefacts are valid, and fix any errors that it finds::\n\n python -m twine check dist/*\n\n#. Upload build artefacts to PyPI::\n\n python -m twine upload dist/*\n\n\n3. Create GitHub Release\n~~~~~~~~~~~~~~~~~~~~~~~~\n#. Create a tag and push to GitHub::\n\n git tag <version>\n git push\n\n ``<version>`` must match the updated version number in ``pyproject.toml``.\n\n#. Go to the `Releases page for the repo`_.\n#. Click ``Draft a new release``.\n#. Select the tag that you created in step 1.\n#. Specify the title of the release (e.g., ``Filters v1.2.3``).\n#. Write a description for the release. Make sure to include:\n - Credit for code contributed by community members.\n - Significant functionality that was added/changed/removed.\n - Any backwards-incompatible changes and/or migration instructions.\n - SHA256 hashes of the build artefacts.\n#. GPG-sign the description for the release (ASCII-armoured).\n#. Attach the build artefacts to the release.\n#. Click ``Publish release``.\n\n.. _Create a PyPI API token: https://pypi.org/manage/account/token/\n.. _Django Filters: https://pypi.python.org/pypi/phx-filters-django\n.. _ISO Filters: https://pypi.python.org/pypi/phx-filters-iso\n.. _Packaging Python Projects Tutorial: https://packaging.python.org/en/latest/tutorials/packaging-projects/\n.. _ReadTheDocs: https://filters.readthedocs.io/\n.. _Releases page for the repo: https://github.com/todofixthis/filters/releases\n.. _tox: https://tox.readthedocs.io/\n.. _Unicode normalization: https://en.wikipedia.org/wiki/Unicode_equivalence\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2016 EFL Global Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Validation and data pipelines made easy!",
"version": "3.4.0",
"project_urls": {
"Changelog": "https://github.com/todofixthis/filters/releases",
"Documentation": "https://filters.readthedocs.io/",
"Issues": "https://github.com/todofixthis/filters/issues",
"Repository": "https://github.com/todofixthis/filters"
},
"split_keywords": [
"data processing",
"pipeline",
"transformation",
"validation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "29911fb10caa150b5ad3f886fb55cb7b89676e8d906e0a85806ed0e04df6890d",
"md5": "f187cff5a68d133c5211b9341dbd453d",
"sha256": "5ad74dfa75e13764191bf7a333d4de57dcf9c7e40bca418096b51050808f7bdf"
},
"downloads": -1,
"filename": "phx_filters-3.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f187cff5a68d133c5211b9341dbd453d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 40597,
"upload_time": "2023-10-06T02:18:28",
"upload_time_iso_8601": "2023-10-06T02:18:28.168164Z",
"url": "https://files.pythonhosted.org/packages/29/91/1fb10caa150b5ad3f886fb55cb7b89676e8d906e0a85806ed0e04df6890d/phx_filters-3.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1e9eb10a99567c6478a693c534d04630a415f9c0cc9002aab32fdfc0c000a756",
"md5": "ae59db6af6405671d31a1e2a7d3311ed",
"sha256": "3ccaa18de8e7ff10cf89d6bbe5ba80dd3e0aa5524fa17d32822abdbb81a2ad41"
},
"downloads": -1,
"filename": "phx-filters-3.4.0.tar.gz",
"has_sig": false,
"md5_digest": "ae59db6af6405671d31a1e2a7d3311ed",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 71176,
"upload_time": "2023-10-06T02:18:31",
"upload_time_iso_8601": "2023-10-06T02:18:31.210192Z",
"url": "https://files.pythonhosted.org/packages/1e/9e/b10a99567c6478a693c534d04630a415f9c0cc9002aab32fdfc0c000a756/phx-filters-3.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-06 02:18:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "todofixthis",
"github_project": "filters",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "phx-filters"
}
