=========
condenser
=========
.. image:: https://github.com/nens/condenser/workflows/Linux/badge.svg
:target: https://github.com/nens/condenser/actions?query=workflow%3ALinux
.. image:: https://img.shields.io/pypi/v/condenser.svg
:target: https://pypi.python.org/pypi/condenser
A fast interface between SQLAlchemy and Numpy
Features
--------
This project reads data from SQLAlchemy into structured numpy arrays. Next to
the builtin SQLAlchemy types, it also supports reading geometries into ``pygeos``
arrays of geometries.
Installation
------------
Install via pip using::
$ pip install condenser
Include geometry support::
$ pip install condenser[geo]
Usage
-----
``condenser`` exposes one object: ``NumpyQuery``, which is used
as a custom SQLAlchemy query class as follows::
>>> from condenser import NumpyQuery
>>> # create the session with a custom query class
>>> session = session_factory(query_cls=NumpyQuery)
If a session is constructed like that, every query object will have an
additional method to dump the selected data into a numpy structured array::
>>> query = session.query(SomeModel.float_type_column, SomeModel.int_type_column)
>>> my_array = query.as_structarray()
Geometry support
----------------
Geometry columns are automatically converted to arrays of ``pygeos.Geometry``
objects. See https://pygeos.readthedocs.io on for (vectorized) numpy functions
that can act on these arrays.
Transform geometries (using the `ST_Transform` database function) as follows::
>>> query.with_transformed_geometries(target_srid=28992)
Note that this will only transform geometries with a known SRID. If an SRID is
known only from another metadata source, use a function appropriate to your
database backend to set the projection before converting it. Another option
is using the ``pyproj`` library in combination with ``pygeos.apply`` to
transform geometries from Python.
Custom dtype mapping
--------------------
`condenser` has a safe approach on guessing the Numpy dtypes from SQLAlchemy
dtypes. It always takes 8-byte signed integers and floats. For some database
backends this can be changed to for example 4-byte datatypes.
Override an SQLAlchemy to NumPy type mapping after constructing the query::
>>> from sqlalchemy import Integer
>>> query = session.query(SomeModel.float_type_column)
>>> query.numpy_settings[Integer]["dtype"] = np.int32
>>> query.as_structarray()
Or globally::
>>> NumpyQuery.default_numpy_settings[Integer]["dtype"] = np.int32
NULL values
-----------
Most numpy datatypes handle NULL (Python: None) values natively. Only integer
typed columns deserve extra attention as they have no equivalent in the NumPy
integer dtype. This package converts NULLs in integer columns to `-1` by
default. A complete list of what to expect:
- NULL in integer columns becomes `-1`
- NULL in float and numeric columns becomes `nan`
- NULL in boolean columns becomes `False`
- NULL in object typed columns (string, text, geometry) becomes `None`
Adjust the NULL value before executing the query::
>>> from sqlalchemy import Integer
>>> query = session.query(SomeModel.float_type_column)
>>> query.numpy_settings[Integer]["null"] = -9999
>>> query.numpy_settings[Boolean]["null"] = True
>>> query.as_structarray()
Or globally::
>>> NumpyQuery.default_numpy_settings[Integer]["null"] = -9999
Credits
-------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
Raw data
{
"_id": null,
"home_page": "https://github.com/nens/condenser",
"name": "condenser",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "condenser",
"author": "Casper van der Wel",
"author_email": "caspervdw@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/7f/a3/5885e009d94435747994342c5bb4436c00c9784d056401ef2022e67f0d46/condenser-0.2.1.tar.gz",
"platform": null,
"description": "=========\ncondenser\n=========\n\n.. image:: https://github.com/nens/condenser/workflows/Linux/badge.svg\n :target: https://github.com/nens/condenser/actions?query=workflow%3ALinux\n\n.. image:: https://img.shields.io/pypi/v/condenser.svg\n :target: https://pypi.python.org/pypi/condenser\n\nA fast interface between SQLAlchemy and Numpy\n\nFeatures\n--------\n\nThis project reads data from SQLAlchemy into structured numpy arrays. Next to\nthe builtin SQLAlchemy types, it also supports reading geometries into ``pygeos``\narrays of geometries.\n\nInstallation\n------------\n\nInstall via pip using::\n\n$ pip install condenser\n\nInclude geometry support::\n\n$ pip install condenser[geo]\n\nUsage\n-----\n\n``condenser`` exposes one object: ``NumpyQuery``, which is used\nas a custom SQLAlchemy query class as follows::\n\n>>> from condenser import NumpyQuery\n>>> # create the session with a custom query class\n>>> session = session_factory(query_cls=NumpyQuery)\n\nIf a session is constructed like that, every query object will have an\nadditional method to dump the selected data into a numpy structured array::\n\n>>> query = session.query(SomeModel.float_type_column, SomeModel.int_type_column)\n>>> my_array = query.as_structarray()\n\nGeometry support\n----------------\n\nGeometry columns are automatically converted to arrays of ``pygeos.Geometry``\nobjects. See https://pygeos.readthedocs.io on for (vectorized) numpy functions\nthat can act on these arrays.\n\nTransform geometries (using the `ST_Transform` database function) as follows::\n\n>>> query.with_transformed_geometries(target_srid=28992)\n\nNote that this will only transform geometries with a known SRID. If an SRID is\nknown only from another metadata source, use a function appropriate to your\ndatabase backend to set the projection before converting it. Another option\nis using the ``pyproj`` library in combination with ``pygeos.apply`` to\ntransform geometries from Python.\n\n\nCustom dtype mapping\n--------------------\n\n`condenser` has a safe approach on guessing the Numpy dtypes from SQLAlchemy\ndtypes. It always takes 8-byte signed integers and floats. For some database\nbackends this can be changed to for example 4-byte datatypes.\nOverride an SQLAlchemy to NumPy type mapping after constructing the query::\n\n>>> from sqlalchemy import Integer\n>>> query = session.query(SomeModel.float_type_column)\n>>> query.numpy_settings[Integer][\"dtype\"] = np.int32\n>>> query.as_structarray()\n\nOr globally::\n\n>>> NumpyQuery.default_numpy_settings[Integer][\"dtype\"] = np.int32\n\nNULL values\n-----------\n\nMost numpy datatypes handle NULL (Python: None) values natively. Only integer\ntyped columns deserve extra attention as they have no equivalent in the NumPy\ninteger dtype. This package converts NULLs in integer columns to `-1` by\ndefault. A complete list of what to expect:\n\n- NULL in integer columns becomes `-1`\n- NULL in float and numeric columns becomes `nan`\n- NULL in boolean columns becomes `False`\n- NULL in object typed columns (string, text, geometry) becomes `None`\n\nAdjust the NULL value before executing the query::\n\n>>> from sqlalchemy import Integer\n>>> query = session.query(SomeModel.float_type_column)\n>>> query.numpy_settings[Integer][\"null\"] = -9999\n>>> query.numpy_settings[Boolean][\"null\"] = True\n>>> query.as_structarray()\n\nOr globally::\n\n>>> NumpyQuery.default_numpy_settings[Integer][\"null\"] = -9999\n\nCredits\n-------\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n",
"bugtrack_url": null,
"license": "BSD license",
"summary": "A fast interface between SQLAlchemy and Numpy",
"version": "0.2.1",
"split_keywords": [
"condenser"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ed6c2380101f86d77bd64b08f944dbce565c9d2457c6f8378a7061ef8158b0ea",
"md5": "e255278ecfdb862ced13bc5b21308323",
"sha256": "fafeea9a6d589db78b8cc3e6dd65a126d250b35afeed744fddf27ee857bc60b8"
},
"downloads": -1,
"filename": "condenser-0.2.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "e255278ecfdb862ced13bc5b21308323",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 6534,
"upload_time": "2023-03-20T12:44:12",
"upload_time_iso_8601": "2023-03-20T12:44:12.139454Z",
"url": "https://files.pythonhosted.org/packages/ed/6c/2380101f86d77bd64b08f944dbce565c9d2457c6f8378a7061ef8158b0ea/condenser-0.2.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7fa35885e009d94435747994342c5bb4436c00c9784d056401ef2022e67f0d46",
"md5": "01bc2aecced1fe7cc7d5a89dee9e5f72",
"sha256": "e416c1e81c85f8e4481cd7eb9e9a50d3a842672300e07f14a6ac9f2916e1b848"
},
"downloads": -1,
"filename": "condenser-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "01bc2aecced1fe7cc7d5a89dee9e5f72",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6770,
"upload_time": "2023-03-20T12:44:13",
"upload_time_iso_8601": "2023-03-20T12:44:13.947672Z",
"url": "https://files.pythonhosted.org/packages/7f/a3/5885e009d94435747994342c5bb4436c00c9784d056401ef2022e67f0d46/condenser-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-20 12:44:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "nens",
"github_project": "condenser",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "condenser"
}
JSON
