Arrex
-----
- [documentation](https://arrex.readthedocs.io/)
- [repository](https://github.com/jimy-byerley/pymadcad)
[](https://img.shields.io/pypi/pyversions/arrex)
[](https://pypi.org/project/arrex/)
[](https://arrex.readthedocs.io/en/latest/?badge=latest)
Arrex is a module that allows to create typed arrays much like `numpy.ndarray` and `array.array`, but resizeable and using any kind of element, not only numbers. Its dtype system is extremely flexible and makes it ideal to work and share structured data with compiled code.
The elements can be many different things, there is just 2 requirements:
- they must be of a fixed binary size
- they must be byte copiable, therefore without any reference or pointer to something else
### interests
- much smaller memory footprint (an arrex array is at most 30x smaller than pure python data storage)
- content can be directly shared with compiled code which improves computation performances
- slice & view without a copy
- compatible with standard python libraries
### basic usage:
```python
>>> from arrex import *
>>> a = typedlist([
myclass(...),
myclass(...),
], dtype=myclass)
>>> a[0]
myclass(...)
```
in that example, `myclass` can be a primitive numpy type, like `np.float64`
```python
>>> import arrex.numpy # this is enabling numpy dtypes for arrex
>>> typedlist(dtype=np.float64)
```
it can be a more complex type, from module `pyglm` for instance
```python
>>> import arrex.glm # this is enabling glm dtypes for arrex
>>> typedlist(dtype=glm.vec4)
```
`typedlist` is a dynamically sized, borrowing array, which mean the internal buffer of data is reallocated on append, but can be used to view and extract from any buffer without a copy.
### Use it as a list:
```python
>>> a = typedlist(dtype=vec3)
# build from an iterable
>>> a = typedlist([], dtype=vec3)
# append some data
>>> a.append(vec3(1,2,3))
# extend with an iterable
>>> a.extend(vec3(i) for i in range(5))
>>> len(a) # the current number of elements
6
>>> a.owner # the current data buffer
b'.........'
>>> a[0]
vec3(1,2,3)
```
### Use it as a slice:
```python
>>> myslice = a[:5] # no data is copied
typedlist(....)
```
### Use it as a view on top of a random buffer
```python
>>> a = np.ones((6,3), dtype='f4')
>>> myslice = typedlist(a, dtype=vec3)
```
### buffer protocol
It does support the buffer protocol, so it can be converted into a great variety of well known arrays, even without any copy
```python
>>> np.array(typedlist([....]))
```
### Which dtype is allowed
answer is: *whatever you want*, but here is some examples:
```python
# an extension type, previously declared
typedlist(dtype=glm.vec3)
# a Struct
typedlist(dtype='ffxI')
# a ctype
typedlist(dtype=ctypes.c_int*5)
# a pure python class
class test_class:
__packlayout__ = 'ff'
_struct = struct.Struct(__packlayout__)
def __init__(self, x, y):
self.x = x
self.y = y
def __bytes__(self):
return self._struct.pack(self.x, self.y)
@classmethod
def frombytes(cls, b):
return cls(*cls._struct.unpack(b))
typedlist(dtype=test_class)
# and so much more !
```
## performances
Time performances comparison between `list`, `numpy.ndarray`, and `arrex.typedlist` (see [benchmark](benchmark_typedlist.py) )
execution time (s) for 10k elements (dvec3)
set item
list: 7.9847e-04 s
numpy: 1.2727e-02 s
arrex: 1.0481e-03 s (10x faster than numpy)
get item
creation: 1.0655e-03 s
list: 5.1503e-04 s
numpy: 1.8619e-03 s
arrex: 8.0111e-04 s (2x faster than numpy)
## Roadmap
There is additionnal features planned, but no precise schedul yet:
- typedarray
a n-dim array view much like numpy arrays but using dtypes as in `typedlist`.
Its purpose is mostly to access its items with n-dim indices and slices.
- a ufunc system
to collect and put defaults to any kind of array scale operations, like `__add__`, `__mul__`, `__matmul__`, ... The goal would be to have a standard way to apply any function to every element of one or more array, that defaults to the python implementation, but can be overloaded with a compiled implementation
- maybe
even extend to the complete API of [numcy](https://github.com/jimy-byerley/numcy/blob/master/proposal.md)
Raw data
{
"_id": null,
"home_page": "https://github.com/jimy-byerley/arrex",
"name": "arrex",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "buffer array list dynamic dtype serialization numeric",
"author": "Jimy Byerley",
"author_email": "jimy.byerley@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e4/97/8568fc9a2bf3d18c8937671b5545620584a791d0fa4333485f6d50ecd680/arrex-0.5.2.tar.gz",
"platform": null,
"description": "Arrex\n-----\n\n- [documentation](https://arrex.readthedocs.io/)\n- [repository](https://github.com/jimy-byerley/pymadcad)\n\n[](https://img.shields.io/pypi/pyversions/arrex)\n[](https://pypi.org/project/arrex/)\n[](https://arrex.readthedocs.io/en/latest/?badge=latest)\n\nArrex is a module that allows to create typed arrays much like `numpy.ndarray` and `array.array`, but resizeable and using any kind of element, not only numbers. Its dtype system is extremely flexible and makes it ideal to work and share structured data with compiled code.\n\nThe elements can be many different things, there is just 2 requirements:\n\n- they must be of a fixed binary size\n- they must be byte copiable, therefore without any reference or pointer to something else\n\n### interests\n\n- much smaller memory footprint (an arrex array is at most 30x smaller than pure python data storage)\n- content can be directly shared with compiled code which improves computation performances\n- slice & view without a copy\n- compatible with standard python libraries\n\n### basic usage:\n\n```python\n>>> from arrex import *\n>>> a = typedlist([\n\t\t\tmyclass(...), \n\t\t\tmyclass(...),\n\t\t\t], dtype=myclass)\n>>> a[0]\nmyclass(...)\n```\n\nin that example, `myclass` can be a primitive numpy type, like `np.float64`\n\n```python\n>>> import arrex.numpy\t\t# this is enabling numpy dtypes for arrex\n>>> typedlist(dtype=np.float64)\n```\n\nit can be a more complex type, from module `pyglm` for instance\n\n```python\n>>> import arrex.glm\t\t# this is enabling glm dtypes for arrex\n>>> typedlist(dtype=glm.vec4)\n```\n\n`typedlist` is a dynamically sized, borrowing array, which mean the internal buffer of data is reallocated on append, but can be used to view and extract from any buffer without a copy.\n\n### Use it as a list:\n\n```python\n>>> a = typedlist(dtype=vec3)\n\n# build from an iterable\n>>> a = typedlist([], dtype=vec3)\n\n# append some data\n>>> a.append(vec3(1,2,3))\n\n# extend with an iterable\n>>> a.extend(vec3(i) for i in range(5))\n\n>>> len(a)\t# the current number of elements\n6\n\n>>> a.owner\t# the current data buffer\nb'.........'\n\n>>> a[0]\nvec3(1,2,3)\n```\n\n### Use it as a slice:\n\n```python\n>>> myslice = a[:5]\t\t# no data is copied\ntypedlist(....)\n```\n\n### Use it as a view on top of a random buffer\n\n```python\n>>> a = np.ones((6,3), dtype='f4')\n>>> myslice = typedlist(a, dtype=vec3)\n```\n\n### buffer protocol\n\nIt does support the buffer protocol, so it can be converted into a great variety of well known arrays, even without any copy\n\n```python\n>>> np.array(typedlist([....]))\n```\n\n### Which dtype is allowed\n\nanswer is: *whatever you want*, but here is some examples:\n\n```python\n# an extension type, previously declared\ntypedlist(dtype=glm.vec3)\n\n# a Struct\ntypedlist(dtype='ffxI')\n\n# a ctype\ntypedlist(dtype=ctypes.c_int*5)\n\n# a pure python class\nclass test_class:\n __packlayout__ = 'ff'\n _struct = struct.Struct(__packlayout__)\n\n def __init__(self, x, y):\n self.x = x\n self.y = y\n\n def __bytes__(self):\n return self._struct.pack(self.x, self.y)\n @classmethod\n def frombytes(cls, b):\n return cls(*cls._struct.unpack(b))\n\ntypedlist(dtype=test_class)\n\n# and so much more !\n```\n\n\n\n## performances\n\nTime performances comparison between `list`, `numpy.ndarray`, and `arrex.typedlist` (see [benchmark](benchmark_typedlist.py) )\n\nexecution time (s) for 10k elements (dvec3)\n\n\tset item\n\t list: 7.9847e-04 s\n\t numpy: 1.2727e-02 s\n\t arrex: 1.0481e-03 s (10x faster than numpy)\n\t\n\tget item\n\t creation: 1.0655e-03 s\n\t list: 5.1503e-04 s\n\t numpy: 1.8619e-03 s\n\t arrex: 8.0111e-04 s (2x faster than numpy)\n\n\n## Roadmap\n\nThere is additionnal features planned, but no precise schedul yet:\n\n- typedarray\n\n\ta n-dim array view much like numpy arrays but using dtypes as in `typedlist`.\n\tIts purpose is mostly to access its items with n-dim indices and slices.\n\t\n- a ufunc system\n\t\n\tto collect and put defaults to any kind of array scale operations, like `__add__`, `__mul__`, `__matmul__`, ... The goal would be to have a standard way to apply any function to every element of one or more array, that defaults to the python implementation, but can be overloaded with a compiled implementation\n\t\n- maybe \n\n\teven extend to the complete API of [numcy](https://github.com/jimy-byerley/numcy/blob/master/proposal.md)\n\n",
"bugtrack_url": null,
"license": "GNU LGPL v3",
"summary": "typed arrays using any custom type as element type",
"version": "0.5.2",
"project_urls": {
"Homepage": "https://github.com/jimy-byerley/arrex"
},
"split_keywords": [
"buffer",
"array",
"list",
"dynamic",
"dtype",
"serialization",
"numeric"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e4978568fc9a2bf3d18c8937671b5545620584a791d0fa4333485f6d50ecd680",
"md5": "87242a157200552d09e78ac74aab5dc3",
"sha256": "d7cb87df825d00287eef19b2e0aab3ca44397bcbfec46af714413e4c86287728"
},
"downloads": -1,
"filename": "arrex-0.5.2.tar.gz",
"has_sig": false,
"md5_digest": "87242a157200552d09e78ac74aab5dc3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 280805,
"upload_time": "2023-08-07T14:08:05",
"upload_time_iso_8601": "2023-08-07T14:08:05.707717Z",
"url": "https://files.pythonhosted.org/packages/e4/97/8568fc9a2bf3d18c8937671b5545620584a791d0fa4333485f6d50ecd680/arrex-0.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-07 14:08:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jimy-byerley",
"github_project": "arrex",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "arrex"
}
JSON
