==========================================
Wendelin.core - Out-of-core NumPy arrays
==========================================
Wendelin.core allows you to work with arrays bigger than RAM and local disk.
Bigarrays are persisted to storage, and can be changed in transactional manner.
In other words bigarrays are something like `numpy.memmap`_ for numpy.ndarray
and OS files, but support transactions and files bigger than disk. The whole
bigarray cannot generally be used as a drop-in replacement for numpy arrays,
but bigarray *slices* are real ndarrays and can be used everywhere ndarray can
be used, including in C/Cython/Fortran code. Slice size is limited by
virtual address-space size, which is ~ max 127TB on Linux/amd64.
The main class to work with is `ZBigArray` and is used like `ndarray` from
`NumPy`_:
1. create array::
from wendelin.bigarray.array_zodb import ZBigArray
import transaction
# root is connected to opened database
root['A'] = A = ZBigArray(shape=..., dtype=...)
transaction.commit()
2. view array as a real ndarray::
a = A[:] # view which covers all array, if it fits into address-space
b = A[10:100]
data for views will be loaded lazily on memory access.
3. work with views, including using C/Cython/Fortran functions from NumPy
and other libraries to read/modify data::
a[2] = 1
a[10:20] = numpy.arange(10)
numpy.mean(a)
| the amount of modifications in one transaction should be less than available RAM.
| the amount of data read is limited only by virtual address-space size.
4. data can be appended to array in O(δ) time::
values # ndarray to append of shape (δ,)
A.append(values)
and array itself can be resized in O(1) time::
A.resize(newshape)
5. changes to array data can be either discarded or saved back to DB::
transaction.abort() # discard all made changes
transaction.commit() # atomically save all changes
When using NEO_ or ZEO_ as a database, bigarrays can be simultaneously used by
several nodes in a cluster.
Please see `demo/demo_zbigarray.py`__ for a complete example.
__ demo/demo_zbigarray.py
Current state and Roadmap
=========================
Wendelin.core works in real life for workloads Nexedi_ is using in production,
including 24/7 projects. We are, however, aware of the following
limitations and things that need to be improved:
- wendelin.core is currently not very fast
- there are big - proportional to input in size - temporary array allocations
in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically
prevent processing out-of-core arrays depending on the functionality used.
Thus
- we are currently working on improved wendelin.core design and implementation,
which will use kernel virtual memory manager (instead of one implemented__ in__
userspace__) with arrays backend presented to kernel via FUSE as virtual
filesystem implemented in Go.
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c
In parallel we will also:
- try wendelin.core 1.0 on large data sets
- identify and incrementally fix big-temporaries allocation issues in NumPy and
scikit-learn
We are open to community help with the above.
Additional materials
====================
- Wendelin.core tutorial__
- Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015
__ https://www.nexedi.com/wendelin-Core.Tutorial.2016
__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret/asEntireHTML
__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret?format=pdf
.. _NumPy: http://www.numpy.org/
.. _scikit-learn: http://scikit-learn.org/
.. _numpy.memmap: http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
.. _NEO: http://www.neoppod.org/
.. _ZEO: https://pypi.python.org/pypi/ZEO
.. _Nexedi: https://www.nexedi.com/
----
Wendelin.core change history
============================
0.13 (2019-06-18)
-----------------
- Add support for Python 3.7 (commit__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f
- Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,
but stores its data in RAM only (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1
- Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5
- Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b
- Fix crash on PyVMA deallocation (commit__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba
- Move `py.bench` to pygolang__ so that it can be used not only in
Wendelin.core (commit__).
__ https://pypi.org/project/pygolang/
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf
- Enhance `t/qemu-runlinux` - utility that is used to work on combined
kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).
This was in particular useful to develop Linux kernel fixes that are needed
for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a
__ https://git.kernel.org/linus/ad2ba64dd489
__ https://git.kernel.org/linus/10dce8af3422
__ https://git.kernel.org/linus/bbd84f33652f
__ https://git.kernel.org/linus/c5bf68fe0c86
__ https://git.kernel.org/linus/438ab720c675
__ https://git.kernel.org/linus/7640682e67b3
__ https://git.kernel.org/linus/d4b13963f217
- Various bugfixes.
0.12 (2018-04-16)
-----------------
- Update licensing to be in line with whole Nexedi stack (`commit`__). Please
see https://www.nexedi.com/licensing for details, rationale and options.
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4
- Add `ArrayRef` utility to find out for a NumPy array its top-level root
parent and how to recreate the array as some view of the root;
this builds the foundation for e.g. sending arrays as references without copy
in CMFActivity joblib backend
(`commit 1`__, 2__, 3__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804
- Don't crash if during `loadblk()` garbage collection was run twice at tricky
times (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39
- Don't crash on writeout if previously `storeblk()` resulted in error
(`commit`__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908
- Fix `py.bench` and rework it to produce output in Go benchmarking format
(`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults
(`commit`__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728
- Use zodbtools/zodburi, if available, to open database by URL
(`commit`__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07
- Start to make sure it works with ZODB5 too (`commit 1`__, 2__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44
- Various bugfixes.
0.11 (2017-03-28)
-----------------
- Switch back to using ZBlk0 format by default (`commit`__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178
0.10 (2017-03-16)
-----------------
- Tell the world `dtype=object` is not supported (`commit`__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761
0.9 (2017-01-17)
----------------
- Avoid deadlocks via doing `storeblk()` calls with virtmem lock released
(`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32
- Don't crash if in `loadblk()` implementation an exception is internally
raised & caught
(`commit 1`__, 2__, 3__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c
0.8 (2016-09-28)
----------------
- Do not leak memory when loading data in ZBlk1 format (`commit`__).
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1
0.7 (2016-07-14)
------------------
- Add support for Python 3.5 (`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19
- Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00
- Various bugfixes
0.6 (2016-06-13)
----------------
- Add support for FORTRAN ordering (`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076
- Avoid deadlocks via doing `loadblk()` calls with virtmem lock released
(`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d
- Various bugfixes
0.5 (2015-10-02)
----------------
- Introduce another storage format, which is optimized for small changes, and
make it the default.
(`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085
- Various bugfixes and documentation improvements
0.4 (2015-08-19)
----------------
- Add support for O(δ) in-place BigArray.append() (commit__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9
- Implement proper multithreading support (commit__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9
- Implement proper RAM pages invalidation when backing ZODB objects are changed
from outside (`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e
- Fix all kind of failures that could happen when ZODB connection changes
worker thread in-between handling requests (`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b
- Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages
(`commit 1`__, 2__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66
- Various bugfixes
0.3 (2015-06-12)
----------------
- Add support for automatic BigArray -> ndarray conversion, so that e.g. the
following::
A = BigArray(...)
numpy.mean(A) # passing BigArray to plain NumPy function
either succeeds, or raises MemoryError if not enough address space is
available to cover whole A. (current limitation is ~ 127TB on linux/amd64)
(commit__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6
- Various bugfixes (build-fixes, crashes, overflows, etc)
0.2 (2015-05-25)
----------------
- Add support for O(1) in-place BigArray.resize() (commit__)
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75
- Various build bugfixes (older systems, non-std python, etc)
0.1 (2015-04-03)
----------------
- Initial release
Raw data
{
"_id": null,
"home_page": "https://lab.nexedi.com/nexedi/wendelin.core",
"name": "wendelin.core",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "bigdata out-of-core numpy virtual-memory",
"author": "Kirill Smelkov",
"author_email": "kirr@nexedi.com",
"download_url": "https://files.pythonhosted.org/packages/9f/16/19c7e649aa33353fc89cc3018f57e50730bbbbb8fb1c6b90f1e549655000/wendelin.core-0.13.tar.gz",
"platform": "",
"description": "==========================================\n Wendelin.core - Out-of-core NumPy arrays\n==========================================\n\nWendelin.core allows you to work with arrays bigger than RAM and local disk.\nBigarrays are persisted to storage, and can be changed in transactional manner.\n\nIn other words bigarrays are something like `numpy.memmap`_ for numpy.ndarray\nand OS files, but support transactions and files bigger than disk. The whole\nbigarray cannot generally be used as a drop-in replacement for numpy arrays,\nbut bigarray *slices* are real ndarrays and can be used everywhere ndarray can\nbe used, including in C/Cython/Fortran code. Slice size is limited by\nvirtual address-space size, which is ~ max 127TB on Linux/amd64.\n\nThe main class to work with is `ZBigArray` and is used like `ndarray` from\n`NumPy`_:\n\n1. create array::\n\n from wendelin.bigarray.array_zodb import ZBigArray\n import transaction\n\n # root is connected to opened database\n root['A'] = A = ZBigArray(shape=..., dtype=...)\n transaction.commit()\n\n2. view array as a real ndarray::\n\n a = A[:] # view which covers all array, if it fits into address-space\n b = A[10:100]\n\n data for views will be loaded lazily on memory access.\n\n3. work with views, including using C/Cython/Fortran functions from NumPy\n and other libraries to read/modify data::\n\n a[2] = 1\n a[10:20] = numpy.arange(10)\n numpy.mean(a)\n\n | the amount of modifications in one transaction should be less than available RAM.\n | the amount of data read is limited only by virtual address-space size.\n\n4. data can be appended to array in O(\u03b4) time::\n\n values # ndarray to append of shape (\u03b4,)\n A.append(values)\n\n and array itself can be resized in O(1) time::\n\n A.resize(newshape)\n\n5. changes to array data can be either discarded or saved back to DB::\n\n transaction.abort() # discard all made changes\n transaction.commit() # atomically save all changes\n\n\n\nWhen using NEO_ or ZEO_ as a database, bigarrays can be simultaneously used by\nseveral nodes in a cluster.\n\n\nPlease see `demo/demo_zbigarray.py`__ for a complete example.\n\n__ demo/demo_zbigarray.py\n\n\nCurrent state and Roadmap\n=========================\n\nWendelin.core works in real life for workloads Nexedi_ is using in production,\nincluding 24/7 projects. We are, however, aware of the following\nlimitations and things that need to be improved:\n\n- wendelin.core is currently not very fast\n- there are big - proportional to input in size - temporary array allocations\n in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically\n prevent processing out-of-core arrays depending on the functionality used.\n\nThus\n\n- we are currently working on improved wendelin.core design and implementation,\n which will use kernel virtual memory manager (instead of one implemented__ in__\n userspace__) with arrays backend presented to kernel via FUSE as virtual\n filesystem implemented in Go.\n\n__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h\n__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c\n__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c\n\nIn parallel we will also:\n\n- try wendelin.core 1.0 on large data sets\n- identify and incrementally fix big-temporaries allocation issues in NumPy and\n scikit-learn\n\nWe are open to community help with the above.\n\n\nAdditional materials\n====================\n\n- Wendelin.core tutorial__\n- Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015\n\n__ https://www.nexedi.com/wendelin-Core.Tutorial.2016\n__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret/asEntireHTML\n__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret?format=pdf\n\n\n.. _NumPy: http://www.numpy.org/\n.. _scikit-learn: http://scikit-learn.org/\n.. _numpy.memmap: http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html\n.. _NEO: http://www.neoppod.org/\n.. _ZEO: https://pypi.python.org/pypi/ZEO\n.. _Nexedi: https://www.nexedi.com/\n\n----\n\nWendelin.core change history\n============================\n\n0.13 (2019-06-18)\n-----------------\n\n- Add support for Python 3.7 (commit__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f\n\n- Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,\n but stores its data in RAM only (`commit 1`__, 2__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1\n\n- Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5\n\n- Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b\n\n- Fix crash on PyVMA deallocation (commit__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba\n\n- Move `py.bench` to pygolang__ so that it can be used not only in\n Wendelin.core (commit__).\n\n __ https://pypi.org/project/pygolang/\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf\n\n- Enhance `t/qemu-runlinux` - utility that is used to work on combined\n kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).\n This was in particular useful to develop Linux kernel fixes that are needed\n for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a\n\n __ https://git.kernel.org/linus/ad2ba64dd489\n __ https://git.kernel.org/linus/10dce8af3422\n __ https://git.kernel.org/linus/bbd84f33652f\n __ https://git.kernel.org/linus/c5bf68fe0c86\n __ https://git.kernel.org/linus/438ab720c675\n __ https://git.kernel.org/linus/7640682e67b3\n __ https://git.kernel.org/linus/d4b13963f217\n\n- Various bugfixes.\n\n0.12 (2018-04-16)\n-----------------\n\n- Update licensing to be in line with whole Nexedi stack (`commit`__). Please\n see https://www.nexedi.com/licensing for details, rationale and options.\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4\n\n- Add `ArrayRef` utility to find out for a NumPy array its top-level root\n parent and how to recreate the array as some view of the root;\n this builds the foundation for e.g. sending arrays as references without copy\n in CMFActivity joblib backend\n (`commit 1`__, 2__, 3__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804\n\n\n- Don't crash if during `loadblk()` garbage collection was run twice at tricky\n times (`commit 1`__, 2__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39\n\n- Don't crash on writeout if previously `storeblk()` resulted in error\n (`commit`__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908\n\n\n\n- Fix `py.bench` and rework it to produce output in Go benchmarking format\n (`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults\n (`commit`__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728\n\n- Use zodbtools/zodburi, if available, to open database by URL\n (`commit`__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07\n\n- Start to make sure it works with ZODB5 too (`commit 1`__, 2__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44\n\n- Various bugfixes.\n\n0.11 (2017-03-28)\n-----------------\n\n- Switch back to using ZBlk0 format by default (`commit`__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178\n\n0.10 (2017-03-16)\n-----------------\n\n- Tell the world `dtype=object` is not supported (`commit`__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761\n\n0.9 (2017-01-17)\n----------------\n\n- Avoid deadlocks via doing `storeblk()` calls with virtmem lock released\n (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32\n\n- Don't crash if in `loadblk()` implementation an exception is internally\n raised & caught\n (`commit 1`__, 2__, 3__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c\n\n0.8 (2016-09-28)\n----------------\n\n- Do not leak memory when loading data in ZBlk1 format (`commit`__).\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1\n\n0.7 (2016-07-14)\n------------------\n\n- Add support for Python 3.5 (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19\n\n- Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00\n\n- Various bugfixes\n\n0.6 (2016-06-13)\n----------------\n\n- Add support for FORTRAN ordering (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076\n\n\n- Avoid deadlocks via doing `loadblk()` calls with virtmem lock released\n (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d\n\n- Various bugfixes\n\n0.5 (2015-10-02)\n----------------\n\n- Introduce another storage format, which is optimized for small changes, and\n make it the default.\n (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085\n\n- Various bugfixes and documentation improvements\n\n\n0.4 (2015-08-19)\n----------------\n\n- Add support for O(\u03b4) in-place BigArray.append() (commit__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9\n\n- Implement proper multithreading support (commit__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9\n\n- Implement proper RAM pages invalidation when backing ZODB objects are changed\n from outside (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e\n\n- Fix all kind of failures that could happen when ZODB connection changes\n worker thread in-between handling requests (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b\n\n- Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages\n (`commit 1`__, 2__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66\n\n- Various bugfixes\n\n\n\n0.3 (2015-06-12)\n----------------\n\n- Add support for automatic BigArray -> ndarray conversion, so that e.g. the\n following::\n\n A = BigArray(...)\n numpy.mean(A) # passing BigArray to plain NumPy function\n\n either succeeds, or raises MemoryError if not enough address space is\n available to cover whole A. (current limitation is ~ 127TB on linux/amd64)\n\n (commit__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6\n\n- Various bugfixes (build-fixes, crashes, overflows, etc)\n\n\n0.2 (2015-05-25)\n----------------\n\n- Add support for O(1) in-place BigArray.resize() (commit__)\n\n __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75\n\n- Various build bugfixes (older systems, non-std python, etc)\n\n\n0.1 (2015-04-03)\n----------------\n\n- Initial release",
"bugtrack_url": null,
"license": "GPLv3+ with wide exception for Open-Source",
"summary": "Out-of-core NumPy arrays",
"version": "0.13",
"split_keywords": [
"bigdata",
"out-of-core",
"numpy",
"virtual-memory"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "363d9231212c2fb70ba24b3df850a979",
"sha256": "b9a42b9006272fe128fb37dfe464a4ff643a502f169f5c62f93740ca73720c7c"
},
"downloads": -1,
"filename": "wendelin.core-0.13.tar.gz",
"has_sig": true,
"md5_digest": "363d9231212c2fb70ba24b3df850a979",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 3105203,
"upload_time": "2019-06-18T15:14:38",
"upload_time_iso_8601": "2019-06-18T15:14:38.508643Z",
"url": "https://files.pythonhosted.org/packages/9f/16/19c7e649aa33353fc89cc3018f57e50730bbbbb8fb1c6b90f1e549655000/wendelin.core-0.13.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2019-06-18 15:14:38",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "wendelin.core"
}