wendelin.core


Namewendelin.core JSON
Version 0.13 PyPI version JSON
download
home_pagehttps://lab.nexedi.com/nexedi/wendelin.core
SummaryOut-of-core NumPy arrays
upload_time2019-06-18 15:14:38
maintainer
docs_urlNone
authorKirill Smelkov
requires_python
licenseGPLv3+ with wide exception for Open-Source
keywords bigdata out-of-core numpy virtual-memory
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            ==========================================
 Wendelin.core - Out-of-core NumPy arrays
==========================================

Wendelin.core allows you to work with arrays bigger than RAM and local disk.
Bigarrays are persisted to storage, and can be changed in transactional manner.

In other words bigarrays are something like `numpy.memmap`_ for numpy.ndarray
and OS files, but support transactions and files bigger than disk. The whole
bigarray cannot generally be used as a drop-in replacement for numpy arrays,
but bigarray *slices* are real ndarrays and can be used everywhere ndarray can
be used, including in C/Cython/Fortran code. Slice size is limited by
virtual address-space size, which is ~ max 127TB on Linux/amd64.

The main class to work with is `ZBigArray` and is used like `ndarray` from
`NumPy`_:

1. create array::

    from wendelin.bigarray.array_zodb import ZBigArray
    import transaction

    # root is connected to opened database
    root['A'] = A = ZBigArray(shape=..., dtype=...)
    transaction.commit()

2. view array as a real ndarray::

    a = A[:]        # view which covers all array, if it fits into address-space
    b = A[10:100]

   data for views will be loaded lazily on memory access.

3. work with views, including using C/Cython/Fortran functions from NumPy
   and other libraries to read/modify data::

    a[2] = 1
    a[10:20] = numpy.arange(10)
    numpy.mean(a)

   | the amount of modifications in one transaction should be less than available RAM.
   | the amount of data read is limited only by virtual address-space size.

4. data can be appended to array in O(δ) time::

    values                  # ndarray to append of shape  (δ,)
    A.append(values)

   and array itself can be resized in O(1) time::

    A.resize(newshape)

5. changes to array data can be either discarded or saved back to DB::

    transaction.abort()     # discard all made changes
    transaction.commit()    # atomically save all changes



When using NEO_ or ZEO_ as a database, bigarrays can be simultaneously used by
several nodes in a cluster.


Please see `demo/demo_zbigarray.py`__ for a complete example.

__ demo/demo_zbigarray.py


Current state and Roadmap
=========================

Wendelin.core works in real life for workloads Nexedi_ is using in production,
including 24/7 projects. We are, however, aware of the following
limitations and things that need to be improved:

- wendelin.core is currently not very fast
- there are big - proportional to input in size - temporary array allocations
  in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically
  prevent processing out-of-core arrays depending on the functionality used.

Thus

- we are currently working on improved wendelin.core design and implementation,
  which will use kernel virtual memory manager (instead of one implemented__ in__
  userspace__) with arrays backend presented to kernel via FUSE as virtual
  filesystem implemented in Go.

__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h
__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c
__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c

In parallel we will also:

- try wendelin.core 1.0 on large data sets
- identify and incrementally fix big-temporaries allocation issues in NumPy and
  scikit-learn

We are open to community help with the above.


Additional materials
====================

- Wendelin.core tutorial__
- Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015

__  https://www.nexedi.com/wendelin-Core.Tutorial.2016
__  http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret/asEntireHTML
__  http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret?format=pdf


.. _NumPy:          http://www.numpy.org/
.. _scikit-learn:   http://scikit-learn.org/
.. _numpy.memmap:   http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
.. _NEO:            http://www.neoppod.org/
.. _ZEO:            https://pypi.python.org/pypi/ZEO
.. _Nexedi:         https://www.nexedi.com/

----

Wendelin.core change history
============================

0.13 (2019-06-18)
-----------------

- Add support for Python 3.7 (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f

- Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,
  but stores its data in RAM only (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1

- Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5

- Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b

- Fix crash on PyVMA deallocation (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba

- Move `py.bench` to pygolang__ so that it can be used not only in
  Wendelin.core (commit__).

  __ https://pypi.org/project/pygolang/
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf

- Enhance `t/qemu-runlinux` - utility that is used to work on combined
  kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).
  This was in particular useful to develop Linux kernel fixes that are needed
  for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a

  __ https://git.kernel.org/linus/ad2ba64dd489
  __ https://git.kernel.org/linus/10dce8af3422
  __ https://git.kernel.org/linus/bbd84f33652f
  __ https://git.kernel.org/linus/c5bf68fe0c86
  __ https://git.kernel.org/linus/438ab720c675
  __ https://git.kernel.org/linus/7640682e67b3
  __ https://git.kernel.org/linus/d4b13963f217

- Various bugfixes.

0.12 (2018-04-16)
-----------------

- Update licensing to be in line with whole Nexedi stack (`commit`__). Please
  see https://www.nexedi.com/licensing for details, rationale and options.

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4

- Add `ArrayRef` utility to find out for a NumPy array its top-level root
  parent and how to recreate the array as some view of the root;
  this builds the foundation for e.g. sending arrays as references without copy
  in CMFActivity joblib backend
  (`commit 1`__, 2__, 3__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804


- Don't crash if during `loadblk()` garbage collection was run twice at tricky
  times (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39

- Don't crash on writeout if previously `storeblk()` resulted in error
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908



- Fix `py.bench` and rework it to produce output in Go benchmarking format
  (`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728

- Use zodbtools/zodburi, if available, to open database by URL
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07

- Start to make sure it works with ZODB5 too (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44

- Various bugfixes.

0.11 (2017-03-28)
-----------------

- Switch back to using ZBlk0 format by default (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178

0.10 (2017-03-16)
-----------------

- Tell the world `dtype=object` is not supported (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761

0.9 (2017-01-17)
----------------

- Avoid deadlocks via doing `storeblk()` calls with virtmem lock released
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32

- Don't crash if in `loadblk()` implementation an exception is internally
  raised & caught
  (`commit 1`__, 2__, 3__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c

0.8 (2016-09-28)
----------------

- Do not leak memory when loading data in ZBlk1 format (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1

0.7 (2016-07-14)
------------------

- Add support for Python 3.5 (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19

- Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00

- Various bugfixes

0.6 (2016-06-13)
----------------

- Add support for FORTRAN ordering (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076


- Avoid deadlocks via doing `loadblk()` calls with virtmem lock released
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d

- Various bugfixes

0.5 (2015-10-02)
----------------

- Introduce another storage format, which is optimized for small changes, and
  make it the default.
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085

- Various bugfixes and documentation improvements


0.4 (2015-08-19)
----------------

- Add support for O(δ) in-place BigArray.append() (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9

- Implement proper multithreading support (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9

- Implement proper RAM pages invalidation when backing ZODB objects are changed
  from outside (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e

- Fix all kind of failures that could happen when ZODB connection changes
  worker thread in-between handling requests (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b

- Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66

- Various bugfixes



0.3 (2015-06-12)
----------------

- Add support for automatic BigArray -> ndarray conversion, so that e.g. the
  following::

    A = BigArray(...)
    numpy.mean(A)       # passing BigArray to plain NumPy function

  either succeeds, or raises MemoryError if not enough address space is
  available to cover whole A. (current limitation is ~ 127TB on linux/amd64)

  (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6

- Various bugfixes (build-fixes, crashes, overflows, etc)


0.2 (2015-05-25)
----------------

- Add support for O(1) in-place BigArray.resize() (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75

- Various build bugfixes (older systems, non-std python, etc)


0.1 (2015-04-03)
----------------

- Initial release
            

Raw data

            {
    "_id": null,
    "home_page": "https://lab.nexedi.com/nexedi/wendelin.core",
    "name": "wendelin.core",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "bigdata out-of-core numpy virtual-memory",
    "author": "Kirill Smelkov",
    "author_email": "kirr@nexedi.com",
    "download_url": "https://files.pythonhosted.org/packages/9f/16/19c7e649aa33353fc89cc3018f57e50730bbbbb8fb1c6b90f1e549655000/wendelin.core-0.13.tar.gz",
    "platform": "",
    "description": "==========================================\n Wendelin.core - Out-of-core NumPy arrays\n==========================================\n\nWendelin.core allows you to work with arrays bigger than RAM and local disk.\nBigarrays are persisted to storage, and can be changed in transactional manner.\n\nIn other words bigarrays are something like `numpy.memmap`_ for numpy.ndarray\nand OS files, but support transactions and files bigger than disk. The whole\nbigarray cannot generally be used as a drop-in replacement for numpy arrays,\nbut bigarray *slices* are real ndarrays and can be used everywhere ndarray can\nbe used, including in C/Cython/Fortran code. Slice size is limited by\nvirtual address-space size, which is ~ max 127TB on Linux/amd64.\n\nThe main class to work with is `ZBigArray` and is used like `ndarray` from\n`NumPy`_:\n\n1. create array::\n\n    from wendelin.bigarray.array_zodb import ZBigArray\n    import transaction\n\n    # root is connected to opened database\n    root['A'] = A = ZBigArray(shape=..., dtype=...)\n    transaction.commit()\n\n2. view array as a real ndarray::\n\n    a = A[:]        # view which covers all array, if it fits into address-space\n    b = A[10:100]\n\n   data for views will be loaded lazily on memory access.\n\n3. work with views, including using C/Cython/Fortran functions from NumPy\n   and other libraries to read/modify data::\n\n    a[2] = 1\n    a[10:20] = numpy.arange(10)\n    numpy.mean(a)\n\n   | the amount of modifications in one transaction should be less than available RAM.\n   | the amount of data read is limited only by virtual address-space size.\n\n4. data can be appended to array in O(\u03b4) time::\n\n    values                  # ndarray to append of shape  (\u03b4,)\n    A.append(values)\n\n   and array itself can be resized in O(1) time::\n\n    A.resize(newshape)\n\n5. changes to array data can be either discarded or saved back to DB::\n\n    transaction.abort()     # discard all made changes\n    transaction.commit()    # atomically save all changes\n\n\n\nWhen using NEO_ or ZEO_ as a database, bigarrays can be simultaneously used by\nseveral nodes in a cluster.\n\n\nPlease see `demo/demo_zbigarray.py`__ for a complete example.\n\n__ demo/demo_zbigarray.py\n\n\nCurrent state and Roadmap\n=========================\n\nWendelin.core works in real life for workloads Nexedi_ is using in production,\nincluding 24/7 projects. We are, however, aware of the following\nlimitations and things that need to be improved:\n\n- wendelin.core is currently not very fast\n- there are big - proportional to input in size - temporary array allocations\n  in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically\n  prevent processing out-of-core arrays depending on the functionality used.\n\nThus\n\n- we are currently working on improved wendelin.core design and implementation,\n  which will use kernel virtual memory manager (instead of one implemented__ in__\n  userspace__) with arrays backend presented to kernel via FUSE as virtual\n  filesystem implemented in Go.\n\n__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h\n__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c\n__  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c\n\nIn parallel we will also:\n\n- try wendelin.core 1.0 on large data sets\n- identify and incrementally fix big-temporaries allocation issues in NumPy and\n  scikit-learn\n\nWe are open to community help with the above.\n\n\nAdditional materials\n====================\n\n- Wendelin.core tutorial__\n- Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015\n\n__  https://www.nexedi.com/wendelin-Core.Tutorial.2016\n__  http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret/asEntireHTML\n__  http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret?format=pdf\n\n\n.. _NumPy:          http://www.numpy.org/\n.. _scikit-learn:   http://scikit-learn.org/\n.. _numpy.memmap:   http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html\n.. _NEO:            http://www.neoppod.org/\n.. _ZEO:            https://pypi.python.org/pypi/ZEO\n.. _Nexedi:         https://www.nexedi.com/\n\n----\n\nWendelin.core change history\n============================\n\n0.13 (2019-06-18)\n-----------------\n\n- Add support for Python 3.7 (commit__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f\n\n- Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,\n  but stores its data in RAM only (`commit 1`__, 2__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1\n\n- Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5\n\n- Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b\n\n- Fix crash on PyVMA deallocation (commit__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba\n\n- Move `py.bench` to pygolang__ so that it can be used not only in\n  Wendelin.core (commit__).\n\n  __ https://pypi.org/project/pygolang/\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf\n\n- Enhance `t/qemu-runlinux` - utility that is used to work on combined\n  kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).\n  This was in particular useful to develop Linux kernel fixes that are needed\n  for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a\n\n  __ https://git.kernel.org/linus/ad2ba64dd489\n  __ https://git.kernel.org/linus/10dce8af3422\n  __ https://git.kernel.org/linus/bbd84f33652f\n  __ https://git.kernel.org/linus/c5bf68fe0c86\n  __ https://git.kernel.org/linus/438ab720c675\n  __ https://git.kernel.org/linus/7640682e67b3\n  __ https://git.kernel.org/linus/d4b13963f217\n\n- Various bugfixes.\n\n0.12 (2018-04-16)\n-----------------\n\n- Update licensing to be in line with whole Nexedi stack (`commit`__). Please\n  see https://www.nexedi.com/licensing for details, rationale and options.\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4\n\n- Add `ArrayRef` utility to find out for a NumPy array its top-level root\n  parent and how to recreate the array as some view of the root;\n  this builds the foundation for e.g. sending arrays as references without copy\n  in CMFActivity joblib backend\n  (`commit 1`__, 2__, 3__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804\n\n\n- Don't crash if during `loadblk()` garbage collection was run twice at tricky\n  times (`commit 1`__, 2__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39\n\n- Don't crash on writeout if previously `storeblk()` resulted in error\n  (`commit`__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908\n\n\n\n- Fix `py.bench` and rework it to produce output in Go benchmarking format\n  (`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults\n  (`commit`__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728\n\n- Use zodbtools/zodburi, if available, to open database by URL\n  (`commit`__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07\n\n- Start to make sure it works with ZODB5 too (`commit 1`__, 2__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44\n\n- Various bugfixes.\n\n0.11 (2017-03-28)\n-----------------\n\n- Switch back to using ZBlk0 format by default (`commit`__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178\n\n0.10 (2017-03-16)\n-----------------\n\n- Tell the world `dtype=object` is not supported (`commit`__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761\n\n0.9 (2017-01-17)\n----------------\n\n- Avoid deadlocks via doing `storeblk()` calls with virtmem lock released\n  (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32\n\n- Don't crash if in `loadblk()` implementation an exception is internally\n  raised & caught\n  (`commit 1`__, 2__, 3__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c\n\n0.8 (2016-09-28)\n----------------\n\n- Do not leak memory when loading data in ZBlk1 format (`commit`__).\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1\n\n0.7 (2016-07-14)\n------------------\n\n- Add support for Python 3.5 (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19\n\n- Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00\n\n- Various bugfixes\n\n0.6 (2016-06-13)\n----------------\n\n- Add support for FORTRAN ordering (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076\n\n\n- Avoid deadlocks via doing `loadblk()` calls with virtmem lock released\n  (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d\n\n- Various bugfixes\n\n0.5 (2015-10-02)\n----------------\n\n- Introduce another storage format, which is optimized for small changes, and\n  make it the default.\n  (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085\n\n- Various bugfixes and documentation improvements\n\n\n0.4 (2015-08-19)\n----------------\n\n- Add support for O(\u03b4) in-place BigArray.append() (commit__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9\n\n- Implement proper multithreading support (commit__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9\n\n- Implement proper RAM pages invalidation when backing ZODB objects are changed\n  from outside (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e\n\n- Fix all kind of failures that could happen when ZODB connection changes\n  worker thread in-between handling requests (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b\n\n- Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages\n  (`commit 1`__, 2__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66\n\n- Various bugfixes\n\n\n\n0.3 (2015-06-12)\n----------------\n\n- Add support for automatic BigArray -> ndarray conversion, so that e.g. the\n  following::\n\n    A = BigArray(...)\n    numpy.mean(A)       # passing BigArray to plain NumPy function\n\n  either succeeds, or raises MemoryError if not enough address space is\n  available to cover whole A. (current limitation is ~ 127TB on linux/amd64)\n\n  (commit__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6\n\n- Various bugfixes (build-fixes, crashes, overflows, etc)\n\n\n0.2 (2015-05-25)\n----------------\n\n- Add support for O(1) in-place BigArray.resize() (commit__)\n\n  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75\n\n- Various build bugfixes (older systems, non-std python, etc)\n\n\n0.1 (2015-04-03)\n----------------\n\n- Initial release",
    "bugtrack_url": null,
    "license": "GPLv3+ with wide exception for Open-Source",
    "summary": "Out-of-core NumPy arrays",
    "version": "0.13",
    "split_keywords": [
        "bigdata",
        "out-of-core",
        "numpy",
        "virtual-memory"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "363d9231212c2fb70ba24b3df850a979",
                "sha256": "b9a42b9006272fe128fb37dfe464a4ff643a502f169f5c62f93740ca73720c7c"
            },
            "downloads": -1,
            "filename": "wendelin.core-0.13.tar.gz",
            "has_sig": true,
            "md5_digest": "363d9231212c2fb70ba24b3df850a979",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 3105203,
            "upload_time": "2019-06-18T15:14:38",
            "upload_time_iso_8601": "2019-06-18T15:14:38.508643Z",
            "url": "https://files.pythonhosted.org/packages/9f/16/19c7e649aa33353fc89cc3018f57e50730bbbbb8fb1c6b90f1e549655000/wendelin.core-0.13.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2019-06-18 15:14:38",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "wendelin.core"
}
        
Elapsed time: 0.01980s