# refcount - Python classes for reference counting
[](https://github.com/csiro-hydroinformatics/pyrefcount/blob/master/LICENSE.txt)  [](https://pyrefcount.readthedocs.io/en/latest/?badge=latest) [](https://codecov.io/gh/csiro-hydroinformatics/pyrefcount) master: [](https://github.com/csiro-hydroinformatics/pyrefcount/actions/workflows/build-matrix.yml) testing: [](https://github.com/csiro-hydroinformatics/pyrefcount/actions/workflows/build-matrix.yml)
This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, **it is deceptively complicated to do so properly** and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in `refcount`.
`refcount` ( >=0.7) includes classes using [cffi](https://cffi.readthedocs.io/). Other low-level interoperability mechanisms may well be added in the future.
## License
MIT (see [License.txt](https://github.com/csiro-hydroinformatics/pyrefcount/blob/master/LICENSE.txt))
## Documentation
Hosted at [refcount via readthedocs.io](https://pyrefcount.readthedocs.io/en/latest/?badge=latest)
## Source code
The code repository is on [GitHub](https://github.com/csiro-hydroinformatics/pyrefcount).
## Installation
### conda-forge
Using `conda` or `mamba`:
```sh
mamba install -c conda-forge refcount
```
### pypi
```sh
pip install refcount
```
### From source (development)
```sh
pip install -r requirements.txt
pip install -e .
```
## Sample use
The following example is based on one of the unit tests.
Say we have a C++ library with objects and a C API:
```C++
#define TEST_DOG_PTR testnative::dog*
#define TEST_OWNER_PTR testnative::owner*
#define TEST_COUNTED_PTR testnative::reference_counter*
testnative::dog* create_dog();
testnative::owner* create_owner(testnative::dog* d);
void say_walk(testnative::owner* owner);
void release(testnative::reference_counter* obj);
// etc.
```
From the outside of the library the API is exported with opaque pointers `void*` (C structs pointers and native C99 types could be handled too).
```C++
void* create_dog();
void* create_owner(void* d);
void say_walk(void* owner);
void release(void* obj);
// etc.
```
Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via `release` C API calls, piggybacking the python GC instead.
```python
dog = Dog()
owner = DogOwner(dog)
owner.say_walk()
print(dog.position)
dog = None # the "native dog" is still alive though, as the owner incremented the ref count
owner = None
```
This is doable with `refcount` and the `cffi` package. One possible design is:
```python
ut_ffi = cffi.FFI()
ut_ffi.cdef('extern void* create_dog();')
ut_ffi.cdef('extern void* create_owner( void* d);')
ut_ffi.cdef('extern void say_walk( void* owner);')
ut_ffi.cdef('extern void release( void* obj);')
# etc.
ut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading
class CustomCffiNativeHandle(CffiNativeHandle):
def __init__(self, pointer, prior_ref_count = 0):
super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)
def _release_handle(self) -> bool:
ut_dll.release(self.get_handle())
return True
class Dog(CustomCffiNativeHandle):
def __init__(self, pointer = None):
if pointer is None:
pointer = ut_dll.create_dog()
super(Dog, self).__init__(pointer)
# etc.
class DogOwner(CustomCffiNativeHandle):
def __init__(self, dog):
super(DogOwner, self).__init__(None)
self._set_handle(ut_dll.create_owner(dog.get_handle()))
self.dog = dog
self.dog.add_ref() # Do note this important reference increment
def say_walk(self):
ut_dll.say_walk(self.get_handle())
def _release_handle(self) -> bool:
super(DogOwner, self)._release_handle()
# super(DogOwner, self)._release_handle()
self.dog.release()
return True
```
## Related work
### Ancestry, acknowledgements
This python package `refcount` actually spawned from prior work for interoperability between C++, R and .NET ([R.NET](https://github.com/rdotnet/rdotnet))
`refcount` features using `cffi` were also significantly informed by Kevin Plastow's [work](https://search.informit.com.au/documentSummary;dn=823898220073899;res=IELENG) while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.
In you have native interop needs you may also want to look at:
* the nuget package [dynamic-interop-dll](https://github.com/rdotnet/dynamic-interop-dll) for .NET/native interop.
* a set of mostly c++ software [tools for interop with C/C++](https://github.com/csiro-hydroinformatics/c-interop)
* a C# library for [generating interop glue code on top of C API glue code](https://github.com/csiro-hydroinformatics/c-api-wrapper-generation).
### Other python packages
`refcount` was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:
* [infi.pyutils](https://pypi.org/project/infi.pyutils/) contains a reference counting class.
Raw data
{
"_id": null,
"home_page": "https://github.com/csiro-hydroinformatics/pyrefcount",
"name": "refcount",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "interop python native-libraries reference-counting handle cffi",
"author": "Jean-Michel Perraud",
"author_email": "per202@csiro.au",
"download_url": "https://files.pythonhosted.org/packages/5f/ad/f8210b5db6f98d8607b7c7c726bd0198de46c323fc27b3066ffae0c8653b/refcount-1.2.0.zip",
"platform": null,
"description": "# refcount - Python classes for reference counting\n\n[](https://github.com/csiro-hydroinformatics/pyrefcount/blob/master/LICENSE.txt)  [](https://pyrefcount.readthedocs.io/en/latest/?badge=latest) [](https://codecov.io/gh/csiro-hydroinformatics/pyrefcount) master: [](https://github.com/csiro-hydroinformatics/pyrefcount/actions/workflows/build-matrix.yml) testing: [](https://github.com/csiro-hydroinformatics/pyrefcount/actions/workflows/build-matrix.yml)\n\n\n![Reference counted native handles](./docs/img/refcount-principles.png \"Reference counted native handles\")\n\nThis package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to \"simply\" maintaining a set of counters, **it is deceptively complicated to do so properly** and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in `refcount`.\n\n`refcount` ( >=0.7) includes classes using [cffi](https://cffi.readthedocs.io/). Other low-level interoperability mechanisms may well be added in the future.\n\n## License\n\nMIT (see [License.txt](https://github.com/csiro-hydroinformatics/pyrefcount/blob/master/LICENSE.txt))\n\n## Documentation\n\nHosted at [refcount via readthedocs.io](https://pyrefcount.readthedocs.io/en/latest/?badge=latest)\n\n## Source code\n\nThe code repository is on [GitHub](https://github.com/csiro-hydroinformatics/pyrefcount).\n\n## Installation\n\n### conda-forge\n\nUsing `conda` or `mamba`:\n\n```sh\nmamba install -c conda-forge refcount\n```\n\n### pypi\n\n```sh\npip install refcount\n```\n\n### From source (development)\n\n```sh\npip install -r requirements.txt\npip install -e .\n```\n\n## Sample use\n\nThe following example is based on one of the unit tests.\n\nSay we have a C++ library with objects and a C API:\n\n```C++\n#define TEST_DOG_PTR testnative::dog*\n#define TEST_OWNER_PTR testnative::owner*\n#define TEST_COUNTED_PTR testnative::reference_counter*\n\ntestnative::dog* create_dog();\ntestnative::owner* create_owner(testnative::dog* d);\nvoid say_walk(testnative::owner* owner);\nvoid release(testnative::reference_counter* obj);\n// etc.\n```\n\nFrom the outside of the library the API is exported with opaque pointers `void*` (C structs pointers and native C99 types could be handled too).\n\n```C++\nvoid* create_dog();\nvoid* create_owner(void* d);\nvoid say_walk(void* owner);\nvoid release(void* obj);\n// etc.\n```\n\nStarting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via `release` C API calls, piggybacking the python GC instead.\n\n```python\ndog = Dog()\nowner = DogOwner(dog)\nowner.say_walk()\nprint(dog.position)\ndog = None # the \"native dog\" is still alive though, as the owner incremented the ref count\nowner = None\n```\n\nThis is doable with `refcount` and the `cffi` package. One possible design is:\n\n```python\nut_ffi = cffi.FFI()\n\nut_ffi.cdef('extern void* create_dog();')\nut_ffi.cdef('extern void* create_owner( void* d);')\nut_ffi.cdef('extern void say_walk( void* owner);')\nut_ffi.cdef('extern void release( void* obj);')\n# etc.\n\nut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading\n\nclass CustomCffiNativeHandle(CffiNativeHandle):\n def __init__(self, pointer, prior_ref_count = 0):\n super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)\n\n def _release_handle(self) -> bool:\n ut_dll.release(self.get_handle())\n return True\n\nclass Dog(CustomCffiNativeHandle):\n def __init__(self, pointer = None):\n if pointer is None:\n pointer = ut_dll.create_dog()\n super(Dog, self).__init__(pointer)\n # etc.\n\nclass DogOwner(CustomCffiNativeHandle):\n\n def __init__(self, dog):\n super(DogOwner, self).__init__(None)\n self._set_handle(ut_dll.create_owner(dog.get_handle()))\n self.dog = dog\n self.dog.add_ref() # Do note this important reference increment\n\n def say_walk(self):\n ut_dll.say_walk(self.get_handle())\n\n def _release_handle(self) -> bool:\n super(DogOwner, self)._release_handle()\n # super(DogOwner, self)._release_handle()\n self.dog.release()\n return True\n```\n\n## Related work\n\n### Ancestry, acknowledgements\n\nThis python package `refcount` actually spawned from prior work for interoperability between C++, R and .NET ([R.NET](https://github.com/rdotnet/rdotnet))\n\n`refcount` features using `cffi` were also significantly informed by Kevin Plastow's [work](https://search.informit.com.au/documentSummary;dn=823898220073899;res=IELENG) while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.\n\nIn you have native interop needs you may also want to look at:\n\n* the nuget package [dynamic-interop-dll](https://github.com/rdotnet/dynamic-interop-dll) for .NET/native interop.\n* a set of mostly c++ software [tools for interop with C/C++](https://github.com/csiro-hydroinformatics/c-interop)\n* a C# library for [generating interop glue code on top of C API glue code](https://github.com/csiro-hydroinformatics/c-api-wrapper-generation).\n\n### Other python packages\n\n`refcount` was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:\n\n* [infi.pyutils](https://pypi.org/project/infi.pyutils/) contains a reference counting class.\n",
"bugtrack_url": null,
"license": "",
"summary": "A Python package for reference counting and interop with native pointers",
"version": "1.2.0",
"split_keywords": [
"interop",
"python",
"native-libraries",
"reference-counting",
"handle",
"cffi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "391d31a132cb4efc39111bed3a0f44c9850ca18fc22030aad3f9e5788a72d425",
"md5": "899cac5dc5ddf5bb876a56b5da8d3d4f",
"sha256": "b9934a38d7d9a5d644c21c3cbc10de69be7db0cb97a49294dff7306024d0782b"
},
"downloads": -1,
"filename": "refcount-1.2.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "899cac5dc5ddf5bb876a56b5da8d3d4f",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 15767,
"upload_time": "2023-01-25T00:54:23",
"upload_time_iso_8601": "2023-01-25T00:54:23.379956Z",
"url": "https://files.pythonhosted.org/packages/39/1d/31a132cb4efc39111bed3a0f44c9850ca18fc22030aad3f9e5788a72d425/refcount-1.2.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5fadf8210b5db6f98d8607b7c7c726bd0198de46c323fc27b3066ffae0c8653b",
"md5": "bb153dcedb911c39cdf792ac5c8969e5",
"sha256": "9672137bdc67b20f6442ac8304494b1c2bd00de5907f5009b202a38a17448738"
},
"downloads": -1,
"filename": "refcount-1.2.0.zip",
"has_sig": false,
"md5_digest": "bb153dcedb911c39cdf792ac5c8969e5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 24865,
"upload_time": "2023-01-25T00:54:24",
"upload_time_iso_8601": "2023-01-25T00:54:24.853336Z",
"url": "https://files.pythonhosted.org/packages/5f/ad/f8210b5db6f98d8607b7c7c726bd0198de46c323fc27b3066ffae0c8653b/refcount-1.2.0.zip",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-25 00:54:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "csiro-hydroinformatics",
"github_project": "pyrefcount",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"appveyor": true,
"requirements": [
{
"name": "cffi",
"specs": [
[
">=",
"1.11.5"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
">=",
"4.3.0"
]
]
}
],
"lcname": "refcount"
}
JSON
