smmap


Namesmmap JSON
Version 3.0.5 PyPI version JSON
download
home_pagehttps://github.com/gitpython-developers/smmap
SummaryA pure Python implementation of a sliding window memory map manager
upload_time2021-01-23 02:01:18
maintainer
docs_urlhttps://pythonhosted.org/smmap/
authorSebastian Thiel
requires_python>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
licenseBSD
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            ## Motivation

When reading from many possibly large files in a fashion similar to random access, it is usually the fastest and most efficient to use memory maps.

Although memory maps have many advantages, they represent a very limited system resource as every map uses one file descriptor, whose amount is limited per process. On 32 bit systems, the amount of memory you can have mapped at a time is naturally limited to theoretical 4GB of memory, which may not be enough for some applications.


## Limitations

* **System resources (file-handles) are likely to be leaked!** This is due to the library authors reliance on a deterministic `__del__()` destructor.
* The memory access is read-only by design.


## Overview

![Python package](https://github.com/gitpython-developers/smmap/workflows/Python%20package/badge.svg)

Smmap wraps an interface around mmap and tracks the mapped files as well as the amount of clients who use it. If the system runs out of resources, or if a memory limit is reached, it will automatically unload unused maps to allow continued operation.

To allow processing large files even on 32 bit systems, it allows only portions of the file to be mapped. Once the user reads beyond the mapped region, smmap will automatically map the next required region, unloading unused regions using a LRU algorithm.

The interface also works around the missing offset parameter in python implementations up to python 2.5.

Although the library can be used most efficiently with its native interface, a Buffer implementation is provided to hide these details behind a simple string-like interface.

For performance critical 64 bit applications, a simplified version of memory mapping is provided which always maps the whole file, but still provides the benefit of unloading unused mappings on demand.



## Prerequisites

* Python 2.7 or 3.4+
* OSX, Windows or Linux

The package was tested on all of the previously mentioned configurations.

## Installing smmap

[![Documentation Status](https://readthedocs.org/projects/smmap/badge/?version=latest)](https://readthedocs.org/projects/smmap/?badge=latest)

Its easiest to install smmap using the [pip](http://www.pip-installer.org/en/latest) program:

```bash
$ pip install smmap
```

As the command will install smmap in your respective python distribution, you will most likely need root permissions to authorize the required changes.

If you have downloaded the source archive, the package can be installed by running the `setup.py` script:

```bash
$ python setup.py install
```

It is advised to have a look at the **Usage Guide** for a brief introduction on the different database implementations.



## Homepage and Links

The project is home on github at https://github.com/gitpython-developers/smmap .

The latest source can be cloned from github as well:

* git://github.com/gitpython-developers/smmap.git


For support, please use the git-python mailing list:

* http://groups.google.com/group/git-python


Issues can be filed on github:

* https://github.com/gitpython-developers/smmap/issues

A link to the pypi page related to this repository:

* https://pypi.org/project/smmap/


## License Information

*smmap* is licensed under the New BSD License.




            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/gitpython-developers/smmap",
    "name": "smmap",
    "maintainer": "",
    "docs_url": "https://pythonhosted.org/smmap/",
    "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
    "maintainer_email": "",
    "keywords": "",
    "author": "Sebastian Thiel",
    "author_email": "byronimo@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/2b/6f/d48bbed5aa971943759f4ede3f12dca40aa7faa44f22bad483de86780508/smmap-3.0.5.tar.gz",
    "platform": "any",
    "description": "## Motivation\n\nWhen reading from many possibly large files in a fashion similar to random access, it is usually the fastest and most efficient to use memory maps.\n\nAlthough memory maps have many advantages, they represent a very limited system resource as every map uses one file descriptor, whose amount is limited per process. On 32 bit systems, the amount of memory you can have mapped at a time is naturally limited to theoretical 4GB of memory, which may not be enough for some applications.\n\n\n## Limitations\n\n* **System resources (file-handles) are likely to be leaked!** This is due to the library authors reliance on a deterministic `__del__()` destructor.\n* The memory access is read-only by design.\n\n\n## Overview\n\n![Python package](https://github.com/gitpython-developers/smmap/workflows/Python%20package/badge.svg)\n\nSmmap wraps an interface around mmap and tracks the mapped files as well as the amount of clients who use it. If the system runs out of resources, or if a memory limit is reached, it will automatically unload unused maps to allow continued operation.\n\nTo allow processing large files even on 32 bit systems, it allows only portions of the file to be mapped. Once the user reads beyond the mapped region, smmap will automatically map the next required region, unloading unused regions using a LRU algorithm.\n\nThe interface also works around the missing offset parameter in python implementations up to python 2.5.\n\nAlthough the library can be used most efficiently with its native interface, a Buffer implementation is provided to hide these details behind a simple string-like interface.\n\nFor performance critical 64 bit applications, a simplified version of memory mapping is provided which always maps the whole file, but still provides the benefit of unloading unused mappings on demand.\n\n\n\n## Prerequisites\n\n* Python 2.7 or 3.4+\n* OSX, Windows or Linux\n\nThe package was tested on all of the previously mentioned configurations.\n\n## Installing smmap\n\n[![Documentation Status](https://readthedocs.org/projects/smmap/badge/?version=latest)](https://readthedocs.org/projects/smmap/?badge=latest)\n\nIts easiest to install smmap using the [pip](http://www.pip-installer.org/en/latest) program:\n\n```bash\n$ pip install smmap\n```\n\nAs the command will install smmap in your respective python distribution, you will most likely need root permissions to authorize the required changes.\n\nIf you have downloaded the source archive, the package can be installed by running the `setup.py` script:\n\n```bash\n$ python setup.py install\n```\n\nIt is advised to have a look at the **Usage Guide** for a brief introduction on the different database implementations.\n\n\n\n## Homepage and Links\n\nThe project is home on github at https://github.com/gitpython-developers/smmap .\n\nThe latest source can be cloned from github as well:\n\n* git://github.com/gitpython-developers/smmap.git\n\n\nFor support, please use the git-python mailing list:\n\n* http://groups.google.com/group/git-python\n\n\nIssues can be filed on github:\n\n* https://github.com/gitpython-developers/smmap/issues\n\nA link to the pypi page related to this repository:\n\n* https://pypi.org/project/smmap/\n\n\n## License Information\n\n*smmap* is licensed under the New BSD License.\n\n\n\n",
    "bugtrack_url": null,
    "license": "BSD",
    "summary": "A pure Python implementation of a sliding window memory map manager",
    "version": "3.0.5",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ad0d21e1306193b8cb1f735d64a6375b",
                "sha256": "7bfcf367828031dc893530a29cb35eb8c8f2d7c8f2d0989354d75d24c8573714"
            },
            "downloads": -1,
            "filename": "smmap-3.0.5-py2.py3-none-any.whl",
            "has_sig": true,
            "md5_digest": "ad0d21e1306193b8cb1f735d64a6375b",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
            "size": 25065,
            "upload_time": "2021-01-23T02:01:16",
            "upload_time_iso_8601": "2021-01-23T02:01:16.563509Z",
            "url": "https://files.pythonhosted.org/packages/d5/1e/6130925131f639b2acde0f7f18b73e33ce082ff2d90783c436b52040af5a/smmap-3.0.5-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "84b4f0ed6522cc1d12a9d8413e8f7bd7",
                "sha256": "84c2751ef3072d4f6b2785ec7ee40244c6f45eb934d9e543e2c51f1bd3d54c50"
            },
            "downloads": -1,
            "filename": "smmap-3.0.5.tar.gz",
            "has_sig": true,
            "md5_digest": "84b4f0ed6522cc1d12a9d8413e8f7bd7",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
            "size": 23228,
            "upload_time": "2021-01-23T02:01:18",
            "upload_time_iso_8601": "2021-01-23T02:01:18.358452Z",
            "url": "https://files.pythonhosted.org/packages/2b/6f/d48bbed5aa971943759f4ede3f12dca40aa7faa44f22bad483de86780508/smmap-3.0.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-01-23 02:01:18",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "gitpython-developers",
    "error": "Could not fetch GitHub repository",
    "lcname": "smmap"
}
        
Elapsed time: 0.18108s