# kvfile
[](https://travis-ci.org/akariv/kvfile)
[](https://coveralls.io/r/akariv/kvfile?branch=master)
A simple Key-Value store that's file based - so can accommodate large data sets with a small memory footprint.
Internally will use the faster `leveldb` as a storage backend or `sqlite3` as fallback if `leveldb` is not available.
## The Basics
The API should feel familiar to anyone working with Python.
It exposes `get`, `keys` and `items` for reading from the DB, and `set` for setting a value in the DB.
### Initializing
```python
import datetime
import decimal
from kvfile import KVFile
kv = KVFile()
```
### Setting values
```python
kv.set('s', 'value')
kv.set('i', 123)
kv.set('d', datetime.datetime.fromtimestamp(12325))
kv.set('n', decimal.Decimal('1234.56'))
kv.set('ss', set(range(10)))
kv.set('o', dict(d=decimal.Decimal('1234.58'),
n=datetime.datetime.fromtimestamp(12325)))
```
### Getting values
```python
assert kv.get('s') == 'value'
assert kv.get('i') == 123
assert kv.get('d') == datetime.datetime.fromtimestamp(12325)
assert kv.get('n') == decimal.Decimal('1234.56')
assert kv.get('ss') == set(range(10))
assert kv.get('o') == dict(d=decimal.Decimal('1234.58'),
n=datetime.datetime.fromtimestamp(12325))
```
### Listing values
`keys()` and `items()` methods return a generator yielding the values for efficient stream processing.
The returned data is sorted ascending (by default) based on the keys
```python
assert list(kv.keys()) == ['d', 'i', 'n', 'o', 's', 'ss']
assert list(kv.items()) == [
('d', datetime.datetime.fromtimestamp(12325)),
('i', 123),
('n', decimal.Decimal('1234.56')),
('o', {'d': decimal.Decimal('1234.58'),
'n': datetime.datetime.fromtimestamp(12325)}),
('s', 'value'),
('ss', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
]
```
Set the `reverse` argument to True for the `keys()` and `items()` methods to sort in descending order.
### Bulk inserting data
The SQLite DB backend can be very slow when bulk inserting data. You can use the insert method to insert efficiently in bulk.
```python
kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)))
```
The batch size is 1000 by default, you should modify it depending on the size of your data and available memory.
```python
kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)), batch_size=40000)
```
If you are inserting data from a generator and need to use the inserted data, use `insert_generator` method:
```python
for key, value in kv.insert_generator(((str(i), ':{}'.format(i)) for i in range(50)), batch_size=10):
print(key, value)
```
## Installing leveldb
On Debian based Linux:
```bash
$ apt-get install libleveldb-dev libleveldb1
```
On Alpine based Linux:
```bash
$ apk --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --update add leveldb leveldb-dev
```
On OS X:
```bash
$ brew install leveldb
```
Raw data
{
"_id": null,
"home_page": "https://github.com/akariv/kvstore",
"name": "kvfile",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "data",
"author": "Adam Kariv",
"author_email": "adam.kariv@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/9f/5c/605d59e80fcc7ea9a520bdd8a5713e2de9cc2952a0ec2cce92135a9fd885/kvfile-1.1.2.tar.gz",
"platform": null,
"description": "# kvfile\n\n[](https://travis-ci.org/akariv/kvfile)\n[](https://coveralls.io/r/akariv/kvfile?branch=master)\n\nA simple Key-Value store that's file based - so can accommodate large data sets with a small memory footprint.\n\nInternally will use the faster `leveldb` as a storage backend or `sqlite3` as fallback if `leveldb` is not available.\n\n## The Basics\n\nThe API should feel familiar to anyone working with Python.\nIt exposes `get`, `keys` and `items` for reading from the DB, and `set` for setting a value in the DB.\n\n### Initializing\n\n```python\nimport datetime\nimport decimal\n\nfrom kvfile import KVFile\n\nkv = KVFile()\n```\n\n### Setting values\n\n```python\nkv.set('s', 'value')\nkv.set('i', 123)\nkv.set('d', datetime.datetime.fromtimestamp(12325))\nkv.set('n', decimal.Decimal('1234.56'))\nkv.set('ss', set(range(10)))\nkv.set('o', dict(d=decimal.Decimal('1234.58'), \n n=datetime.datetime.fromtimestamp(12325)))\n```\n\n### Getting values\n\n```python\nassert kv.get('s') == 'value'\nassert kv.get('i') == 123\nassert kv.get('d') == datetime.datetime.fromtimestamp(12325)\nassert kv.get('n') == decimal.Decimal('1234.56')\nassert kv.get('ss') == set(range(10))\nassert kv.get('o') == dict(d=decimal.Decimal('1234.58'), \n n=datetime.datetime.fromtimestamp(12325))\n```\n\n### Listing values\n\n`keys()` and `items()` methods return a generator yielding the values for efficient stream processing.\n\nThe returned data is sorted ascending (by default) based on the keys\n\n```python\nassert list(kv.keys()) == ['d', 'i', 'n', 'o', 's', 'ss']\nassert list(kv.items()) == [\n ('d', datetime.datetime.fromtimestamp(12325)), \n ('i', 123), \n ('n', decimal.Decimal('1234.56')), \n ('o', {'d': decimal.Decimal('1234.58'), \n 'n': datetime.datetime.fromtimestamp(12325)}), \n ('s', 'value'), \n ('ss', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})\n]\n```\n\nSet the `reverse` argument to True for the `keys()` and `items()` methods to sort in descending order.\n\n### Bulk inserting data\n\nThe SQLite DB backend can be very slow when bulk inserting data. You can use the insert method to insert efficiently in bulk.\n\n```python\nkv.insert(((str(i), ':{}'.format(i)) for i in range(50000)))\n```\n\nThe batch size is 1000 by default, you should modify it depending on the size of your data and available memory.\n\n```python\nkv.insert(((str(i), ':{}'.format(i)) for i in range(50000)), batch_size=40000)\n```\n\nIf you are inserting data from a generator and need to use the inserted data, use `insert_generator` method:\n\n```python\nfor key, value in kv.insert_generator(((str(i), ':{}'.format(i)) for i in range(50)), batch_size=10):\n print(key, value)\n```\n\n## Installing leveldb\n\nOn Debian based Linux:\n```bash\n$ apt-get install libleveldb-dev libleveldb1\n```\n\nOn Alpine based Linux:\n```bash\n$ apk --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --update add leveldb leveldb-dev\n```\n\nOn OS X:\n```bash\n$ brew install leveldb\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Simple File-based KV-Store",
"version": "1.1.2",
"project_urls": {
"Homepage": "https://github.com/akariv/kvstore"
},
"split_keywords": [
"data"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "40fb80e3f56d31730eee744418451a2baa12903b33b80823777a03875dcb7c24",
"md5": "5ad58490efd8ac0dcc4af6a434902dd7",
"sha256": "ac40a79b2df1b7b3806c336827327d4ff52d3df975dd44b4543b443bf686fed6"
},
"downloads": -1,
"filename": "kvfile-1.1.2-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "5ad58490efd8ac0dcc4af6a434902dd7",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 9384,
"upload_time": "2024-04-10T06:21:00",
"upload_time_iso_8601": "2024-04-10T06:21:00.107486Z",
"url": "https://files.pythonhosted.org/packages/40/fb/80e3f56d31730eee744418451a2baa12903b33b80823777a03875dcb7c24/kvfile-1.1.2-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9f5c605d59e80fcc7ea9a520bdd8a5713e2de9cc2952a0ec2cce92135a9fd885",
"md5": "ccda6adc1d81891ef53f0d8d2efc57a4",
"sha256": "5af57c40aa0f26c60b715360d08b6e46d13f3771466f40444133c752f34a581d"
},
"downloads": -1,
"filename": "kvfile-1.1.2.tar.gz",
"has_sig": false,
"md5_digest": "ccda6adc1d81891ef53f0d8d2efc57a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11696,
"upload_time": "2024-04-10T06:21:01",
"upload_time_iso_8601": "2024-04-10T06:21:01.887465Z",
"url": "https://files.pythonhosted.org/packages/9f/5c/605d59e80fcc7ea9a520bdd8a5713e2de9cc2952a0ec2cce92135a9fd885/kvfile-1.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-10 06:21:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "akariv",
"github_project": "kvstore",
"github_not_found": true,
"lcname": "kvfile"
}
JSON
