| Name | borgstore JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | None |
| Summary | key/value store |
| upload_time | 2024-10-15 16:04:10 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | BSD |
| keywords |
kv
key/value
store
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
BorgStore
=========
A key/value store implementation in Python, supporting multiple backends.
Keys
----
A key (str) can look like:
- 0123456789abcdef... (usually a long, hex-encoded hash value)
- Any other pure ASCII string without "/" or ".." or " ".
Namespaces
----------
To keep stuff apart, keys should get prefixed with a namespace, like:
- config/settings
- meta/0123456789abcdef...
- data/0123456789abcdef...
Please note:
1. you should always use namespaces.
2. nested namespaces like namespace1/namespace2/key are not supported.
3. the code could work without a namespace (namespace ""), but then you
can't add another namespace later, because then you would have created
nested namespaces.
Values
------
Values can be any arbitrary binary data (bytes).
Store Operations
----------------
The high-level Store API implementation transparently deals with nesting and
soft deletion, so the caller doesn't have to care much for that and the Backend
API can be much simpler:
- create/destroy: initialize or remove the whole store.
- list: flat list of the items in the given namespace, with or without soft
deleted items.
- store: write a new item into the store (giving its key/value pair)
- load: read a value from the store (giving its key), partial loads giving
offset and/or size are supported.
- info: get information about an item via its key (exists? size? ...)
- delete: immediately remove an item from the store (giving its key)
- move: implements rename, soft delete / undelete, move to current
nesting level
- stats: api call counters, time spent in api methods, data volume/throughput
- latency/bandwidth emulator: can emulate higher latency (via BORGSTORE_LATENCY
[us]) and lower bandwidth (via BORGSTORE_BANDWIDTH [bit/s]) than what is
actually provided by the backend.
Automatic Nesting
-----------------
For the Store user, items have names like e.g.:
namespace/0123456789abcdef...
namespace/abcdef0123456789...
If there are very many items in the namespace, this could lead to scalability
issues in the backend, thus the Store implementation offers transparent
nesting, so that internally the Backend API will be called with
names like e.g.:
namespace/01/23/56/0123456789abcdef...
namespace/ab/cd/ef/abcdef0123456789...
The nesting depth can be configured from 0 (= no nesting) to N levels and
there can be different nesting configurations depending on the namespace.
The Store supports operating at different nesting levels in the same
namespace at the same time.
When using nesting depth > 0, the backends will assume that keys are hashes
(have hex digits) because some backends will want to pre-create the nesting
directories at backend initialization time to optimize for better performance
while using the backend.
Soft deletion
-------------
To soft delete an item (so its value could be still read or it could be
undeleted), the store just renames the item, appending ".del" to its name.
Undelete reverses this by removing the ".del" suffix from the name.
Some store operations have a boolean flag "deleted" to choose whether they
shall consider soft deleted items.
Backends
--------
The backend API is rather simple, one only needs to provide some very
basic operations.
Existing backends are listed below, more might come in future.
posixfs
~~~~~~~
Use storage on a local POSIX filesystem:
- URL: ``file:///absolute/path``
- it is the caller's task to create an absolute fs path from a relative one.
- namespaces: directories
- values: in key-named files
- pre-creates nesting directories
sftp
~~~~
Use storage on a sftp server:
- URL: ``sftp://user@server:port/relative/path`` (strongly recommended)
For user's and admin's convenience, mapping the URL path to the server fs path
depends on the server configuration (home directory, sshd/sftpd config, ...).
Usually the path is relative to the user's home directory.
- URL: ``sftp://user@server:port//absolute/path``
As this uses an absolute path, things are more difficult here:
- user's config might break if server admin moves a user home to a new location.
- users must know the full absolute path of space they have permission to use.
- namespaces: directories
- values: in key-named files
- pre-creates nesting directories
rclone
~~~~~~
Use storage on any of the many cloud providers `rclone <https://rclone.org/>`_ supports:
- URL: ``rclone:remote:path``, we just prefix "rclone:" and give all to the right
of that to rclone, see: https://rclone.org/docs/#syntax-of-remote-paths
- implementation of this primarily depends on the specific remote.
Scalability
-----------
- Count of key/value pairs stored in a namespace: automatic nesting is
provided for keys to address common scalability issues.
- Key size: there are no special provisions for extremely long keys (like:
more than backend limitations). Usually this is not a problem though.
- Value size: there are no special provisions for dealing with large value
sizes (like: more than free memory, more than backend storage limitations,
etc.). If one deals with very large values, one usually cuts them into
chunks before storing them into the store.
- Partial loads improve performance by avoiding a full load if only a part
of the value is needed (e.g. a header with metadata).
Installation
------------
Install without the ``sftp:`` backend::
pip install borgstore
Install with the ``sftp:`` backend (more dependencies)::
pip install "borgstore[sftp]"
Please note that ``rclone:`` also supports sftp remotes.
Want a demo?
------------
Run this to get instructions how to run the demo:
python3 -m borgstore
State of this project
---------------------
**API is still unstable and expected to change as development goes on.**
**As long as the API is unstable, there will be no data migration tools,
like e.g. for upgrading an existing store's data to a new release.**
There are tests and they succeed for the basic functionality, so some of the
stuff is already working well.
There might be missing features or optimization potential, feedback welcome!
There are a lot of possible, but still missing backends. If you want to create
and support one: pull requests are welcome.
Borg?
-----
Please note that this code is currently **not** used by the stable release of
BorgBackup (aka "borg"), but only by borg2 beta 10+ and master branch.
License
-------
BSD license.
Raw data
{
"_id": null,
"home_page": null,
"name": "borgstore",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "kv, key/value, store",
"author": null,
"author_email": "Thomas Waldmann <tw@waldmann-edv.de>",
"download_url": "https://files.pythonhosted.org/packages/e9/7b/3dcbb254138039453d504e8d57d2c9e0339a7c847af42d7d62279483fe28/borgstore-0.1.0.tar.gz",
"platform": null,
"description": "BorgStore\n=========\n\nA key/value store implementation in Python, supporting multiple backends.\n\nKeys\n----\n\nA key (str) can look like:\n\n- 0123456789abcdef... (usually a long, hex-encoded hash value)\n- Any other pure ASCII string without \"/\" or \"..\" or \" \".\n\n\nNamespaces\n----------\n\nTo keep stuff apart, keys should get prefixed with a namespace, like:\n\n- config/settings\n- meta/0123456789abcdef...\n- data/0123456789abcdef...\n\nPlease note:\n\n1. you should always use namespaces.\n2. nested namespaces like namespace1/namespace2/key are not supported.\n3. the code could work without a namespace (namespace \"\"), but then you\n can't add another namespace later, because then you would have created\n nested namespaces.\n\nValues\n------\n\nValues can be any arbitrary binary data (bytes).\n\nStore Operations\n----------------\n\nThe high-level Store API implementation transparently deals with nesting and\nsoft deletion, so the caller doesn't have to care much for that and the Backend\nAPI can be much simpler:\n\n- create/destroy: initialize or remove the whole store.\n- list: flat list of the items in the given namespace, with or without soft\n deleted items.\n- store: write a new item into the store (giving its key/value pair)\n- load: read a value from the store (giving its key), partial loads giving\n offset and/or size are supported.\n- info: get information about an item via its key (exists? size? ...)\n- delete: immediately remove an item from the store (giving its key)\n- move: implements rename, soft delete / undelete, move to current\n nesting level\n- stats: api call counters, time spent in api methods, data volume/throughput\n- latency/bandwidth emulator: can emulate higher latency (via BORGSTORE_LATENCY\n [us]) and lower bandwidth (via BORGSTORE_BANDWIDTH [bit/s]) than what is\n actually provided by the backend.\n\nAutomatic Nesting\n-----------------\n\nFor the Store user, items have names like e.g.:\n\nnamespace/0123456789abcdef...\nnamespace/abcdef0123456789...\n\nIf there are very many items in the namespace, this could lead to scalability\nissues in the backend, thus the Store implementation offers transparent\nnesting, so that internally the Backend API will be called with\nnames like e.g.:\n\nnamespace/01/23/56/0123456789abcdef...\nnamespace/ab/cd/ef/abcdef0123456789...\n\nThe nesting depth can be configured from 0 (= no nesting) to N levels and\nthere can be different nesting configurations depending on the namespace.\n\nThe Store supports operating at different nesting levels in the same\nnamespace at the same time.\n\nWhen using nesting depth > 0, the backends will assume that keys are hashes\n(have hex digits) because some backends will want to pre-create the nesting\ndirectories at backend initialization time to optimize for better performance\nwhile using the backend.\n\nSoft deletion\n-------------\n\nTo soft delete an item (so its value could be still read or it could be\nundeleted), the store just renames the item, appending \".del\" to its name.\n\nUndelete reverses this by removing the \".del\" suffix from the name.\n\nSome store operations have a boolean flag \"deleted\" to choose whether they\nshall consider soft deleted items.\n\nBackends\n--------\n\nThe backend API is rather simple, one only needs to provide some very\nbasic operations.\n\nExisting backends are listed below, more might come in future.\n\nposixfs\n~~~~~~~\n\nUse storage on a local POSIX filesystem:\n\n- URL: ``file:///absolute/path``\n- it is the caller's task to create an absolute fs path from a relative one.\n- namespaces: directories\n- values: in key-named files\n- pre-creates nesting directories\n\nsftp\n~~~~\n\nUse storage on a sftp server:\n\n- URL: ``sftp://user@server:port/relative/path`` (strongly recommended)\n\n For user's and admin's convenience, mapping the URL path to the server fs path\n depends on the server configuration (home directory, sshd/sftpd config, ...).\n Usually the path is relative to the user's home directory.\n- URL: ``sftp://user@server:port//absolute/path``\n\n As this uses an absolute path, things are more difficult here:\n\n - user's config might break if server admin moves a user home to a new location.\n - users must know the full absolute path of space they have permission to use.\n- namespaces: directories\n- values: in key-named files\n- pre-creates nesting directories\n\nrclone\n~~~~~~\n\nUse storage on any of the many cloud providers `rclone <https://rclone.org/>`_ supports:\n\n- URL: ``rclone:remote:path``, we just prefix \"rclone:\" and give all to the right\n of that to rclone, see: https://rclone.org/docs/#syntax-of-remote-paths\n- implementation of this primarily depends on the specific remote.\n\n\nScalability\n-----------\n\n- Count of key/value pairs stored in a namespace: automatic nesting is\n provided for keys to address common scalability issues.\n- Key size: there are no special provisions for extremely long keys (like:\n more than backend limitations). Usually this is not a problem though.\n- Value size: there are no special provisions for dealing with large value\n sizes (like: more than free memory, more than backend storage limitations,\n etc.). If one deals with very large values, one usually cuts them into\n chunks before storing them into the store.\n- Partial loads improve performance by avoiding a full load if only a part\n of the value is needed (e.g. a header with metadata).\n\nInstallation\n------------\n\nInstall without the ``sftp:`` backend::\n\n pip install borgstore\n\nInstall with the ``sftp:`` backend (more dependencies)::\n\n pip install \"borgstore[sftp]\"\n\nPlease note that ``rclone:`` also supports sftp remotes.\n\nWant a demo?\n------------\n\nRun this to get instructions how to run the demo:\n\npython3 -m borgstore\n\nState of this project\n---------------------\n\n**API is still unstable and expected to change as development goes on.**\n\n**As long as the API is unstable, there will be no data migration tools,\nlike e.g. for upgrading an existing store's data to a new release.**\n\nThere are tests and they succeed for the basic functionality, so some of the\nstuff is already working well.\n\nThere might be missing features or optimization potential, feedback welcome!\n\nThere are a lot of possible, but still missing backends. If you want to create\nand support one: pull requests are welcome.\n\nBorg?\n-----\n\nPlease note that this code is currently **not** used by the stable release of\nBorgBackup (aka \"borg\"), but only by borg2 beta 10+ and master branch.\n\nLicense\n-------\n\nBSD license.\n\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "key/value store",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/borgbackup/borgstore"
},
"split_keywords": [
"kv",
" key/value",
" store"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e97b3dcbb254138039453d504e8d57d2c9e0339a7c847af42d7d62279483fe28",
"md5": "7bfaa1b1028efe540cb3d4544bffeed5",
"sha256": "bfbf7dd970f3cb9fe124ddbdf66c382dcdbd563ee698f5b36b6025cf9292dd2c"
},
"downloads": -1,
"filename": "borgstore-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "7bfaa1b1028efe540cb3d4544bffeed5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 23255,
"upload_time": "2024-10-15T16:04:10",
"upload_time_iso_8601": "2024-10-15T16:04:10.152537Z",
"url": "https://files.pythonhosted.org/packages/e9/7b/3dcbb254138039453d504e8d57d2c9e0339a7c847af42d7d62279483fe28/borgstore-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-15 16:04:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "borgbackup",
"github_project": "borgstore",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "borgstore"
}
