This is a fork of `sqlitedict`.
`restricted-sqlitedict` deserializes values using the `restricted_loads` function to prevent code injection attacks.
See:
- `CVE-2024-35515 <https://wha13.github.io/2024/06/13/mfcve/>`_
- `Issue in the original repository <https://github.com/piskvorky/sqlitedict/issues/174>`_
- `Pickle's documentation <https://docs.python.org/3/library/pickle.html#restricting-globals>`_
The original README is below.
===================================================
sqlitedict -- persistent ``dict``, backed by SQLite
===================================================
|GithubActions|_
|License|_
.. |GithubActions| image:: https://github.com/RaRe-Technologies/sqlitedict/actions/workflows/python-package.yml/badge.svg
.. |Downloads| image:: https://img.shields.io/pypi/dm/sqlitedict.svg
.. |License| image:: https://img.shields.io/pypi/l/sqlitedict.svg
.. _GithubActions: https://github.com/RaRe-Technologies/sqlitedict/actions/workflows/python-package.yml
.. _Downloads: https://pypi.python.org/pypi/sqlitedict
.. _License: https://pypi.python.org/pypi/sqlitedict
A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic
dict-like interface and support for multi-thread access:
Usage
=====
Write
-----
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>> db = SqliteDict("example.sqlite")
>>>
>>> db["1"] = {"name": "first item"}
>>> db["2"] = {"name": "second item"}
>>> db["3"] = {"name": "yet another item"}
>>>
>>> # Commit to save the objects.
>>> db.commit()
>>>
>>> db["4"] = {"name": "yet another item"}
>>> # Oops, forgot to commit here, that object will never be saved.
>>> # Always remember to commit, or enable autocommit with SqliteDict("example.sqlite", autocommit=True)
>>> # Autocommit is off by default for performance.
>>>
>>> db.close()
Read
----
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>> db = SqliteDict("example.sqlite")
>>>
>>> print("There are %d items in the database" % len(db))
There are 3 items in the database
>>>
>>> # Standard dict interface. items() values() keys() etc...
>>> for key, item in db.items():
... print("%s=%s" % (key, item))
1={'name': 'first item'}
2={'name': 'second item'}
3={'name': 'yet another item'}
>>>
>>> db.close()
Efficiency
----------
By default, sqlitedict's exception handling favors verbosity over efficiency.
It extracts and outputs the outer exception stack to the error logs.
If you favor efficiency, then initialize the DB with outer_stack=False.
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>> db = SqliteDict("example.sqlite", outer_stack=False) # True is the default
>>> db[1]
{'name': 'first item'}
Context Manager
---------------
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>>
>>> # The database is automatically closed when leaving the with section.
>>> # Uncommitted objects are not saved on close. REMEMBER TO COMMIT!
>>>
>>> with SqliteDict("example.sqlite") as db:
... print("There are %d items in the database" % len(db))
There are 3 items in the database
Tables
------
A database file can store multiple tables.
A default table is used when no table name is specified.
Note: Writes are serialized, having multiple tables does not improve performance.
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>>
>>> products = SqliteDict("example.sqlite", tablename="product", autocommit=True)
>>> manufacturers = SqliteDict("example.sqlite", tablename="manufacturer", autocommit=True)
>>>
>>> products["1"] = {"name": "first item", "manufacturer_id": "1"}
>>> products["2"] = {"name": "second item", "manufacturer_id": "1"}
>>>
>>> manufacturers["1"] = {"manufacturer_name": "afactory", "location": "US"}
>>> manufacturers["2"] = {"manufacturer_name": "anotherfactory", "location": "UK"}
>>>
>>> tables = products.get_tablenames('example.sqlite')
>>> print(tables)
['unnamed', 'product', 'manufacturer']
>>>
>>> products.close()
>>> manufacturers.close()
In case you're wondering, the unnamed table comes from the previous examples,
where we did not specify a table name.
Serialization
-------------
Keys are strings. Values are any serializeable object.
By default Pickle is used internally to (de)serialize the values.
It's possible to use a custom (de)serializer, notably for JSON and for compression.
.. code-block:: python
>>> # Use JSON instead of pickle
>>> import json
>>> with SqliteDict("example.sqlite", encode=json.dumps, decode=json.loads) as mydict:
... pass
>>>
>>> # Apply zlib compression after pickling
>>> import zlib, pickle, sqlite3
>>>
>>> def my_encode(obj):
... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
>>>
>>> def my_decode(obj):
... return pickle.loads(zlib.decompress(bytes(obj)))
>>>
>>> with SqliteDict("example.sqlite", encode=my_encode, decode=my_decode) as mydict:
... pass
It's also possible to use a custom (de)serializer for keys to allow non-string keys.
.. code-block:: python
>>> # Use key encoding instead of default string keys only
>>> from sqlitedict import encode_key, decode_key
>>> with SqliteDict("example.sqlite", encode_key=encode_key, decode_key=decode_key) as mydict:
... pass
More
----
Functions are well documented, see docstrings directly in ``sqlitedict.py`` or call ``help(sqlitedict)``.
**Beware**: because of Python semantics, ``sqlitedict`` cannot know when a mutable
SqliteDict-backed entry was modified in RAM. You'll need to
explicitly assign the mutated object back to SqliteDict:
.. code-block:: python
>>> from sqlitedict import SqliteDict
>>> db = SqliteDict("example.sqlite")
>>> db["colors"] = {"red": (255, 0, 0)}
>>> db.commit()
>>>
>>> colors = db["colors"]
>>> colors["blue"] = (0, 0, 255) # sqlite DB not updated here!
>>> db["colors"] = colors # now updated
>>>
>>> db.commit() # remember to commit (or set autocommit)
>>> db.close()
Features
========
* Values can be **any picklable objects** (uses ``pickle`` with the highest protocol).
* Support for **multiple tables** (=dicts) living in the same database file.
* Support for **access from multiple threads** to the same connection (needed by e.g. Pyro).
Vanilla sqlite3 gives you ``ProgrammingError: SQLite objects created in a thread can
only be used in that same thread.``
Concurrent requests are still serialized internally, so this "multithreaded support"
**doesn't** give you any performance benefits. It is a work-around for sqlite limitations in Python.
* Support for **custom serialization or compression**:
.. code-block:: python
# use JSON instead of pickle
>>> import json
>>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)
# apply zlib compression after pickling
>>> import zlib, pickle, sqlite3
>>> def my_encode(obj):
... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
>>> def my_decode(obj):
... return pickle.loads(zlib.decompress(bytes(obj)))
>>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)
* sqlite is efficient and can work effectively with large databases (multi gigabytes), not limited by memory.
* sqlitedict is mostly a thin wrapper around sqlite.
* ``items()`` ``keys()`` ``values()`` are iterating one by one, the rows are loaded in a worker thread and queued in memory.
* ``len()`` is calling sqlite to count rows, that is scanning the whole table.
* For better performance, write objects in batch and ``commit()`` once.
Installation
============
The module has no dependencies beyond Python itself.
The minimum supported Python version is 3.7, continuously tested on Python 3.7, 3.8, 3.9, and 3.10 `on Travis <https://travis-ci.org/RaRe-Technologies/sqlitedict>`_.
Install or upgrade with::
pip install -U sqlitedict
or from the `source tar.gz <http://pypi.python.org/pypi/sqlitedict>`_::
python setup.py install
Contributions
=============
Testing
-------
Install::
$ pip install pytest coverage pytest-coverage
To perform all tests::
$ mkdir -p tests/db
$ pytest tests
$ python -m doctest README.rst
To perform all tests with coverage::
$ pytest tests --cov=sqlitedict
Comments, bug reports
---------------------
``sqlitedict`` resides on `github <https://github.com/RaRe-Technologies/sqlitedict>`_. You can file
issues or pull requests there.
License
=======
``sqlitedict`` is open source software released under the `Apache 2.0 license <http://opensource.org/licenses/apache2.0.php>`_.
Copyright (c) 2011-now `Radim Řehůřek <http://radimrehurek.com>`_ and contributors.
Housekeeping
============
Clean up the test database to keep each doctest run idempotent:
.. code-block:: python
>>> import os
>>> if __name__ == '__main__':
... os.unlink('example.sqlite')
Raw data
{
"_id": null,
"home_page": "https://github.com/mstopa/restricted-sqlitedict",
"name": "restricted-sqlitedict",
"maintainer": "Radim Rehurek",
"docs_url": null,
"requires_python": null,
"maintainer_email": "me@radimrehurek.com",
"keywords": "sqlite, persistent dict, multithreaded",
"author": "Radim Rehurek, Victor R. Escobar, Andrey Usov, Prasanna Swaminathan, Jeff Quast, Maciek Stopa",
"author_email": "me@radimrehurek.com",
"download_url": "https://files.pythonhosted.org/packages/8f/51/6b813927379e47f04eceac8c52831f55f22c318c16db656dcae321032877/restricted_sqlitedict-1.0.0.tar.gz",
"platform": "any",
"description": "This is a fork of `sqlitedict`. \n\n`restricted-sqlitedict` deserializes values using the `restricted_loads` function to prevent code injection attacks.\nSee:\n\n- `CVE-2024-35515 <https://wha13.github.io/2024/06/13/mfcve/>`_\n- `Issue in the original repository <https://github.com/piskvorky/sqlitedict/issues/174>`_\n- `Pickle's documentation <https://docs.python.org/3/library/pickle.html#restricting-globals>`_\n\n\nThe original README is below.\n\n===================================================\nsqlitedict -- persistent ``dict``, backed by SQLite\n===================================================\n\n|GithubActions|_\n|License|_\n\n.. |GithubActions| image:: https://github.com/RaRe-Technologies/sqlitedict/actions/workflows/python-package.yml/badge.svg\n.. |Downloads| image:: https://img.shields.io/pypi/dm/sqlitedict.svg\n.. |License| image:: https://img.shields.io/pypi/l/sqlitedict.svg\n.. _GithubActions: https://github.com/RaRe-Technologies/sqlitedict/actions/workflows/python-package.yml\n.. _Downloads: https://pypi.python.org/pypi/sqlitedict\n.. _License: https://pypi.python.org/pypi/sqlitedict\n\nA lightweight wrapper around Python's sqlite3 database with a simple, Pythonic\ndict-like interface and support for multi-thread access:\n\nUsage\n=====\n\nWrite\n-----\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>> db = SqliteDict(\"example.sqlite\")\n >>>\n >>> db[\"1\"] = {\"name\": \"first item\"}\n >>> db[\"2\"] = {\"name\": \"second item\"}\n >>> db[\"3\"] = {\"name\": \"yet another item\"}\n >>>\n >>> # Commit to save the objects.\n >>> db.commit()\n >>>\n >>> db[\"4\"] = {\"name\": \"yet another item\"}\n >>> # Oops, forgot to commit here, that object will never be saved.\n >>> # Always remember to commit, or enable autocommit with SqliteDict(\"example.sqlite\", autocommit=True)\n >>> # Autocommit is off by default for performance.\n >>>\n >>> db.close()\n\nRead\n----\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>> db = SqliteDict(\"example.sqlite\")\n >>>\n >>> print(\"There are %d items in the database\" % len(db))\n There are 3 items in the database\n >>>\n >>> # Standard dict interface. items() values() keys() etc...\n >>> for key, item in db.items():\n ... print(\"%s=%s\" % (key, item))\n 1={'name': 'first item'}\n 2={'name': 'second item'}\n 3={'name': 'yet another item'}\n >>>\n >>> db.close()\n\nEfficiency\n----------\n\nBy default, sqlitedict's exception handling favors verbosity over efficiency.\nIt extracts and outputs the outer exception stack to the error logs.\nIf you favor efficiency, then initialize the DB with outer_stack=False.\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>> db = SqliteDict(\"example.sqlite\", outer_stack=False) # True is the default\n >>> db[1]\n {'name': 'first item'}\n\nContext Manager\n---------------\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>>\n >>> # The database is automatically closed when leaving the with section.\n >>> # Uncommitted objects are not saved on close. REMEMBER TO COMMIT!\n >>>\n >>> with SqliteDict(\"example.sqlite\") as db:\n ... print(\"There are %d items in the database\" % len(db))\n There are 3 items in the database\n\nTables\n------\n\nA database file can store multiple tables.\nA default table is used when no table name is specified.\n\nNote: Writes are serialized, having multiple tables does not improve performance.\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>>\n >>> products = SqliteDict(\"example.sqlite\", tablename=\"product\", autocommit=True)\n >>> manufacturers = SqliteDict(\"example.sqlite\", tablename=\"manufacturer\", autocommit=True)\n >>>\n >>> products[\"1\"] = {\"name\": \"first item\", \"manufacturer_id\": \"1\"}\n >>> products[\"2\"] = {\"name\": \"second item\", \"manufacturer_id\": \"1\"}\n >>>\n >>> manufacturers[\"1\"] = {\"manufacturer_name\": \"afactory\", \"location\": \"US\"}\n >>> manufacturers[\"2\"] = {\"manufacturer_name\": \"anotherfactory\", \"location\": \"UK\"}\n >>>\n >>> tables = products.get_tablenames('example.sqlite')\n >>> print(tables)\n ['unnamed', 'product', 'manufacturer']\n >>>\n >>> products.close()\n >>> manufacturers.close()\n\nIn case you're wondering, the unnamed table comes from the previous examples,\nwhere we did not specify a table name.\n\nSerialization\n-------------\n\nKeys are strings. Values are any serializeable object.\n\nBy default Pickle is used internally to (de)serialize the values.\n\nIt's possible to use a custom (de)serializer, notably for JSON and for compression.\n\n.. code-block:: python\n\n >>> # Use JSON instead of pickle\n >>> import json\n >>> with SqliteDict(\"example.sqlite\", encode=json.dumps, decode=json.loads) as mydict:\n ... pass\n >>>\n >>> # Apply zlib compression after pickling\n >>> import zlib, pickle, sqlite3\n >>>\n >>> def my_encode(obj):\n ... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))\n >>>\n >>> def my_decode(obj):\n ... return pickle.loads(zlib.decompress(bytes(obj)))\n >>>\n >>> with SqliteDict(\"example.sqlite\", encode=my_encode, decode=my_decode) as mydict:\n ... pass\n\nIt's also possible to use a custom (de)serializer for keys to allow non-string keys.\n\n.. code-block:: python\n\n >>> # Use key encoding instead of default string keys only\n >>> from sqlitedict import encode_key, decode_key\n >>> with SqliteDict(\"example.sqlite\", encode_key=encode_key, decode_key=decode_key) as mydict:\n ... pass\n\nMore\n----\n\nFunctions are well documented, see docstrings directly in ``sqlitedict.py`` or call ``help(sqlitedict)``.\n\n**Beware**: because of Python semantics, ``sqlitedict`` cannot know when a mutable\nSqliteDict-backed entry was modified in RAM. You'll need to\nexplicitly assign the mutated object back to SqliteDict:\n\n.. code-block:: python\n\n >>> from sqlitedict import SqliteDict\n >>> db = SqliteDict(\"example.sqlite\")\n >>> db[\"colors\"] = {\"red\": (255, 0, 0)}\n >>> db.commit()\n >>>\n >>> colors = db[\"colors\"]\n >>> colors[\"blue\"] = (0, 0, 255) # sqlite DB not updated here!\n >>> db[\"colors\"] = colors # now updated\n >>>\n >>> db.commit() # remember to commit (or set autocommit)\n >>> db.close()\n\nFeatures\n========\n\n* Values can be **any picklable objects** (uses ``pickle`` with the highest protocol).\n* Support for **multiple tables** (=dicts) living in the same database file.\n* Support for **access from multiple threads** to the same connection (needed by e.g. Pyro).\n Vanilla sqlite3 gives you ``ProgrammingError: SQLite objects created in a thread can\n only be used in that same thread.``\n\n Concurrent requests are still serialized internally, so this \"multithreaded support\"\n **doesn't** give you any performance benefits. It is a work-around for sqlite limitations in Python.\n\n* Support for **custom serialization or compression**:\n\n.. code-block:: python\n\n # use JSON instead of pickle\n >>> import json\n >>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)\n\n # apply zlib compression after pickling\n >>> import zlib, pickle, sqlite3\n >>> def my_encode(obj):\n ... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))\n >>> def my_decode(obj):\n ... return pickle.loads(zlib.decompress(bytes(obj)))\n >>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)\n\n* sqlite is efficient and can work effectively with large databases (multi gigabytes), not limited by memory.\n* sqlitedict is mostly a thin wrapper around sqlite.\n* ``items()`` ``keys()`` ``values()`` are iterating one by one, the rows are loaded in a worker thread and queued in memory.\n* ``len()`` is calling sqlite to count rows, that is scanning the whole table.\n* For better performance, write objects in batch and ``commit()`` once.\n\nInstallation\n============\n\nThe module has no dependencies beyond Python itself.\nThe minimum supported Python version is 3.7, continuously tested on Python 3.7, 3.8, 3.9, and 3.10 `on Travis <https://travis-ci.org/RaRe-Technologies/sqlitedict>`_.\n\nInstall or upgrade with::\n\n pip install -U sqlitedict\n\nor from the `source tar.gz <http://pypi.python.org/pypi/sqlitedict>`_::\n\n python setup.py install\n\nContributions\n=============\n\nTesting\n-------\n\nInstall::\n\n $ pip install pytest coverage pytest-coverage\n\nTo perform all tests::\n\n $ mkdir -p tests/db\n $ pytest tests\n $ python -m doctest README.rst\n\nTo perform all tests with coverage::\n\n $ pytest tests --cov=sqlitedict\n\nComments, bug reports\n---------------------\n\n``sqlitedict`` resides on `github <https://github.com/RaRe-Technologies/sqlitedict>`_. You can file\nissues or pull requests there.\n\nLicense\n=======\n\n``sqlitedict`` is open source software released under the `Apache 2.0 license <http://opensource.org/licenses/apache2.0.php>`_.\nCopyright (c) 2011-now `Radim \u0158eh\u016f\u0159ek <http://radimrehurek.com>`_ and contributors.\n\nHousekeeping\n============\n\nClean up the test database to keep each doctest run idempotent:\n\n.. code-block:: python\n\n >>> import os\n >>> if __name__ == '__main__':\n ... os.unlink('example.sqlite')\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Fork of sqlitedict with restricted pickle loading.",
"version": "1.0.0",
"project_urls": {
"Download": "http://pypi.python.org/pypi/restricted-sqlitedict",
"Homepage": "https://github.com/mstopa/restricted-sqlitedict"
},
"split_keywords": [
"sqlite",
" persistent dict",
" multithreaded"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "048c65c15491bc0a5ca1f95dd514a09ef1ad8ee8b6cb7062bccd417a338cfe05",
"md5": "3be1de1ffb449953d73421d14bc2fbd0",
"sha256": "31cdeac0f48ee16cdbfeb8e8c14e8c1a17ebb67e6f098f5724a46515c2d2d3c6"
},
"downloads": -1,
"filename": "restricted_sqlitedict-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3be1de1ffb449953d73421d14bc2fbd0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 17458,
"upload_time": "2024-09-25T11:59:07",
"upload_time_iso_8601": "2024-09-25T11:59:07.851444Z",
"url": "https://files.pythonhosted.org/packages/04/8c/65c15491bc0a5ca1f95dd514a09ef1ad8ee8b6cb7062bccd417a338cfe05/restricted_sqlitedict-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8f516b813927379e47f04eceac8c52831f55f22c318c16db656dcae321032877",
"md5": "3e46f9807b0c1a46a46103e6118ccbd6",
"sha256": "f4c319ca51a01b37b2350f418640a9d80061402d58a7df7fcb1e0147b5d9cf13"
},
"downloads": -1,
"filename": "restricted_sqlitedict-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "3e46f9807b0c1a46a46103e6118ccbd6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 21729,
"upload_time": "2024-09-25T11:59:09",
"upload_time_iso_8601": "2024-09-25T11:59:09.581755Z",
"url": "https://files.pythonhosted.org/packages/8f/51/6b813927379e47f04eceac8c52831f55f22c318c16db656dcae321032877/restricted_sqlitedict-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-25 11:59:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mstopa",
"github_project": "restricted-sqlitedict",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "restricted-sqlitedict"
}
JSON
