.. image:: https://readthedocs.org/projects/fast-dynamodb-json/badge/?version=latest
:target: https://fast-dynamodb-json.readthedocs.io/en/latest/
:alt: Documentation Status
.. image:: https://github.com/MacHu-GWU/fast_dynamodb_json-project/actions/workflows/main.yml/badge.svg
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/actions?query=workflow:CI
.. image:: https://codecov.io/gh/MacHu-GWU/fast_dynamodb_json-project/branch/main/graph/badge.svg
:target: https://codecov.io/gh/MacHu-GWU/fast_dynamodb_json-project
.. image:: https://img.shields.io/pypi/v/fast-dynamodb-json.svg
:target: https://pypi.python.org/pypi/fast-dynamodb-json
.. image:: https://img.shields.io/pypi/l/fast-dynamodb-json.svg
:target: https://pypi.python.org/pypi/fast-dynamodb-json
.. image:: https://img.shields.io/pypi/pyversions/fast-dynamodb-json.svg
:target: https://pypi.python.org/pypi/fast-dynamodb-json
.. image:: https://img.shields.io/badge/Release_History!--None.svg?style=social
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/blob/main/release-history.rst
.. image:: https://img.shields.io/badge/STAR_Me_on_GitHub!--None.svg?style=social
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project
------
.. image:: https://img.shields.io/badge/Link-Document-blue.svg
:target: https://fast-dynamodb-json.readthedocs.io/en/latest/
.. image:: https://img.shields.io/badge/Link-API-blue.svg
:target: https://fast-dynamodb-json.readthedocs.io/en/latest/py-modindex.html
.. image:: https://img.shields.io/badge/Link-Install-blue.svg
:target: `install`_
.. image:: https://img.shields.io/badge/Link-GitHub-blue.svg
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project
.. image:: https://img.shields.io/badge/Link-Submit_Issue-blue.svg
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/issues
.. image:: https://img.shields.io/badge/Link-Request_Feature-blue.svg
:target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/issues
.. image:: https://img.shields.io/badge/Link-Download-blue.svg
:target: https://pypi.org/pypi/fast-dynamodb-json#files
Welcome to ``fast_dynamodb_json`` Documentation
==============================================================================
.. image:: https://fast-dynamodb-json.readthedocs.io/en/latest/_static/fast_dynamodb_json-logo.png
:target: https://fast-dynamodb-json.readthedocs.io/en/latest/
AWS DynamoDB is a powerful NoSQL database service, but it comes with a unique challenge: its data is not stored in standard JSON format. Instead, DynamoDB encodes everything as strings. For example, a string "hello" is encoded as ``{"S": "hello"}``, and a number 123 as ``{"N": "123"}``. This encoding necessitates deserialization into regular Python dictionaries for data processing, which can be a significant bottleneck in high-volume operations.
Existing Python libraries like `pynamodb` and `dynamodb_json` offer solutions, but their implementations rely on Python for-loops and recursive programming. While functional, these approaches struggle with performance when handling large volumes of DynamoDB items. Moreover, their design doesn't lend itself well to vectorized computing or optimization through JIT technologies like PyPy or Numba.
As the founder of a startup and a former Amazon Web Services data architect, I've encountered this challenge firsthand. Our company operates an analytics data lake and data pipeline processing billions of DynamoDB items daily for our customers. This scale demanded a more efficient solution.
Enter `fast_dynamodb_json` - a revolutionary approach to deserializing large quantities of DynamoDB items. Our benchmarks demonstrate that this new method is up to 5000 times faster than pure Python implementations, maintaining this impressive speed advantage across both simple and complex datasets.
This project aims to dramatically improve the efficiency of DynamoDB data processing, enabling businesses to handle massive datasets with unprecedented speed and resource efficiency.
.. _install:
Install
------------------------------------------------------------------------------
``fast_dynamodb_json`` is released on PyPI, so all you need is to:
.. code-block:: console
$ pip install fast-dynamodb-json
To upgrade to latest version:
.. code-block:: console
$ pip install --upgrade fast-dynamodb-json
Raw data
{
"_id": null,
"home_page": "https://github.com/MacHu-GWU/fast_dynamodb_json-project",
"name": "fast-dynamodb-json",
"maintainer": "Sanhe Hu",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "husanhe@email.com",
"keywords": null,
"author": "Sanhe Hu",
"author_email": "husanhe@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e3/8d/a4c624e3e43898daf06e18e76870e6a49adfd1f8151abfb010a50c4c8a27/fast_dynamodb_json-0.1.1.tar.gz",
"platform": "Windows",
"description": "\n.. image:: https://readthedocs.org/projects/fast-dynamodb-json/badge/?version=latest\n :target: https://fast-dynamodb-json.readthedocs.io/en/latest/\n :alt: Documentation Status\n\n.. image:: https://github.com/MacHu-GWU/fast_dynamodb_json-project/actions/workflows/main.yml/badge.svg\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/actions?query=workflow:CI\n\n.. image:: https://codecov.io/gh/MacHu-GWU/fast_dynamodb_json-project/branch/main/graph/badge.svg\n :target: https://codecov.io/gh/MacHu-GWU/fast_dynamodb_json-project\n\n.. image:: https://img.shields.io/pypi/v/fast-dynamodb-json.svg\n :target: https://pypi.python.org/pypi/fast-dynamodb-json\n\n.. image:: https://img.shields.io/pypi/l/fast-dynamodb-json.svg\n :target: https://pypi.python.org/pypi/fast-dynamodb-json\n\n.. image:: https://img.shields.io/pypi/pyversions/fast-dynamodb-json.svg\n :target: https://pypi.python.org/pypi/fast-dynamodb-json\n\n.. image:: https://img.shields.io/badge/Release_History!--None.svg?style=social\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/blob/main/release-history.rst\n\n.. image:: https://img.shields.io/badge/STAR_Me_on_GitHub!--None.svg?style=social\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project\n\n------\n\n.. image:: https://img.shields.io/badge/Link-Document-blue.svg\n :target: https://fast-dynamodb-json.readthedocs.io/en/latest/\n\n.. image:: https://img.shields.io/badge/Link-API-blue.svg\n :target: https://fast-dynamodb-json.readthedocs.io/en/latest/py-modindex.html\n\n.. image:: https://img.shields.io/badge/Link-Install-blue.svg\n :target: `install`_\n\n.. image:: https://img.shields.io/badge/Link-GitHub-blue.svg\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project\n\n.. image:: https://img.shields.io/badge/Link-Submit_Issue-blue.svg\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/issues\n\n.. image:: https://img.shields.io/badge/Link-Request_Feature-blue.svg\n :target: https://github.com/MacHu-GWU/fast_dynamodb_json-project/issues\n\n.. image:: https://img.shields.io/badge/Link-Download-blue.svg\n :target: https://pypi.org/pypi/fast-dynamodb-json#files\n\n\nWelcome to ``fast_dynamodb_json`` Documentation\n==============================================================================\n.. image:: https://fast-dynamodb-json.readthedocs.io/en/latest/_static/fast_dynamodb_json-logo.png\n :target: https://fast-dynamodb-json.readthedocs.io/en/latest/\n\nAWS DynamoDB is a powerful NoSQL database service, but it comes with a unique challenge: its data is not stored in standard JSON format. Instead, DynamoDB encodes everything as strings. For example, a string \"hello\" is encoded as ``{\"S\": \"hello\"}``, and a number 123 as ``{\"N\": \"123\"}``. This encoding necessitates deserialization into regular Python dictionaries for data processing, which can be a significant bottleneck in high-volume operations.\n\nExisting Python libraries like `pynamodb` and `dynamodb_json` offer solutions, but their implementations rely on Python for-loops and recursive programming. While functional, these approaches struggle with performance when handling large volumes of DynamoDB items. Moreover, their design doesn't lend itself well to vectorized computing or optimization through JIT technologies like PyPy or Numba.\n\nAs the founder of a startup and a former Amazon Web Services data architect, I've encountered this challenge firsthand. Our company operates an analytics data lake and data pipeline processing billions of DynamoDB items daily for our customers. This scale demanded a more efficient solution.\n\nEnter `fast_dynamodb_json` - a revolutionary approach to deserializing large quantities of DynamoDB items. Our benchmarks demonstrate that this new method is up to 5000 times faster than pure Python implementations, maintaining this impressive speed advantage across both simple and complex datasets.\n\nThis project aims to dramatically improve the efficiency of DynamoDB data processing, enabling businesses to handle massive datasets with unprecedented speed and resource efficiency.\n\n\n.. _install:\n\nInstall\n------------------------------------------------------------------------------\n\n``fast_dynamodb_json`` is released on PyPI, so all you need is to:\n\n.. code-block:: console\n\n $ pip install fast-dynamodb-json\n\nTo upgrade to latest version:\n\n.. code-block:: console\n\n $ pip install --upgrade fast-dynamodb-json\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Blazingly fast DynamoDB Json serialization and deserialization library.",
"version": "0.1.1",
"project_urls": {
"Download": "https://pypi.python.org/pypi/fast_dynamodb_json/0.1.1#downloads",
"Homepage": "https://github.com/MacHu-GWU/fast_dynamodb_json-project"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "40b0c73f0c49dba9dba2e98479ffe9676249e0e60fcb705fd46580a74d1532dc",
"md5": "0103eb54f177cee5e7fc908f95d060f7",
"sha256": "d7a72dff8be490ea86f7ccae29f8d06392795b05188605585ab30005d8913861"
},
"downloads": -1,
"filename": "fast_dynamodb_json-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0103eb54f177cee5e7fc908f95d060f7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 25389,
"upload_time": "2024-08-08T01:25:42",
"upload_time_iso_8601": "2024-08-08T01:25:42.602929Z",
"url": "https://files.pythonhosted.org/packages/40/b0/c73f0c49dba9dba2e98479ffe9676249e0e60fcb705fd46580a74d1532dc/fast_dynamodb_json-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e38da4c624e3e43898daf06e18e76870e6a49adfd1f8151abfb010a50c4c8a27",
"md5": "c62921432c165212addfca171df52674",
"sha256": "0f44a6315ebbd0473e661721288dedc7cfba67eb0709b345b645eb496c1395a3"
},
"downloads": -1,
"filename": "fast_dynamodb_json-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "c62921432c165212addfca171df52674",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 22488,
"upload_time": "2024-08-08T01:25:44",
"upload_time_iso_8601": "2024-08-08T01:25:44.060105Z",
"url": "https://files.pythonhosted.org/packages/e3/8d/a4c624e3e43898daf06e18e76870e6a49adfd1f8151abfb010a50c4c8a27/fast_dynamodb_json-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-08 01:25:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MacHu-GWU",
"github_project": "fast_dynamodb_json-project",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "polars",
"specs": [
[
">=",
"1.2.1"
],
[
"<",
"2.0.0"
]
]
}
],
"lcname": "fast-dynamodb-json"
}
JSON
