| Name | duckingit JSON |
| Version |
0.0.11
JSON |
| download |
| home_page | |
| Summary | A framework to leverage clusters of serverless functions for analytics. Powered by DuckDB |
| upload_time | 2023-05-23 19:26:49 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | |
| keywords |
serverless
duckdb
data engineering
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
A framework to leverage the endless capabilities of serverless computing powered by DuckDB.
Please note that the framework currently supports only AWS Lambda functions. To use the framework, you must first create a Lambda layer of DuckDB that can be used within a Lambda function. Additionally, you must create a Lambda Executor function that can execute the actual DuckDB SQL. Once you've completed these setup steps, you can leverage the power of serverless functions through the SDK written in Python to perform analytics on a Data Lake.
While Apache Spark can perform similar (and more advanced) functions, the cost of running Spark clusters can be prohibitively expensive. As a result, a much more affordable alternative is to use a cluster of serverless functions, such as Lambda functions, to perform the same actions as Spark, without the need to turn them off manually.
## Installation
To install the Python SDK from PyPI execute the command below. Nonetheless, it's recommended that you first review the [setup](/README.md#setup) section in order to properly utilize the package.
```bash
pip install duckingit
```
## Setup
Before setting up the infrastructure, please make sure that you have installed both Docker and Terraform.
To interact with the DuckDB instances, the entire infrastructure must be set up first because the SDK functions as an entryway to the serverless function cluster. DuckDB is packaged as a layer to be pre-installed in AWS Lambda, similar to other packages. Docker must be installed to create the layer.
Running the command below will generate a duckdb-layer.zip file in the image/release/ folder:
```bash
make release-image
```
To set up the infrastructure on AWS, follow the commands below:
```bash
make release-infra
```
After waiting for a minute or two, the infrastructure should be set up, and you can check for the presence of a Lambda function called DuckExecutor and a lambda layer called duckdb under Lambda layers.
Once you have verified the above components, the infrastructure should be set up and fully operational.
## Usage
The developer API is inspired by the API of Spark, but it uses Python's naming conventions because the framework is implemented in Python.
```python
from duckingit import DuckSession, DuckConfig
query = "SELECT * FROM READ_PARQUET(['s3://BUCKET_NAME/2023/*'])"
# Following command will print possible configurations
DuckConfig.show_configurations()
# Configuration
conf = DuckConfig() \
.set("aws_lambda.FunctionName", "TestFunc") \
.set("aws_lambda.MemorySize", 256) \
.set("aws_lambda.WarmUp", True)
# Creates an entrypoint to use serverless DuckDB instances
session = DuckSession(conf=conf)
# Create a Dataset from the query
ds = session.sql(query=query)
# Execute SQL query
ds.show()
```
... To be continued
## Contribution
Thank you for taking an interest in my project on GitHub. I am always looking for new contributors to help me improve and evolve my codebase. If you're interested in contributing, feel free to fork the repository and submit a pull request with your changes.
I welcome all kinds of contributions, from bug fixes to feature additions and documentation improvements. If you're not sure where to start, take a look at the issues tab or reach out to us for guidance.
Let's collaborate and make our project better together!
Raw data
{
"_id": null,
"home_page": "",
"name": "duckingit",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "Serverless,DuckDB,Data Engineering",
"author": "",
"author_email": "Tobias Egelund <egelundtobias@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/78/86/4e3336341aa133033154ecfb95bb20d9854c790f15ab9a4d4046c37af022/duckingit-0.0.11.tar.gz",
"platform": null,
"description": "\n\nA framework to leverage the endless capabilities of serverless computing powered by DuckDB.\n\nPlease note that the framework currently supports only AWS Lambda functions. To use the framework, you must first create a Lambda layer of DuckDB that can be used within a Lambda function. Additionally, you must create a Lambda Executor function that can execute the actual DuckDB SQL. Once you've completed these setup steps, you can leverage the power of serverless functions through the SDK written in Python to perform analytics on a Data Lake.\n\nWhile Apache Spark can perform similar (and more advanced) functions, the cost of running Spark clusters can be prohibitively expensive. As a result, a much more affordable alternative is to use a cluster of serverless functions, such as Lambda functions, to perform the same actions as Spark, without the need to turn them off manually.\n\n## Installation\nTo install the Python SDK from PyPI execute the command below. Nonetheless, it's recommended that you first review the [setup](/README.md#setup) section in order to properly utilize the package.\n\n```bash\npip install duckingit\n```\n\n## Setup\nBefore setting up the infrastructure, please make sure that you have installed both Docker and Terraform.\n\nTo interact with the DuckDB instances, the entire infrastructure must be set up first because the SDK functions as an entryway to the serverless function cluster. DuckDB is packaged as a layer to be pre-installed in AWS Lambda, similar to other packages. Docker must be installed to create the layer.\n\nRunning the command below will generate a duckdb-layer.zip file in the image/release/ folder:\n```bash\nmake release-image\n```\n\nTo set up the infrastructure on AWS, follow the commands below:\n```bash\nmake release-infra\n```\n\nAfter waiting for a minute or two, the infrastructure should be set up, and you can check for the presence of a Lambda function called DuckExecutor and a lambda layer called duckdb under Lambda layers.\n\nOnce you have verified the above components, the infrastructure should be set up and fully operational.\n\n## Usage\nThe developer API is inspired by the API of Spark, but it uses Python's naming conventions because the framework is implemented in Python.\n\n```python\nfrom duckingit import DuckSession, DuckConfig\n\nquery = \"SELECT * FROM READ_PARQUET(['s3://BUCKET_NAME/2023/*'])\"\n\n# Following command will print possible configurations\nDuckConfig.show_configurations()\n\n# Configuration\nconf = DuckConfig() \\\n .set(\"aws_lambda.FunctionName\", \"TestFunc\") \\\n .set(\"aws_lambda.MemorySize\", 256) \\\n .set(\"aws_lambda.WarmUp\", True)\n\n# Creates an entrypoint to use serverless DuckDB instances\nsession = DuckSession(conf=conf)\n\n# Create a Dataset from the query\nds = session.sql(query=query)\n\n# Execute SQL query\nds.show()\n```\n\n... To be continued\n\n## Contribution\nThank you for taking an interest in my project on GitHub. I am always looking for new contributors to help me improve and evolve my codebase. If you're interested in contributing, feel free to fork the repository and submit a pull request with your changes.\n\nI welcome all kinds of contributions, from bug fixes to feature additions and documentation improvements. If you're not sure where to start, take a look at the issues tab or reach out to us for guidance.\n\nLet's collaborate and make our project better together!\n",
"bugtrack_url": null,
"license": "",
"summary": "A framework to leverage clusters of serverless functions for analytics. Powered by DuckDB",
"version": "0.0.11",
"project_urls": {
"Bug Tracker": "https://github.com/tobiasegelund/duckingit/issues",
"Homepage": "https://github.com/tobiasegelund/duckingit"
},
"split_keywords": [
"serverless",
"duckdb",
"data engineering"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "58141e18f204e09925372b10a2c0a9881235bdd0b63260c0d4bfba714ef3a6f5",
"md5": "dfdb6d8d25e40f4c039c3945cbaf3fc2",
"sha256": "05aca24a445886698851ea56d24cde233835654ec33c79ea37f1e726c704bfc5"
},
"downloads": -1,
"filename": "duckingit-0.0.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dfdb6d8d25e40f4c039c3945cbaf3fc2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 23804,
"upload_time": "2023-05-23T19:26:47",
"upload_time_iso_8601": "2023-05-23T19:26:47.489676Z",
"url": "https://files.pythonhosted.org/packages/58/14/1e18f204e09925372b10a2c0a9881235bdd0b63260c0d4bfba714ef3a6f5/duckingit-0.0.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "78864e3336341aa133033154ecfb95bb20d9854c790f15ab9a4d4046c37af022",
"md5": "fb82e023e9f635340e0f4f948ef0d857",
"sha256": "d32c69518f91a8d3f5052c94b3c2b0bb683be3f94b8746f698a6c7017b625d6e"
},
"downloads": -1,
"filename": "duckingit-0.0.11.tar.gz",
"has_sig": false,
"md5_digest": "fb82e023e9f635340e0f4f948ef0d857",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 21252,
"upload_time": "2023-05-23T19:26:49",
"upload_time_iso_8601": "2023-05-23T19:26:49.075270Z",
"url": "https://files.pythonhosted.org/packages/78/86/4e3336341aa133033154ecfb95bb20d9854c790f15ab9a4d4046c37af022/duckingit-0.0.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-23 19:26:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "tobiasegelund",
"github_project": "duckingit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "duckingit"
}
