dlt


Namedlt JSON
Version 0.4.4 PyPI version JSON
download
home_pagehttps://github.com/dlt-hub
Summarydlt is an open-source python-first scalable data loading library that does not require any backend to run.
upload_time2024-02-11 22:50:18
maintainerMarcin Rudolf
docs_urlNone
authordltHub Inc.
requires_python>=3.8.1,<3.13
licenseApache-2.0
keywords etl
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <h1 align="center">
    <strong>data load tool (dlt) — the open-source Python library for data loading</strong>
</h1>
<p align="center">
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,<br/>or a GPT-4 assisted development playground—<strong>dlt</strong> can be dropped in anywhere.
</p>


<h3 align="center">

🚀 Join our thriving community of likeminded developers and build the future together!

</h3>

<div align="center">
  <a target="_blank" href="https://dlthub.com/community" style="background:none">
    <img src="https://img.shields.io/badge/slack-join-dlt.svg?labelColor=191937&color=6F6FF7&logo=slack" style="width: 260px;"  />
  </a>
</div>
<div align="center">
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/v/dlt?labelColor=191937&color=6F6FF7">
  </a>
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/pyversions/dlt?labelColor=191937&color=6F6FF7">
  </a>
</div>

## Installation

dlt supports Python 3.8+.

```bash
pip install dlt
```

## Quick Start

Load chess game data from chess.com API and save it in DuckDB:

```python
import dlt
from dlt.sources.helpers import requests
# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
    pipeline_name='chess_pipeline',
    destination='duckdb',
    dataset_name='player_data'
)
# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
    response = requests.get(f'https://api.chess.com/pub/player/{player}')
    response.raise_for_status()
    data.append(response.json())
# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
```


Try it out in our **[Colab Demo](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)**

## Features

- **Automatic Schema:** Data structure inspection and schema creation for the destination.
- **Data Normalization:** Consistent and verified data before loading.
- **Seamless Integration:** Colab, AWS Lambda, Airflow, and local environments.
- **Scalable:** Adapts to growing data needs in production.
- **Easy Maintenance:** Clear data pipeline structure for updates.
- **Rapid Exploration:** Quickly explore and gain insights from new data sources.
- **Versatile Usage:** Suitable for ad-hoc exploration to advanced loading infrastructures.
- **Start in Seconds with CLI:** Powerful CLI for managing, deploying and inspecting local pipelines.
- **Incremental Loading:** Load only new or changed data and avoid loading old records again.
- **Open Source:** Free and Apache 2.0 Licensed.

## Ready to use Sources and Destinations

Explore ready to use sources (e.g. Google Sheets) in the [Verified Sources docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and supported destinations (e.g. DuckDB) in the [Destinations docs](https://dlthub.com/docs/dlt-ecosystem/destinations).

## Documentation

For detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).

## Examples

You can find examples for various use cases in the [examples](docs/examples) folder.

## Adding as dependency

`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern. Currently, we are using **pre-release versioning** with the major version being 0.

- `minor` version change means breaking changes
- `patch` version change means new features that should be backward compatible
- any suffix change, e.g., `post10` -> `post11`, is considered a patch

We suggest that you allow only `patch` level updates automatically:
* Using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=0.3.10** allows only versions **>=0.3.10** and less than **<0.4**
* Poetry [caret requirements](https://python-poetry.org/docs/dependency-specification/). For example **^0.3.10** allows only versions **>=0.3.10** to **<0.4**
## Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)
- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)
- **Contribute Verified Sources**: Contribute your custom sources to the [dlt-hub/verified-sources](https://github.com/dlt-hub/verified-sources) to help other folks in handling their data tasks.
- **Contribute code**: Check out our [contributing guidelines](CONTRIBUTING.md) for information on how to make a pull request.
- **Improve documentation**: Help us enhance the dlt documentation.

## License

DLT is released under the [Apache 2.0 License](LICENSE.txt).

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/dlt-hub",
    "name": "dlt",
    "maintainer": "Marcin Rudolf",
    "docs_url": null,
    "requires_python": ">=3.8.1,<3.13",
    "maintainer_email": "marcin@dlthub.com",
    "keywords": "etl",
    "author": "dltHub Inc.",
    "author_email": "services@dlthub.com",
    "download_url": "https://files.pythonhosted.org/packages/15/7c/936a70fd82c7f21a50877cb299746d8f393f5f5c7cf80e10cb77ee8413ae/dlt-0.4.4.tar.gz",
    "platform": null,
    "description": "<h1 align=\"center\">\n    <strong>data load tool (dlt) \u2014 the open-source Python library for data loading</strong>\n</h1>\n<p align=\"center\">\nBe it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,<br/>or a GPT-4 assisted development playground\u2014<strong>dlt</strong> can be dropped in anywhere.\n</p>\n\n\n<h3 align=\"center\">\n\n\ud83d\ude80 Join our thriving community of likeminded developers and build the future together!\n\n</h3>\n\n<div align=\"center\">\n  <a target=\"_blank\" href=\"https://dlthub.com/community\" style=\"background:none\">\n    <img src=\"https://img.shields.io/badge/slack-join-dlt.svg?labelColor=191937&color=6F6FF7&logo=slack\" style=\"width: 260px;\"  />\n  </a>\n</div>\n<div align=\"center\">\n  <a target=\"_blank\" href=\"https://pypi.org/project/dlt/\" style=\"background:none\">\n    <img src=\"https://img.shields.io/pypi/v/dlt?labelColor=191937&color=6F6FF7\">\n  </a>\n  <a target=\"_blank\" href=\"https://pypi.org/project/dlt/\" style=\"background:none\">\n    <img src=\"https://img.shields.io/pypi/pyversions/dlt?labelColor=191937&color=6F6FF7\">\n  </a>\n</div>\n\n## Installation\n\ndlt supports Python 3.8+.\n\n```bash\npip install dlt\n```\n\n## Quick Start\n\nLoad chess game data from chess.com API and save it in DuckDB:\n\n```python\nimport dlt\nfrom dlt.sources.helpers import requests\n# Create a dlt pipeline that will load\n# chess player data to the DuckDB destination\npipeline = dlt.pipeline(\n    pipeline_name='chess_pipeline',\n    destination='duckdb',\n    dataset_name='player_data'\n)\n# Grab some player data from Chess.com API\ndata = []\nfor player in ['magnuscarlsen', 'rpragchess']:\n    response = requests.get(f'https://api.chess.com/pub/player/{player}')\n    response.raise_for_status()\n    data.append(response.json())\n# Extract, normalize, and load the data\npipeline.run(data, table_name='player')\n```\n\n\nTry it out in our **[Colab Demo](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)**\n\n## Features\n\n- **Automatic Schema:** Data structure inspection and schema creation for the destination.\n- **Data Normalization:** Consistent and verified data before loading.\n- **Seamless Integration:** Colab, AWS Lambda, Airflow, and local environments.\n- **Scalable:** Adapts to growing data needs in production.\n- **Easy Maintenance:** Clear data pipeline structure for updates.\n- **Rapid Exploration:** Quickly explore and gain insights from new data sources.\n- **Versatile Usage:** Suitable for ad-hoc exploration to advanced loading infrastructures.\n- **Start in Seconds with CLI:** Powerful CLI for managing, deploying and inspecting local pipelines.\n- **Incremental Loading:** Load only new or changed data and avoid loading old records again.\n- **Open Source:** Free and Apache 2.0 Licensed.\n\n## Ready to use Sources and Destinations\n\nExplore ready to use sources (e.g. Google Sheets) in the [Verified Sources docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and supported destinations (e.g. DuckDB) in the [Destinations docs](https://dlthub.com/docs/dlt-ecosystem/destinations).\n\n## Documentation\n\nFor detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).\n\n## Examples\n\nYou can find examples for various use cases in the [examples](docs/examples) folder.\n\n## Adding as dependency\n\n`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern. Currently, we are using **pre-release versioning** with the major version being 0.\n\n- `minor` version change means breaking changes\n- `patch` version change means new features that should be backward compatible\n- any suffix change, e.g., `post10` -> `post11`, is considered a patch\n\nWe suggest that you allow only `patch` level updates automatically:\n* Using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=0.3.10** allows only versions **>=0.3.10** and less than **<0.4**\n* Poetry [caret requirements](https://python-poetry.org/docs/dependency-specification/). For example **^0.3.10** allows only versions **>=0.3.10** to **<0.4**\n## Get Involved\n\nThe dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:\n\n- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)\n- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.\n- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)\n- **Contribute Verified Sources**: Contribute your custom sources to the [dlt-hub/verified-sources](https://github.com/dlt-hub/verified-sources) to help other folks in handling their data tasks.\n- **Contribute code**: Check out our [contributing guidelines](CONTRIBUTING.md) for information on how to make a pull request.\n- **Improve documentation**: Help us enhance the dlt documentation.\n\n## License\n\nDLT is released under the [Apache 2.0 License](LICENSE.txt).\n",
    "bugtrack_url": null,
    "license": "Apache-2.0",
    "summary": "dlt is an open-source python-first scalable data loading library that does not require any backend to run.",
    "version": "0.4.4",
    "project_urls": {
        "Homepage": "https://github.com/dlt-hub",
        "Repository": "https://github.com/dlt-hub/dlt"
    },
    "split_keywords": [
        "etl"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5b0a3bcfec9a96d8f77b0cd8e3ce34ab452a92bcddfb5398b008e2ea2ae2a5a0",
                "md5": "52b1e8c3ed98e61285def7d08c1135a0",
                "sha256": "dfa1d0fd1ba5e2741f0d58314ca56aad26ec25032039bc3fa5d873d4611d8568"
            },
            "downloads": -1,
            "filename": "dlt-0.4.4-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "52b1e8c3ed98e61285def7d08c1135a0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8.1,<3.13",
            "size": 540725,
            "upload_time": "2024-02-11T22:50:15",
            "upload_time_iso_8601": "2024-02-11T22:50:15.746944Z",
            "url": "https://files.pythonhosted.org/packages/5b/0a/3bcfec9a96d8f77b0cd8e3ce34ab452a92bcddfb5398b008e2ea2ae2a5a0/dlt-0.4.4-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "157c936a70fd82c7f21a50877cb299746d8f393f5f5c7cf80e10cb77ee8413ae",
                "md5": "03507847bc6e7d7bd5fba7e7f32b7d4f",
                "sha256": "9a9619f78fe06cc157a23179b4fb17a059606e8c980756ea0652b167b91356fa"
            },
            "downloads": -1,
            "filename": "dlt-0.4.4.tar.gz",
            "has_sig": false,
            "md5_digest": "03507847bc6e7d7bd5fba7e7f32b7d4f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8.1,<3.13",
            "size": 416659,
            "upload_time": "2024-02-11T22:50:18",
            "upload_time_iso_8601": "2024-02-11T22:50:18.663474Z",
            "url": "https://files.pythonhosted.org/packages/15/7c/936a70fd82c7f21a50877cb299746d8f393f5f5c7cf80e10cb77ee8413ae/dlt-0.4.4.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-02-11 22:50:18",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "dlt-hub",
    "github_project": "dlt",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "tox": true,
    "lcname": "dlt"
}
        
Elapsed time: 0.23354s