<p align="center"><img src="https://github.com/slingdata-io/sling-python/raw/main/logo-with-text.png" alt="logo" width="250"/></p>
<p align="center">Slings from a data source to a data target.</p>
## Installation
`pip install sling`
Then you should be able to run `sling --help` from command line.
## Running a Extract-Load Task
### CLI
```shell
sling run --src-conn MY_PG --src-stream myschema.mytable \
--tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
--mode full-refresh
```
Or passing a yaml/json string or file
```shell
sling run -c '
source:
conn: MY_PG
stream: myschema.mytable
target:
conn: YOUR_SNOWFLAKE
object: yourschema.yourtable
mode: full-refresh
'
# OR
sling run -c /path/to/config.json
```
### From Lib
```python
from sling import Sling
config = {
'source': {
'conn': 'MY_PG',
'stream': "select * from my_table",
},
'target': {
'conn': "s3://my_bucket/my_folder/new_file.csv",
},
}
Sling(**config).run()
```
## Config Schema
`--src-conn`/`source.conn` and `--tgt-conn`/`target.conn` can be a name or URL of a folder:
- `MY_PG` (connection ref in db, profile or env)
- `postgresql://user:password!@host.loc:5432/database`
- `s3://my_bucket/my_folder/file.csv`
- `gs://my_google_bucket/my_folder/file.json`
- `file:///tmp/my_folder/file.csv` (local storage)
`--src-stream`/`source.stream` can be an object name to stream from:
- `TABLE1`
- `SCHEMA1.TABLE2`
- `OBJECT_NAME`
- `select * from SCHEMA1.TABLE3`
- `/path/to/file.sql` (if source conn is DB)
`--tgt-object`/`target.object` can be an object name to write to:
- `TABLE1`
- `SCHEMA1.TABLE2`
### Example as JSON
```json
{
"source": {
"conn": "MY_PG_URL",
"stream": "select * from my_table",
"options": {}
},
"target": {
"conn": "s3://my_bucket/my_folder/new_file.csv",
"options": {
"header": false
}
}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/slingdata-io/sling-python",
"name": "sling",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "sling, etl, elt, extract, load",
"author": "Fritz Larco",
"author_email": "fritz@slingdata.io",
"download_url": "https://files.pythonhosted.org/packages/ae/50/85e00fc5b753bff7bb982dbf47bf77fc4acefa6ae0d7d4482d3e21ad893c/sling-1.2.6.tar.gz",
"platform": null,
"description": "<p align=\"center\"><img src=\"https://github.com/slingdata-io/sling-python/raw/main/logo-with-text.png\" alt=\"logo\" width=\"250\"/></p>\n\n<p align=\"center\">Slings from a data source to a data target.</p>\n\n## Installation\n\n`pip install sling`\n\nThen you should be able to run `sling --help` from command line.\n\n## Running a Extract-Load Task\n\n### CLI\n\n```shell\nsling run --src-conn MY_PG --src-stream myschema.mytable \\\n --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \\\n --mode full-refresh\n```\n\nOr passing a yaml/json string or file\n\n```shell\nsling run -c '\nsource:\n conn: MY_PG\n stream: myschema.mytable\n\ntarget:\n conn: YOUR_SNOWFLAKE\n object: yourschema.yourtable\n\nmode: full-refresh\n'\n# OR\nsling run -c /path/to/config.json\n```\n\n### From Lib\n\n```python\nfrom sling import Sling\n\nconfig = {\n 'source': {\n 'conn': 'MY_PG',\n 'stream': \"select * from my_table\",\n },\n 'target': {\n 'conn': \"s3://my_bucket/my_folder/new_file.csv\",\n },\n}\n\nSling(**config).run()\n```\n\n## Config Schema\n\n`--src-conn`/`source.conn` and `--tgt-conn`/`target.conn` can be a name or URL of a folder:\n- `MY_PG` (connection ref in db, profile or env)\n- `postgresql://user:password!@host.loc:5432/database`\n- `s3://my_bucket/my_folder/file.csv`\n- `gs://my_google_bucket/my_folder/file.json`\n- `file:///tmp/my_folder/file.csv` (local storage)\n\n`--src-stream`/`source.stream` can be an object name to stream from:\n- `TABLE1`\n- `SCHEMA1.TABLE2`\n- `OBJECT_NAME`\n- `select * from SCHEMA1.TABLE3`\n- `/path/to/file.sql` (if source conn is DB)\n\n`--tgt-object`/`target.object` can be an object name to write to:\n- `TABLE1`\n- `SCHEMA1.TABLE2`\n\n### Example as JSON\n\n```json\n{\n \"source\": {\n \"conn\": \"MY_PG_URL\",\n \"stream\": \"select * from my_table\",\n \"options\": {}\n },\n \"target\": {\n \"conn\": \"s3://my_bucket/my_folder/new_file.csv\",\n \"options\": {\n \"header\": false\n }\n }\n}\n```\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Slings data from a source to a target",
"version": "1.2.6",
"project_urls": {
"Download": "https://github.com/slingdata-io/sling-python/archive/master.zip",
"Homepage": "https://github.com/slingdata-io/sling-python"
},
"split_keywords": [
"sling",
" etl",
" elt",
" extract",
" load"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ae5085e00fc5b753bff7bb982dbf47bf77fc4acefa6ae0d7d4482d3e21ad893c",
"md5": "6a95fc058742da0a4b5b9ee81ed3d252",
"sha256": "af73e592d0e37d31327604a92c5648b74c964c0d22beb8030cc5859d859e6745"
},
"downloads": -1,
"filename": "sling-1.2.6.tar.gz",
"has_sig": false,
"md5_digest": "6a95fc058742da0a4b5b9ee81ed3d252",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6677,
"upload_time": "2024-04-22T11:36:12",
"upload_time_iso_8601": "2024-04-22T11:36:12.246024Z",
"url": "https://files.pythonhosted.org/packages/ae/50/85e00fc5b753bff7bb982dbf47bf77fc4acefa6ae0d7d4482d3e21ad893c/sling-1.2.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-22 11:36:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "slingdata-io",
"github_project": "sling-python",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "sling"
}
JSON
