sling


Namesling JSON
Version 1.1.6 PyPI version JSON
download
home_pagehttps://github.com/slingdata-io/sling-python
SummarySlings data from a source to a target
upload_time2024-02-19 23:28:45
maintainer
docs_urlNone
authorFritz Larco
requires_python
license
keywords sling etl elt extract load
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <p align="center"><img src="https://github.com/slingdata-io/sling-python/raw/main/logo-with-text.png" alt="logo" width="250"/></p>

<p align="center">Slings from a data source to a data target.</p>

## Installation

`pip install sling`

Then you should be able to run `sling --help` from command line.

## Running a Extract-Load Task

### CLI

```shell
sling run --src-conn MY_PG --src-stream myschema.mytable \
  --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
  --mode full-refresh
```

Or passing a yaml/json string or file

```shell
sling run -c '
source:
  conn: MY_PG
  stream: myschema.mytable

target:
  conn: YOUR_SNOWFLAKE
  object: yourschema.yourtable

mode: full-refresh
'
# OR
sling run -c /path/to/config.json
```

### From Lib

```python
from sling import Sling

config = {
  'source': {
    'conn': 'MY_PG',
    'stream': "select * from my_table",
  },
  'target': {
    'conn':  "s3://my_bucket/my_folder/new_file.csv",
  },
}

Sling(**config).run()
```

## Config Schema

`--src-conn`/`source.conn` and `--tgt-conn`/`target.conn`  can be a name or URL of a folder:
- `MY_PG` (connection ref in db, profile or env)
- `postgresql://user:password!@host.loc:5432/database`
- `s3://my_bucket/my_folder/file.csv`
- `gs://my_google_bucket/my_folder/file.json`
- `file:///tmp/my_folder/file.csv` (local storage)

`--src-stream`/`source.stream` can be an object name to stream from:
- `TABLE1`
- `SCHEMA1.TABLE2`
- `OBJECT_NAME`
- `select * from SCHEMA1.TABLE3`
- `/path/to/file.sql` (if source conn is DB)

`--tgt-object`/`target.object` can be an object name to write to:
- `TABLE1`
- `SCHEMA1.TABLE2`

### Example as JSON

```json
{
  "source": {
    "conn": "MY_PG_URL",
    "stream": "select * from my_table",
    "options": {}
  },
  "target": {
    "conn": "s3://my_bucket/my_folder/new_file.csv",
    "options": {
      "header": false
    }
  }
}
```



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/slingdata-io/sling-python",
    "name": "sling",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "sling,etl,elt,extract,load",
    "author": "Fritz Larco",
    "author_email": "fritz@slingdata.io",
    "download_url": "https://files.pythonhosted.org/packages/02/78/2850a081568c56a6acdf608f70931490ff06c05ec6bb95fe6e2e80053328/sling-1.1.6.tar.gz",
    "platform": null,
    "description": "<p align=\"center\"><img src=\"https://github.com/slingdata-io/sling-python/raw/main/logo-with-text.png\" alt=\"logo\" width=\"250\"/></p>\n\n<p align=\"center\">Slings from a data source to a data target.</p>\n\n## Installation\n\n`pip install sling`\n\nThen you should be able to run `sling --help` from command line.\n\n## Running a Extract-Load Task\n\n### CLI\n\n```shell\nsling run --src-conn MY_PG --src-stream myschema.mytable \\\n  --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \\\n  --mode full-refresh\n```\n\nOr passing a yaml/json string or file\n\n```shell\nsling run -c '\nsource:\n  conn: MY_PG\n  stream: myschema.mytable\n\ntarget:\n  conn: YOUR_SNOWFLAKE\n  object: yourschema.yourtable\n\nmode: full-refresh\n'\n# OR\nsling run -c /path/to/config.json\n```\n\n### From Lib\n\n```python\nfrom sling import Sling\n\nconfig = {\n  'source': {\n    'conn': 'MY_PG',\n    'stream': \"select * from my_table\",\n  },\n  'target': {\n    'conn':  \"s3://my_bucket/my_folder/new_file.csv\",\n  },\n}\n\nSling(**config).run()\n```\n\n## Config Schema\n\n`--src-conn`/`source.conn` and `--tgt-conn`/`target.conn`  can be a name or URL of a folder:\n- `MY_PG` (connection ref in db, profile or env)\n- `postgresql://user:password!@host.loc:5432/database`\n- `s3://my_bucket/my_folder/file.csv`\n- `gs://my_google_bucket/my_folder/file.json`\n- `file:///tmp/my_folder/file.csv` (local storage)\n\n`--src-stream`/`source.stream` can be an object name to stream from:\n- `TABLE1`\n- `SCHEMA1.TABLE2`\n- `OBJECT_NAME`\n- `select * from SCHEMA1.TABLE3`\n- `/path/to/file.sql` (if source conn is DB)\n\n`--tgt-object`/`target.object` can be an object name to write to:\n- `TABLE1`\n- `SCHEMA1.TABLE2`\n\n### Example as JSON\n\n```json\n{\n  \"source\": {\n    \"conn\": \"MY_PG_URL\",\n    \"stream\": \"select * from my_table\",\n    \"options\": {}\n  },\n  \"target\": {\n    \"conn\": \"s3://my_bucket/my_folder/new_file.csv\",\n    \"options\": {\n      \"header\": false\n    }\n  }\n}\n```\n\n\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "Slings data from a source to a target",
    "version": "1.1.6",
    "project_urls": {
        "Download": "https://github.com/slingdata-io/sling-python/archive/master.zip",
        "Homepage": "https://github.com/slingdata-io/sling-python"
    },
    "split_keywords": [
        "sling",
        "etl",
        "elt",
        "extract",
        "load"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "02782850a081568c56a6acdf608f70931490ff06c05ec6bb95fe6e2e80053328",
                "md5": "7f7a7c50928b0523450a4977f8ca8f1d",
                "sha256": "c89fca4ad47ff67045f9f004b408d224e27f556c33228067748a28cb0514bb1b"
            },
            "downloads": -1,
            "filename": "sling-1.1.6.tar.gz",
            "has_sig": false,
            "md5_digest": "7f7a7c50928b0523450a4977f8ca8f1d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6643,
            "upload_time": "2024-02-19T23:28:45",
            "upload_time_iso_8601": "2024-02-19T23:28:45.499454Z",
            "url": "https://files.pythonhosted.org/packages/02/78/2850a081568c56a6acdf608f70931490ff06c05ec6bb95fe6e2e80053328/sling-1.1.6.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-02-19 23:28:45",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "slingdata-io",
    "github_project": "sling-python",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "sling"
}
        
Elapsed time: 0.26590s