pyred


Namepyred JSON
Version 0.1.2 PyPI version JSON
download
home_pagehttps://github.com/dacker-team/pyred
SummaryEasily send data to Amazon Redshift
upload_time2018-04-16 13:57:20
maintainer
docs_urlNone
authorDacker
requires_python
licenseCopyright (c) 2018, Dacker All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
keywords send data amazon redshift easy
VCS
bugtrack_url
requirements psycopg2-binary
Travis-CI No Travis.
coveralls test coverage No coveralls.
            pyred
=====

A python package to easily send data to Amazon Redshift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1) Installation
'''''''''''''''

Open a terminal and install pyred package
                                                   
::

    pip install pyred


2) Use
''''''

1) Be sure that you have set environment variables with Redshift credentials like this:


::

    export RED_{INSTANCE}_DATABASE=""
    export RED_{INSTANCE}_USERNAME=""
    export RED_{INSTANCE}_HOST=""
    export RED_{INSTANCE}_PORT=""
    export RED_{INSTANCE}_PASSWORD=""

2) Be also sure that your IP address is authorized for the redshift cluster/instance.

3) Prepare your data like that:


.. code:: python

    data = {
            "table_name"    : 'name_of_the_redshift_schema' + '.' + 'name_of_the_redshift_table'
            "columns_name"  : [first_column_name,second_column_name,...,last_column_name],
            "rows"      : [[first_raw_value,second_raw_value,...,last_raw_value],...]
        }

4) Send your data (use the same {INSTANCE} parameter as environment variables):


.. code:: python

    import pyred
    pyred.send_to_redshift(instance, data, replace=True, batch_size=1000, types=None, primary_key=(), create_boolean=False)

-  replace (default=True) argument allows you to replace or append data
   in the table
-  batch\_size (default=1000) argument also exists to send data in
   batchs
- types, primary_key and create_boolean are explained below

3) First Example
''''''''''''''''

You have a table called dog in you animal scheme. This table has two columns : 'name' and 'size'.
You want to add two dogs (= two rows) : Pif which is big and Milou which is small.
*data* will be like that:

.. code:: python

    import pyred
    data = {
            "table_name"    : 'animal.dog'
            "columns_name"  : ['name','size'],
            "rows"      : [['Pif','big'], ['Milou','small']]
        }
    pyred.send_to_redshift({INSTANCE},data)

4) Function *create_table*
''''''''''''''''''''''''''
pyred has a *create_table* function with this signature:

.. code:: python

    import pyred
    pyred.create_table({INSTANCE}, data, primary_key=(), types=None)

This function is automatically called in the *send_to_redshift* function if the table is not created. You can also call it with the "create_boolean" parameter of the *send_to_reshift* function or by setting "primary_key" or "types" parameters.

-  primary_key : if you have 3 columns (ie: columns_name=[a,b,c]) and you want to set b as primary key, set primary_key=(b)
-  types: *create_table* function guesses types of each column. But you can set a "types" argument. It is a dictionary like {'b': 'VARCHAR(12)'} or  {'b': 'INTEGER NOT NULL'} to set types of b column.

            

Raw data

            {
    "maintainer": "", 
    "docs_url": null, 
    "requires_python": "", 
    "maintainer_email": "", 
    "cheesecake_code_kwalitee_id": null, 
    "keywords": "send data amazon redshift easy", 
    "upload_time": "2018-04-16 13:57:20", 
    "requirements": [
        {
            "name": "psycopg2-binary", 
            "specs": [
                [
                    "==", 
                    "2.7.4"
                ]
            ]
        }
    ], 
    "author": "Dacker", 
    "home_page": "https://github.com/dacker-team/pyred", 
    "github_user": "dacker-team", 
    "download_url": "https://pypi.python.org/packages/02/52/7d5d40c73d8ac8ad25fbe05c71a3f13938e362b4065f5676d4fc69bc5b0a/pyred-0.1.2.tar.gz", 
    "platform": "", 
    "version": "0.1.2", 
    "cheesecake_documentation_id": null, 
    "description": "pyred\n=====\n\nA python package to easily send data to Amazon Redshift\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1) Installation\n'''''''''''''''\n\nOpen a terminal and install pyred package\n                                                   \n::\n\n    pip install pyred\n\n\n2) Use\n''''''\n\n1) Be sure that you have set environment variables with Redshift credentials like this:\n\n\n::\n\n    export RED_{INSTANCE}_DATABASE=\"\"\n    export RED_{INSTANCE}_USERNAME=\"\"\n    export RED_{INSTANCE}_HOST=\"\"\n    export RED_{INSTANCE}_PORT=\"\"\n    export RED_{INSTANCE}_PASSWORD=\"\"\n\n2) Be also sure that your IP address is authorized for the redshift cluster/instance.\n\n3) Prepare your data like that:\n\n\n.. code:: python\n\n    data = {\n            \"table_name\"    : 'name_of_the_redshift_schema' + '.' + 'name_of_the_redshift_table'\n            \"columns_name\"  : [first_column_name,second_column_name,...,last_column_name],\n            \"rows\"      : [[first_raw_value,second_raw_value,...,last_raw_value],...]\n        }\n\n4) Send your data (use the same {INSTANCE} parameter as environment variables):\n\n\n.. code:: python\n\n    import pyred\n    pyred.send_to_redshift(instance, data, replace=True, batch_size=1000, types=None, primary_key=(), create_boolean=False)\n\n-  replace (default=True) argument allows you to replace or append data\n   in the table\n-  batch\\_size (default=1000) argument also exists to send data in\n   batchs\n- types, primary_key and create_boolean are explained below\n\n3) First Example\n''''''''''''''''\n\nYou have a table called dog in you animal scheme. This table has two columns : 'name' and 'size'.\nYou want to add two dogs (= two rows) : Pif which is big and Milou which is small.\n*data* will be like that:\n\n.. code:: python\n\n    import pyred\n    data = {\n            \"table_name\"    : 'animal.dog'\n            \"columns_name\"  : ['name','size'],\n            \"rows\"      : [['Pif','big'], ['Milou','small']]\n        }\n    pyred.send_to_redshift({INSTANCE},data)\n\n4) Function *create_table*\n''''''''''''''''''''''''''\npyred has a *create_table* function with this signature:\n\n.. code:: python\n\n    import pyred\n    pyred.create_table({INSTANCE}, data, primary_key=(), types=None)\n\nThis function is automatically called in the *send_to_redshift* function if the table is not created. You can also call it with the \"create_boolean\" parameter of the *send_to_reshift* function or by setting \"primary_key\" or \"types\" parameters.\n\n-  primary_key : if you have 3 columns (ie: columns_name=[a,b,c]) and you want to set b as primary key, set primary_key=(b)\n-  types: *create_table* function guesses types of each column. But you can set a \"types\" argument. It is a dictionary like {'b': 'VARCHAR(12)'} or  {'b': 'INTEGER NOT NULL'} to set types of b column.\n", 
    "lcname": "pyred", 
    "bugtrack_url": null, 
    "github": true, 
    "coveralls": false, 
    "name": "pyred", 
    "license": "Copyright (c) 2018, Dacker\n\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\nRedistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\nRedistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n", 
    "travis_ci": false, 
    "github_project": "pyred", 
    "summary": "Easily send data to Amazon Redshift", 
    "split_keywords": [
        "send", 
        "data", 
        "amazon", 
        "redshift", 
        "easy"
    ], 
    "author_email": "hello@dacker.co", 
    "urls": [
        {
            "has_sig": false, 
            "upload_time": "2018-04-16T13:57:20", 
            "comment_text": "", 
            "python_version": "3.6", 
            "url": "https://pypi.python.org/packages/3d/f2/1873d14df922793d998daa66cb96a04361cc6658ea91dca2621d3bf13f4b/pyred-0.1.2-py3-none-any.whl", 
            "md5_digest": "5be66e8f2ea56aa7ff60893b9ac4302f", 
            "downloads": 0, 
            "filename": "pyred-0.1.2-py3-none-any.whl", 
            "packagetype": "bdist_wheel", 
            "path": "3d/f2/1873d14df922793d998daa66cb96a04361cc6658ea91dca2621d3bf13f4b/pyred-0.1.2-py3-none-any.whl", 
            "digests": {
                "sha256": "6160928c8630fea144c70027f40b33507eae172cb31ebbef630957c4e0309011", 
                "md5": "5be66e8f2ea56aa7ff60893b9ac4302f"
            }, 
            "sha256_digest": "6160928c8630fea144c70027f40b33507eae172cb31ebbef630957c4e0309011", 
            "size": 11187
        }, 
        {
            "has_sig": false, 
            "upload_time": "2018-04-16T13:57:20", 
            "comment_text": "", 
            "python_version": "source", 
            "url": "https://pypi.python.org/packages/02/52/7d5d40c73d8ac8ad25fbe05c71a3f13938e362b4065f5676d4fc69bc5b0a/pyred-0.1.2.tar.gz", 
            "md5_digest": "1023594ded7f47d88d1bb0c705b9cbad", 
            "downloads": 0, 
            "filename": "pyred-0.1.2.tar.gz", 
            "packagetype": "sdist", 
            "path": "02/52/7d5d40c73d8ac8ad25fbe05c71a3f13938e362b4065f5676d4fc69bc5b0a/pyred-0.1.2.tar.gz", 
            "digests": {
                "sha256": "ffcb925bd315cf17cae53bb5c484be45dbad13cca52bb9ea7f1497e8b76fc4d2", 
                "md5": "1023594ded7f47d88d1bb0c705b9cbad"
            }, 
            "sha256_digest": "ffcb925bd315cf17cae53bb5c484be45dbad13cca52bb9ea7f1497e8b76fc4d2", 
            "size": 5817
        }
    ], 
    "_id": null, 
    "cheesecake_installability_id": null
}