glide-etl


Nameglide-etl JSON
Version 2.4.2 PyPI version JSON
download
home_pagehttps://github.com/glide-rollup/glide-etl
SummaryTools for exporting Glide blockchain data to CSV or JSON
upload_time2024-10-14 04:16:58
maintainerNone
docs_urlNone
authorAvenbreaks
requires_python<4,>=3.7.2
licenseNone
keywords ethereum
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI
coveralls test coverage No coveralls. 
            # Ethereum ETL

[![Build Status](https://app.travis-ci.com/blockchain-etl/ethereum-etl.svg?branch=develop)](https://travis-ci.com/github/blockchain-etl/ethereum-etl)
[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Telegram](https://img.shields.io/badge/telegram-join%20chat-blue.svg)](https://t.me/BlockchainETL)
[![Discord](https://img.shields.io/badge/discord-join%20chat-blue.svg)](https://discord.gg/tRKG7zGKtF)

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

*Do you just want to query Ethereum data right away? Use the [public dataset in BigQuery](https://console.cloud.google.com/marketplace/details/ethereum/crypto-ethereum-blockchain).*

[Full documentation available here](http://ethereum-etl.readthedocs.io/).

## Quickstart

Install Ethereum ETL:

```bash
pip3 install ethereum-etl
```

Export blocks and transactions ([Schema](docs/schema.md#blockscsv), [Reference](docs/commands.md#export_blocks_and_transactions)):

```bash
> glideetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c
```

Export ERC20 and ERC721 transfers ([Schema](docs/schema.md#token_transferscsv), [Reference](docs/commands.md##export_token_transfers)):

```bash
> glideetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv
```

Export traces ([Schema](docs/schema.md#tracescsv), [Reference](docs/commands.md#export_traces)):

```bash
> glideetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv
```

---

Stream blocks, transactions, logs, token_transfers continually to console ([Reference](docs/commands.md#stream)):

```bash
> pip3 install ethereum-etl[streaming]
> glideetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c
```

Find other commands [here](https://ethereum-etl.readthedocs.io/en/latest/commands/).

For the latest version, check out the repo and call 
```bash
> pip3 install -e . 
> python3 glideetl.py
```

## Useful Links

- [Schema](https://ethereum-etl.readthedocs.io/en/latest/schema/)
- [Command Reference](https://ethereum-etl.readthedocs.io/en/latest/commands/)
- [Documentation](https://ethereum-etl.readthedocs.io/)
- [Public Datasets in BigQuery](https://github.com/blockchain-etl/public-datasets)  
- [Exporting the Blockchain](https://ethereum-etl.readthedocs.io/en/latest/exporting-the-blockchain/)
- [Querying in Amazon Athena](https://ethereum-etl.readthedocs.io/en/latest/amazon-athena/)
- [Querying in Google BigQuery](https://ethereum-etl.readthedocs.io/en/latest/google-bigquery/)
- [Querying in Kaggle](https://www.kaggle.com/bigquery/ethereum-blockchain)
- [Airflow DAGs](https://github.com/blockchain-etl/ethereum-etl-airflow)
- [Postgres ETL](https://github.com/blockchain-etl/ethereum-etl-postgresql)
- [Ethereum 2.0 ETL](https://github.com/blockchain-etl/ethereum2-etl)

## Running Tests

```bash
> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv
``` 

### Running Tox Tests

```bash
> pip3 install tox
> tox
```

## Running in Docker

1. Install Docker: https://docs.docker.com/get-docker/

2. Build a docker image
        
        > docker build -t ethereum-etl:latest .
        > docker image ls
        
3. Run a container out of the image

        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io

4. Run streaming to console or Pub/Sub

        > docker build -t ethereum-etl:latest .
        > echo "Stream to console"
        > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
        > echo "Stream to Pub/Sub"
        > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum

If running on an Apple M1 chip add the `--platform linux/x86_64` option to the `build` and `run` commands e.g.:

```
docker build --platform linux/x86_64 -t ethereum-etl:latest .
docker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000
```

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/glide-rollup/glide-etl",
    "name": "glide-etl",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4,>=3.7.2",
    "maintainer_email": null,
    "keywords": "ethereum",
    "author": "Avenbreaks",
    "author_email": "joyeth@alumni.unair.ac.id",
    "download_url": "https://files.pythonhosted.org/packages/c8/a5/4480e0313842d67846c00c3cffffa028ecf70379e6dc229232ce319d3233/glide-etl-2.4.2.tar.gz",
    "platform": null,
    "description": "# Ethereum ETL\n\n[![Build Status](https://app.travis-ci.com/blockchain-etl/ethereum-etl.svg?branch=develop)](https://travis-ci.com/github/blockchain-etl/ethereum-etl)\n[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n[![Telegram](https://img.shields.io/badge/telegram-join%20chat-blue.svg)](https://t.me/BlockchainETL)\n[![Discord](https://img.shields.io/badge/discord-join%20chat-blue.svg)](https://discord.gg/tRKG7zGKtF)\n\nEthereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.\n\n*Do you just want to query Ethereum data right away? Use the [public dataset in BigQuery](https://console.cloud.google.com/marketplace/details/ethereum/crypto-ethereum-blockchain).*\n\n[Full documentation available here](http://ethereum-etl.readthedocs.io/).\n\n## Quickstart\n\nInstall Ethereum ETL:\n\n```bash\npip3 install ethereum-etl\n```\n\nExport blocks and transactions ([Schema](docs/schema.md#blockscsv), [Reference](docs/commands.md#export_blocks_and_transactions)):\n\n```bash\n> glideetl export_blocks_and_transactions --start-block 0 --end-block 500000 \\\n--blocks-output blocks.csv --transactions-output transactions.csv \\\n--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c\n```\n\nExport ERC20 and ERC721 transfers ([Schema](docs/schema.md#token_transferscsv), [Reference](docs/commands.md##export_token_transfers)):\n\n```bash\n> glideetl export_token_transfers --start-block 0 --end-block 500000 \\\n--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv\n```\n\nExport traces ([Schema](docs/schema.md#tracescsv), [Reference](docs/commands.md#export_traces)):\n\n```bash\n> glideetl export_traces --start-block 0 --end-block 500000 \\\n--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv\n```\n\n---\n\nStream blocks, transactions, logs, token_transfers continually to console ([Reference](docs/commands.md#stream)):\n\n```bash\n> pip3 install ethereum-etl[streaming]\n> glideetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \\\n--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c\n```\n\nFind other commands [here](https://ethereum-etl.readthedocs.io/en/latest/commands/).\n\nFor the latest version, check out the repo and call \n```bash\n> pip3 install -e . \n> python3 glideetl.py\n```\n\n## Useful Links\n\n- [Schema](https://ethereum-etl.readthedocs.io/en/latest/schema/)\n- [Command Reference](https://ethereum-etl.readthedocs.io/en/latest/commands/)\n- [Documentation](https://ethereum-etl.readthedocs.io/)\n- [Public Datasets in BigQuery](https://github.com/blockchain-etl/public-datasets)  \n- [Exporting the Blockchain](https://ethereum-etl.readthedocs.io/en/latest/exporting-the-blockchain/)\n- [Querying in Amazon Athena](https://ethereum-etl.readthedocs.io/en/latest/amazon-athena/)\n- [Querying in Google BigQuery](https://ethereum-etl.readthedocs.io/en/latest/google-bigquery/)\n- [Querying in Kaggle](https://www.kaggle.com/bigquery/ethereum-blockchain)\n- [Airflow DAGs](https://github.com/blockchain-etl/ethereum-etl-airflow)\n- [Postgres ETL](https://github.com/blockchain-etl/ethereum-etl-postgresql)\n- [Ethereum 2.0 ETL](https://github.com/blockchain-etl/ethereum2-etl)\n\n## Running Tests\n\n```bash\n> pip3 install -e .[dev,streaming]\n> export ETHEREUM_ETL_RUN_SLOW_TESTS=True\n> export PROVIDER_URL=<your_porvider_uri>\n> pytest -vv\n``` \n\n### Running Tox Tests\n\n```bash\n> pip3 install tox\n> tox\n```\n\n## Running in Docker\n\n1. Install Docker: https://docs.docker.com/get-docker/\n\n2. Build a docker image\n        \n        > docker build -t ethereum-etl:latest .\n        > docker image ls\n        \n3. Run a container out of the image\n\n        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io\n        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io\n\n4. Run streaming to console or Pub/Sub\n\n        > docker build -t ethereum-etl:latest .\n        > echo \"Stream to console\"\n        > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt\n        > echo \"Stream to Pub/Sub\"\n        > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum\n\nIf running on an Apple M1 chip add the `--platform linux/x86_64` option to the `build` and `run` commands e.g.:\n\n```\ndocker build --platform linux/x86_64 -t ethereum-etl:latest .\ndocker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000\n```\n\n\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "Tools for exporting Glide blockchain data to CSV or JSON",
    "version": "2.4.2",
    "project_urls": {
        "Bug Reports": "https://github.com/glide-rollup/glide-etl/issues",
        "Homepage": "https://github.com/glide-rollup/glide-etl",
        "Source": "https://github.com/glide-rollup/glide-etl"
    },
    "split_keywords": [
        "ethereum"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "ff5a75459eccdf480c3ef6db25c93e305f3713addee1b217429b47274f40cc80",
                "md5": "b601d0f5c31c290a5c2f982c289ca535",
                "sha256": "f0e2a7a628909e3005ca43613ca1786b946d391501f738d6f299df631ddf4a06"
            },
            "downloads": -1,
            "filename": "glide_etl-2.4.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b601d0f5c31c290a5c2f982c289ca535",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4,>=3.7.2",
            "size": 480434,
            "upload_time": "2024-10-14T04:16:56",
            "upload_time_iso_8601": "2024-10-14T04:16:56.854334Z",
            "url": "https://files.pythonhosted.org/packages/ff/5a/75459eccdf480c3ef6db25c93e305f3713addee1b217429b47274f40cc80/glide_etl-2.4.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "c8a54480e0313842d67846c00c3cffffa028ecf70379e6dc229232ce319d3233",
                "md5": "ba9508ff6ed22562a511b48b2eb041ff",
                "sha256": "e019c78e588a07356c876e49d4c33251d1aa9caa714b954313610ac9020dfe4b"
            },
            "downloads": -1,
            "filename": "glide-etl-2.4.2.tar.gz",
            "has_sig": false,
            "md5_digest": "ba9508ff6ed22562a511b48b2eb041ff",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4,>=3.7.2",
            "size": 341717,
            "upload_time": "2024-10-14T04:16:58",
            "upload_time_iso_8601": "2024-10-14T04:16:58.894028Z",
            "url": "https://files.pythonhosted.org/packages/c8/a5/4480e0313842d67846c00c3cffffa028ecf70379e6dc229232ce319d3233/glide-etl-2.4.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-10-14 04:16:58",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "glide-rollup",
    "github_project": "glide-etl",
    "travis_ci": true,
    "coveralls": false,
    "github_actions": true,
    "tox": true,
    "lcname": "glide-etl"
}
Avenbreaks
Elapsed time: 0.33686s