ethereum-etl-temp


Nameethereum-etl-temp JSON
Version 2.3.4 PyPI version JSON
download
home_pagehttps://github.com/blockchain-etl/ethereum-etl
SummaryTools for exporting Ethereum blockchain data to CSV or JSON
upload_time2024-03-27 23:11:25
maintainerNone
docs_urlNone
authorEvgeny Medvedev
requires_python<4,>=3.7.2
licenseNone
keywords ethereum
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI
coveralls test coverage No coveralls. 
            # Ethereum ETL

[![Build Status](https://app.travis-ci.com/blockchain-etl/ethereum-etl.svg?branch=develop)](https://travis-ci.com/github/blockchain-etl/ethereum-etl)
[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Telegram](https://img.shields.io/badge/telegram-join%20chat-blue.svg)](https://t.me/BlockchainETL)
[![Discord](https://img.shields.io/badge/discord-join%20chat-blue.svg)](https://discord.gg/tRKG7zGKtF)

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

*Do you just want to query Ethereum data right away? Use the [public dataset in BigQuery](https://console.cloud.google.com/marketplace/details/ethereum/crypto-ethereum-blockchain).*

[Full documentation available here](http://ethereum-etl.readthedocs.io/).

## Quickstart

Install Ethereum ETL:

```bash
pip3 install ethereum-etl
```

Export blocks and transactions ([Schema](docs/schema.md#blockscsv), [Reference](docs/commands.md#export_blocks_and_transactions)):

```bash
> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c
```

Export ERC20 and ERC721 transfers ([Schema](docs/schema.md#token_transferscsv), [Reference](docs/commands.md##export_token_transfers)):

```bash
> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv
```

Export traces ([Schema](docs/schema.md#tracescsv), [Reference](docs/commands.md#export_traces)):

```bash
> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv
```

---

Stream blocks, transactions, logs, token_transfers continually to console ([Reference](docs/commands.md#stream)):

```bash
> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c
```

Find other commands [here](https://ethereum-etl.readthedocs.io/en/latest/commands/).

For the latest version, check out the repo and call 
```bash
> pip3 install -e . 
> python3 ethereumetl.py
```

## Useful Links

- [Schema](https://ethereum-etl.readthedocs.io/en/latest/schema/)
- [Command Reference](https://ethereum-etl.readthedocs.io/en/latest/commands/)
- [Documentation](https://ethereum-etl.readthedocs.io/)
- [Public Datasets in BigQuery](https://github.com/blockchain-etl/public-datasets)  
- [Exporting the Blockchain](https://ethereum-etl.readthedocs.io/en/latest/exporting-the-blockchain/)
- [Querying in Amazon Athena](https://ethereum-etl.readthedocs.io/en/latest/amazon-athena/)
- [Querying in Google BigQuery](https://ethereum-etl.readthedocs.io/en/latest/google-bigquery/)
- [Querying in Kaggle](https://www.kaggle.com/bigquery/ethereum-blockchain)
- [Airflow DAGs](https://github.com/blockchain-etl/ethereum-etl-airflow)
- [Postgres ETL](https://github.com/blockchain-etl/ethereum-etl-postgresql)
- [Ethereum 2.0 ETL](https://github.com/blockchain-etl/ethereum2-etl)

## Running Tests

```bash
> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv
``` 

### Running Tox Tests

```bash
> pip3 install tox
> tox
```

## Running in Docker

1. Install Docker: https://docs.docker.com/get-docker/

2. Build a docker image
        
        > docker build -t ethereum-etl:latest .
        > docker image ls
        
3. Run a container out of the image

        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io

4. Run streaming to console or Pub/Sub

        > docker build -t ethereum-etl:latest .
        > echo "Stream to console"
        > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
        > echo "Stream to Pub/Sub"
        > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum

If running on Apple M1 chip add the `--platform linux/x86_64` option to the `build` and `run` commands e.g.:

```
docker build --platform linux/x86_64 -t ethereum-etl:latest .
docker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000
```

## Projects using Ethereum ETL
* [Google](https://goo.gl/oY5BCQ) - Public BigQuery Ethereum datasets
* [Nansen](https://nansen.ai/query?ref=ethereumetl) - Analytics platform for Ethereum

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/blockchain-etl/ethereum-etl",
    "name": "ethereum-etl-temp",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4,>=3.7.2",
    "maintainer_email": null,
    "keywords": "ethereum",
    "author": "Evgeny Medvedev",
    "author_email": "evge.medvedev@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/d2/f3/26f68b50805fb68da3ac35867f01b5f1c145345bfeaa7aceba01d663cb6b/ethereum-etl-temp-2.3.4.tar.gz",
    "platform": null,
    "description": "# Ethereum ETL\n\n[![Build Status](https://app.travis-ci.com/blockchain-etl/ethereum-etl.svg?branch=develop)](https://travis-ci.com/github/blockchain-etl/ethereum-etl)\n[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n[![Telegram](https://img.shields.io/badge/telegram-join%20chat-blue.svg)](https://t.me/BlockchainETL)\n[![Discord](https://img.shields.io/badge/discord-join%20chat-blue.svg)](https://discord.gg/tRKG7zGKtF)\n\nEthereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.\n\n*Do you just want to query Ethereum data right away? Use the [public dataset in BigQuery](https://console.cloud.google.com/marketplace/details/ethereum/crypto-ethereum-blockchain).*\n\n[Full documentation available here](http://ethereum-etl.readthedocs.io/).\n\n## Quickstart\n\nInstall Ethereum ETL:\n\n```bash\npip3 install ethereum-etl\n```\n\nExport blocks and transactions ([Schema](docs/schema.md#blockscsv), [Reference](docs/commands.md#export_blocks_and_transactions)):\n\n```bash\n> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \\\n--blocks-output blocks.csv --transactions-output transactions.csv \\\n--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c\n```\n\nExport ERC20 and ERC721 transfers ([Schema](docs/schema.md#token_transferscsv), [Reference](docs/commands.md##export_token_transfers)):\n\n```bash\n> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \\\n--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv\n```\n\nExport traces ([Schema](docs/schema.md#tracescsv), [Reference](docs/commands.md#export_traces)):\n\n```bash\n> ethereumetl export_traces --start-block 0 --end-block 500000 \\\n--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv\n```\n\n---\n\nStream blocks, transactions, logs, token_transfers continually to console ([Reference](docs/commands.md#stream)):\n\n```bash\n> pip3 install ethereum-etl[streaming]\n> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \\\n--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c\n```\n\nFind other commands [here](https://ethereum-etl.readthedocs.io/en/latest/commands/).\n\nFor the latest version, check out the repo and call \n```bash\n> pip3 install -e . \n> python3 ethereumetl.py\n```\n\n## Useful Links\n\n- [Schema](https://ethereum-etl.readthedocs.io/en/latest/schema/)\n- [Command Reference](https://ethereum-etl.readthedocs.io/en/latest/commands/)\n- [Documentation](https://ethereum-etl.readthedocs.io/)\n- [Public Datasets in BigQuery](https://github.com/blockchain-etl/public-datasets)  \n- [Exporting the Blockchain](https://ethereum-etl.readthedocs.io/en/latest/exporting-the-blockchain/)\n- [Querying in Amazon Athena](https://ethereum-etl.readthedocs.io/en/latest/amazon-athena/)\n- [Querying in Google BigQuery](https://ethereum-etl.readthedocs.io/en/latest/google-bigquery/)\n- [Querying in Kaggle](https://www.kaggle.com/bigquery/ethereum-blockchain)\n- [Airflow DAGs](https://github.com/blockchain-etl/ethereum-etl-airflow)\n- [Postgres ETL](https://github.com/blockchain-etl/ethereum-etl-postgresql)\n- [Ethereum 2.0 ETL](https://github.com/blockchain-etl/ethereum2-etl)\n\n## Running Tests\n\n```bash\n> pip3 install -e .[dev,streaming]\n> export ETHEREUM_ETL_RUN_SLOW_TESTS=True\n> export PROVIDER_URL=<your_porvider_uri>\n> pytest -vv\n``` \n\n### Running Tox Tests\n\n```bash\n> pip3 install tox\n> tox\n```\n\n## Running in Docker\n\n1. Install Docker: https://docs.docker.com/get-docker/\n\n2. Build a docker image\n        \n        > docker build -t ethereum-etl:latest .\n        > docker image ls\n        \n3. Run a container out of the image\n\n        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io\n        > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io\n\n4. Run streaming to console or Pub/Sub\n\n        > docker build -t ethereum-etl:latest .\n        > echo \"Stream to console\"\n        > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt\n        > echo \"Stream to Pub/Sub\"\n        > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum\n\nIf running on Apple M1 chip add the `--platform linux/x86_64` option to the `build` and `run` commands e.g.:\n\n```\ndocker build --platform linux/x86_64 -t ethereum-etl:latest .\ndocker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000\n```\n\n## Projects using Ethereum ETL\n* [Google](https://goo.gl/oY5BCQ) - Public BigQuery Ethereum datasets\n* [Nansen](https://nansen.ai/query?ref=ethereumetl) - Analytics platform for Ethereum",
    "bugtrack_url": null,
    "license": null,
    "summary": "Tools for exporting Ethereum blockchain data to CSV or JSON",
    "version": "2.3.4",
    "project_urls": {
        "Bug Reports": "https://github.com/blockchain-etl/ethereum-etl/issues",
        "Chat": "https://gitter.im/ethereum-etl/Lobby",
        "Homepage": "https://github.com/blockchain-etl/ethereum-etl",
        "Source": "https://github.com/blockchain-etl/ethereum-etl"
    },
    "split_keywords": [
        "ethereum"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "d2f326f68b50805fb68da3ac35867f01b5f1c145345bfeaa7aceba01d663cb6b",
                "md5": "cbab7a8a8283492113aa950d310043e5",
                "sha256": "b9c3751dd0486193b194ab6288bacb6507f3dffd1fe77b943a482f7f51f1e60a"
            },
            "downloads": -1,
            "filename": "ethereum-etl-temp-2.3.4.tar.gz",
            "has_sig": false,
            "md5_digest": "cbab7a8a8283492113aa950d310043e5",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4,>=3.7.2",
            "size": 341717,
            "upload_time": "2024-03-27T23:11:25",
            "upload_time_iso_8601": "2024-03-27T23:11:25.131840Z",
            "url": "https://files.pythonhosted.org/packages/d2/f3/26f68b50805fb68da3ac35867f01b5f1c145345bfeaa7aceba01d663cb6b/ethereum-etl-temp-2.3.4.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-03-27 23:11:25",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "blockchain-etl",
    "github_project": "ethereum-etl",
    "travis_ci": true,
    "coveralls": false,
    "github_actions": true,
    "tox": true,
    "lcname": "ethereum-etl-temp"
}
Evgeny Medvedev
Elapsed time: 0.50856s