# TradingView Scraper
[](https://www.python.org/downloads/release/python-380/)
[](https://github.com/psf/black)
[](https://opensource.org/licenses/MIT)
This is a Python library for scraping ideas and indicators from [TradingView.com](https://www.tradingview.com). The goal is to develop this package to scrape anything on [TradingView.com](https://www.tradingview.com) with real-time responses.
**Thanks to contributors!**
## To-Do List
### Completed Tasks
- [x] Export as a `CSV` file
- [x] Export as `JSON`
### Pending Tasks
- [ ] Scrape Symbol Subpages:
- [x] [Ideas](https://www.tradingview.com/symbols/BTCUSD/ideas/)
- [x] [Indicators](https://www.tradingview.com/symbols/BTCUSD/technicals/)
- [ ] [Overview](https://www.tradingview.com/symbols/BTCUSD/)
- [x] [News](https://www.tradingview.com/symbols/BTCUSD/news/)
- [ ] [Minds](https://www.tradingview.com/symbols/BTCUSD/minds/)
- [ ] [Technical](https://www.tradingview.com/symbols/BTCUSD/technicals/)
- [ ] [Market](https://www.tradingview.com/symbols/BTCUSD/markets/)
- [ ] Get data using TradingView WebSocket
- [ ] Additional suggestions welcome!
### To be aware of the latest changes, go to the [end of this page](https://github.com/mnwato/tradingview-scraper#changes).
## Features
- **Idea page Scraping**
- Title
- Paragraph
- Preview Image
- Author
- Comments Count
- Boosts Count
- Publication Datetime
- Is Updated
- Idea Strategy
- **News page Scraping**
- Breadcrumbs
- Title
- Published datetime
- Related symbols
- Body
- Tags
- **Webpage Scraping Options**
- Scrape All Pages
- Scrape a Specific Range of Pages
- **Indicator Extraction**
- Extract values for indicators like `RSI`, `Stoch.K`, etc.
- [Full list of indicators](https://github.com/mnwato/tradingview-scraper/blob/dev/tradingview_scraper/indicators.txt)
- **Export Formats**
- CSV
- JSON
- **Output Format**
- Returns data in JSON format
Here’s a revised version of the Installation section that enhances clarity and encourages user engagement:
## Installation
To get started with the TradingView Scraper library, follow these simple steps:
1. **Open your terminal**: Launch your preferred command line interface.
2. **Install the package**: Run the following command to install the TradingView Scraper:
```sh
pip install tradingview-scraper
```
3. **Upgrade if necessary**: If you already have the library installed and want to upgrade to the latest version, use:
```sh
pip install --upgrade tradingview-scraper
```
Here’s a revised version of the Examples section, focusing on clarity, ease of understanding, and providing essential information about default values:
## Examples
### 1. Fast Running (Default Parameters)
To quickly scrape ideas using default settings, use the following code:
```python
from symbols.ideas import Ideas
# Initialize the Ideas scraper with default parameters
ideas_scraper = Ideas() # Default: export_result=False, export_type='json'
ideas = ideas_scraper.scrape() # Default symbol: 'BTCUSD'
print("Ideas:", ideas)
```
**Default Parameters:**
- `export_result`: `False` (no file will be saved)
- `export_type`: `'json'` (output format)
### 2. Getting Ideas for a Specific Symbol, Export Type, and Pages
To scrape ideas for a specific symbol and export them as a CSV file, you can specify the parameters:
```python
from symbols.ideas import Ideas
# Initialize the Ideas scraper with custom parameters
ideas_scraper = Ideas(
export_result=True, # Set to True to save the results
export_type='csv' # Specify the export type (json or csv)
)
# Scrape ideas for the ETHUSD symbol, from page 1 to page 2
ideas = ideas_scraper.scrape(
symbol="ETHUSD",
startPage=1,
endPage=2,
sort="popular" # Could be 'popupar' or 'recent'
)
print("Ideas:", ideas)
```
**Output Format:**
The output will always be a list of Python dictionaries, structured as follows:
```json
[
{
"title": "Bitcoin -65% crash, ETH -83%, DOGE -89%, SHIBA -90%",
"paragraph": "Name your altcoin in the comment section, and I will do a technical analysis for you!\n\nThe crypto market looks ...",
"preview_image": "https://s3.tradingview.com/6/6VQphWH6_mid.png",
"author": "Xanrox",
"comments_count": "295",
"boosts_count": "678",
"publication_datetime": "2024-08-18T05:55:19.000Z",
"is_updated": "True",
"idea_strategy": "Short"
}
]
```
- When `export_result=True`, the default `export_type` is `'json'`, and a JSON file will be saved in the `/export` directory. To save as CSV, set `export_type='csv'`.
### 3. Getting Indicators Status
To scrape the status of specific indicators, use the following code:
```python
from symbols.indicators import Indicators
# Initialize the Indicators scraper with export options
indicators_scraper = Indicators(export_result=True, export_type='json')
# Scrape indicators for the BTCUSD symbol from the BINANCE exchange
indicators = indicators_scraper.scrape(
exchange="BINANCE",
symbol="BTCUSD",
indicators=["RSI", "Stoch.K"]
)
print("Indicators:", indicators)
```
**Output:**
```json
{"RSI": "46.34926112", "Stoch.K": "40.40173723"}
```
### 4. Getting All Indicators
If you want to retrieve all available indicators for a symbol, set `allIndicators=True`:
```python
# Scrape all indicators for the BTCUSD symbol
indicators_scraper = Indicators(export_result=True, export_type='json')
indicators = indicators_scraper.scrape(
symbol="BTCUSD",
allIndicators=True
)
print("All Indicators:", indicators)
```
### 5. Getting News Headlines/Content
```python
# Create an instance of the NewsScraper with export options
news_scraper = NewsScraper(export_result=True, export_type='json')
# Retrieve news headlines from a specific provider
news_headlines = news_scraper.scrape_headlines(
provider='newsbtc', # Specify the news provider
# symbol='BTCUSD', # Uncomment and specify if needed
# exchange='BINANCE', # Uncomment and specify if needed
sort='latest'
)
# Retrieve detailed news content for a specific story
news_content = news_scraper.scrape_news_content(
story_path=news_headlines[0]['story_path'] # Specify the story path from scraped headlines
)
```
- To Retrieve News by Providers:
- Specify a `provider`.
- Ensure that both `symbol` and `exchange` are left empty.
- Retrieve news by symbol:
- Leave the `provider` empty.
- Specify both `symbol` and `exchange`.
#### Output (news headline):
```json
[
{
"breadcrumbs": "News > U.Today > Bitcoin ETFs Record Enormous Outflows",
"title": "Bitcoin ETFs Record Enormous Outflows",
"published_datetime": "Wed, 04 Sep 2024 07:55:38 GMT",
"related_symbols": [
{
"name": "BTCUSDT",
"logo": "https://s3-symbol-logo.tradingview.com/crypto/XTVCUSDT.svg"
}
],
"body": ["""<List of text page content>"""],
"tags": ["Crypto", "U.Today"]}
]
```
#### Output (news content):
```json
[
{
"id": "tag:reuters.com,2024:newsml_L1N3KM09S:0",
"title": "Goldman Sachs sees biggest boost to US economy from Harris win",
"provider": "reuters",
"sourceLogoId": "reuters",
"published": 1725443676,
"source": "Reuters",
"urgency": 2,
"permission": "preview",
"relatedSymbols": [
{
"symbol": "BITMEX:XBTETH.P",
"currency-logoid": "country/US",
"base-currency-logoid": "crypto/XTVCBTC"
},
{
"symbol": "ICEUS:DXY",
"logoid": "indices/u-s-dollar-index"
}
],
"storyPath": "/news/reuters.com,2024:newsml_L1N3KM09S:0-goldman-sachs-sees-biggest-boost-to-us-economy-from-harris-win/"
}
]
```
## Changes:
- Release `0.3.0`:
Add news scraper
- Release `0.2.9`:
Refactor for new TradingView structure
- Release `0.1.0`:
The name of `ClassA` changed to `Ideas`
## License:
```
[MIT]
```
Raw data
{
"_id": null,
"home_page": "https://github.com/mnwato/tradingview-scraper",
"name": "tradingview-scraper",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "tradingview, scraper, python, crawler, financial",
"author": "Mostafa Najmi",
"author_email": "m.n.irib@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/42/95/be9f8874e1d898b49762b741e400b205b686964ffbf1d9755b75950bc38f/tradingview-scraper-0.3.0.tar.gz",
"platform": null,
"description": "# TradingView Scraper\n[](https://www.python.org/downloads/release/python-380/)\n[](https://github.com/psf/black)\n[](https://opensource.org/licenses/MIT)\n\n\nThis is a Python library for scraping ideas and indicators from [TradingView.com](https://www.tradingview.com). The goal is to develop this package to scrape anything on [TradingView.com](https://www.tradingview.com) with real-time responses. \n**Thanks to contributors!**\n\n## To-Do List\n\n### Completed Tasks\n- [x] Export as a `CSV` file\n- [x] Export as `JSON`\n\n### Pending Tasks\n- [ ] Scrape Symbol Subpages:\n - [x] [Ideas](https://www.tradingview.com/symbols/BTCUSD/ideas/)\n - [x] [Indicators](https://www.tradingview.com/symbols/BTCUSD/technicals/)\n - [ ] [Overview](https://www.tradingview.com/symbols/BTCUSD/)\n - [x] [News](https://www.tradingview.com/symbols/BTCUSD/news/)\n - [ ] [Minds](https://www.tradingview.com/symbols/BTCUSD/minds/)\n - [ ] [Technical](https://www.tradingview.com/symbols/BTCUSD/technicals/)\n - [ ] [Market](https://www.tradingview.com/symbols/BTCUSD/markets/)\n - [ ] Get data using TradingView WebSocket\n - [ ] Additional suggestions welcome!\n\n### To be aware of the latest changes, go to the [end of this page](https://github.com/mnwato/tradingview-scraper#changes).\n\n## Features\n\n- **Idea page Scraping**\n - Title\n - Paragraph\n - Preview Image\n - Author\n - Comments Count\n - Boosts Count\n - Publication Datetime\n - Is Updated\n - Idea Strategy\n\n- **News page Scraping**\n - Breadcrumbs\n - Title\n - Published datetime\n - Related symbols\n - Body\n - Tags\n\n- **Webpage Scraping Options**\n - Scrape All Pages\n - Scrape a Specific Range of Pages\n\n- **Indicator Extraction**\n - Extract values for indicators like `RSI`, `Stoch.K`, etc. \n - [Full list of indicators](https://github.com/mnwato/tradingview-scraper/blob/dev/tradingview_scraper/indicators.txt)\n\n- **Export Formats**\n - CSV\n - JSON\n\n- **Output Format**\n - Returns data in JSON format\n\nHere\u2019s a revised version of the Installation section that enhances clarity and encourages user engagement:\n\n\n## Installation\n\nTo get started with the TradingView Scraper library, follow these simple steps:\n\n1. **Open your terminal**: Launch your preferred command line interface.\n\n2. **Install the package**: Run the following command to install the TradingView Scraper:\n ```sh\n pip install tradingview-scraper\n ```\n\n3. **Upgrade if necessary**: If you already have the library installed and want to upgrade to the latest version, use:\n ```sh\n pip install --upgrade tradingview-scraper\n ```\n\nHere\u2019s a revised version of the Examples section, focusing on clarity, ease of understanding, and providing essential information about default values:\n\n\n## Examples\n\n### 1. Fast Running (Default Parameters)\nTo quickly scrape ideas using default settings, use the following code:\n```python\nfrom symbols.ideas import Ideas\n\n# Initialize the Ideas scraper with default parameters\nideas_scraper = Ideas() # Default: export_result=False, export_type='json'\nideas = ideas_scraper.scrape() # Default symbol: 'BTCUSD'\nprint(\"Ideas:\", ideas)\n```\n**Default Parameters:**\n- `export_result`: `False` (no file will be saved)\n- `export_type`: `'json'` (output format)\n\n### 2. Getting Ideas for a Specific Symbol, Export Type, and Pages\nTo scrape ideas for a specific symbol and export them as a CSV file, you can specify the parameters:\n```python\nfrom symbols.ideas import Ideas\n\n# Initialize the Ideas scraper with custom parameters\nideas_scraper = Ideas(\n export_result=True, # Set to True to save the results\n export_type='csv' # Specify the export type (json or csv)\n)\n\n# Scrape ideas for the ETHUSD symbol, from page 1 to page 2\nideas = ideas_scraper.scrape(\n symbol=\"ETHUSD\",\n startPage=1,\n endPage=2,\n sort=\"popular\" # Could be 'popupar' or 'recent'\n)\nprint(\"Ideas:\", ideas)\n```\n\n**Output Format:**\nThe output will always be a list of Python dictionaries, structured as follows:\n```json\n[\n {\n \"title\": \"Bitcoin -65% crash, ETH -83%, DOGE -89%, SHIBA -90%\",\n \"paragraph\": \"Name your altcoin in the comment section, and I will do a technical analysis for you!\\n\\nThe crypto market looks ...\",\n \"preview_image\": \"https://s3.tradingview.com/6/6VQphWH6_mid.png\",\n \"author\": \"Xanrox\",\n \"comments_count\": \"295\",\n \"boosts_count\": \"678\",\n \"publication_datetime\": \"2024-08-18T05:55:19.000Z\",\n \"is_updated\": \"True\",\n \"idea_strategy\": \"Short\"\n }\n]\n```\n- When `export_result=True`, the default `export_type` is `'json'`, and a JSON file will be saved in the `/export` directory. To save as CSV, set `export_type='csv'`.\n\n### 3. Getting Indicators Status\nTo scrape the status of specific indicators, use the following code:\n```python\nfrom symbols.indicators import Indicators\n\n# Initialize the Indicators scraper with export options\nindicators_scraper = Indicators(export_result=True, export_type='json')\n\n# Scrape indicators for the BTCUSD symbol from the BINANCE exchange\nindicators = indicators_scraper.scrape(\n exchange=\"BINANCE\",\n symbol=\"BTCUSD\",\n indicators=[\"RSI\", \"Stoch.K\"]\n)\nprint(\"Indicators:\", indicators)\n```\n\n**Output:**\n```json\n{\"RSI\": \"46.34926112\", \"Stoch.K\": \"40.40173723\"}\n```\n\n### 4. Getting All Indicators\nIf you want to retrieve all available indicators for a symbol, set `allIndicators=True`:\n```python\n# Scrape all indicators for the BTCUSD symbol\nindicators_scraper = Indicators(export_result=True, export_type='json')\nindicators = indicators_scraper.scrape(\n symbol=\"BTCUSD\",\n allIndicators=True\n)\nprint(\"All Indicators:\", indicators)\n```\n\n### 5. Getting News Headlines/Content\n```python\n# Create an instance of the NewsScraper with export options\nnews_scraper = NewsScraper(export_result=True, export_type='json')\n\n# Retrieve news headlines from a specific provider\nnews_headlines = news_scraper.scrape_headlines(\n provider='newsbtc', # Specify the news provider\n # symbol='BTCUSD', # Uncomment and specify if needed\n # exchange='BINANCE', # Uncomment and specify if needed\n sort='latest'\n)\n\n# Retrieve detailed news content for a specific story\nnews_content = news_scraper.scrape_news_content(\n story_path=news_headlines[0]['story_path'] # Specify the story path from scraped headlines\n)\n```\n- To Retrieve News by Providers:\n - Specify a `provider`.\n - Ensure that both `symbol` and `exchange` are left empty.\n- Retrieve news by symbol:\n - Leave the `provider` empty.\n - Specify both `symbol` and `exchange`.\n\n#### Output (news headline):\n```json\n[\n {\n \"breadcrumbs\": \"News > U.Today > Bitcoin ETFs Record Enormous Outflows\",\n \"title\": \"Bitcoin ETFs Record Enormous Outflows\",\n \"published_datetime\": \"Wed, 04 Sep 2024 07:55:38 GMT\",\n \"related_symbols\": [\n {\n \"name\": \"BTCUSDT\",\n \"logo\": \"https://s3-symbol-logo.tradingview.com/crypto/XTVCUSDT.svg\"\n }\n ],\n \"body\": [\"\"\"<List of text page content>\"\"\"],\n \"tags\": [\"Crypto\", \"U.Today\"]}\n]\n```\n#### Output (news content):\n```json\n[\n {\n \"id\": \"tag:reuters.com,2024:newsml_L1N3KM09S:0\",\n \"title\": \"Goldman Sachs sees biggest boost to US economy from Harris win\",\n \"provider\": \"reuters\",\n \"sourceLogoId\": \"reuters\",\n \"published\": 1725443676,\n \"source\": \"Reuters\",\n \"urgency\": 2,\n \"permission\": \"preview\",\n \"relatedSymbols\": [\n {\n \"symbol\": \"BITMEX:XBTETH.P\",\n \"currency-logoid\": \"country/US\",\n \"base-currency-logoid\": \"crypto/XTVCBTC\"\n },\n {\n \"symbol\": \"ICEUS:DXY\",\n \"logoid\": \"indices/u-s-dollar-index\"\n }\n ],\n \"storyPath\": \"/news/reuters.com,2024:newsml_L1N3KM09S:0-goldman-sachs-sees-biggest-boost-to-us-economy-from-harris-win/\"\n }\n]\n```\n\n## Changes:\n- Release `0.3.0`: \n Add news scraper\n- Release `0.2.9`: \n Refactor for new TradingView structure\n- Release `0.1.0`: \n The name of `ClassA` changed to `Ideas`\n\n## License:\n```\n[MIT]\n```\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Tradingview scraper tool",
"version": "0.3.0",
"project_urls": {
"Download": "https://github.com/mnwato/tradingview-scraper/archive/refs/tags/0.3.0.zip",
"Homepage": "https://github.com/mnwato/tradingview-scraper"
},
"split_keywords": [
"tradingview",
" scraper",
" python",
" crawler",
" financial"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cfbdc8f9476680bc1efa0e5c64c456f9523f3ad1bfc5ba51906a2674e0b33819",
"md5": "d02ca891a69a1e3ba3bedfc2c9a03fbf",
"sha256": "3daaafe40fde991f39a2223ed7e8a8ab878038ded4ba331838bf78bec6123daa"
},
"downloads": -1,
"filename": "tradingview_scraper-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d02ca891a69a1e3ba3bedfc2c9a03fbf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 16810,
"upload_time": "2024-09-04T12:29:08",
"upload_time_iso_8601": "2024-09-04T12:29:08.964434Z",
"url": "https://files.pythonhosted.org/packages/cf/bd/c8f9476680bc1efa0e5c64c456f9523f3ad1bfc5ba51906a2674e0b33819/tradingview_scraper-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4295be9f8874e1d898b49762b741e400b205b686964ffbf1d9755b75950bc38f",
"md5": "43f7403333ddfcbc0b82a233a9acc09e",
"sha256": "9c76938a74aa88061f6e717cc95b0f12748683879797760cd7eab7d22336bb74"
},
"downloads": -1,
"filename": "tradingview-scraper-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "43f7403333ddfcbc0b82a233a9acc09e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 17220,
"upload_time": "2024-09-04T12:29:11",
"upload_time_iso_8601": "2024-09-04T12:29:11.229919Z",
"url": "https://files.pythonhosted.org/packages/42/95/be9f8874e1d898b49762b741e400b205b686964ffbf1d9755b75950bc38f/tradingview-scraper-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-04 12:29:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mnwato",
"github_project": "tradingview-scraper",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "tradingview-scraper"
}
JSON
