| Name | scrapling JSON |
| Version |
0.3.2
JSON |
| download |
| home_page | https://github.com/D4Vinci/Scrapling |
| Summary | Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be! |
| upload_time | 2025-09-15 01:29:32 |
| maintainer | None |
| docs_url | None |
| author | Karim Shoair |
| requires_python | >=3.10 |
| license | BSD 3-Clause License
Copyright (c) 2024, Karim shoair
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| keywords |
web-scraping
scraping
automation
browser-automation
data-extraction
html-parsing
undetectable
playwright
selenium-alternative
web-crawler
browser
crawling
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align=center>
<br>
<a href="https://scrapling.readthedocs.io/en/latest/" target="_blank"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/poster.png" style="width: 50%; height: 100%;"/></a>
<br>
<i><code>Easy, effortless Web Scraping as it should be!</code></i>
</p>
<p align="center">
<a href="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml" alt="Tests">
<img alt="Tests" src="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg"></a>
<a href="https://badge.fury.io/py/Scrapling" alt="PyPI version">
<img alt="PyPI version" src="https://badge.fury.io/py/Scrapling.svg"></a>
<a href="https://pepy.tech/project/scrapling" alt="PyPI Downloads">
<img alt="PyPI Downloads" src="https://static.pepy.tech/personalized-badge/scrapling?period=total&units=INTERNATIONAL_SYSTEM&left_color=GRAY&right_color=GREEN&left_text=Downloads"></a>
<br/>
<a href="https://discord.gg/EMgGbDceNQ" alt="Discord" target="_blank">
<img alt="Discord" src="https://img.shields.io/discord/1360786381042880532?style=social&logo=discord&link=https%3A%2F%2Fdiscord.gg%2FEMgGbDceNQ">
</a>
<a href="https://x.com/Scrapling_dev" alt="X (formerly Twitter)">
<img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/Scrapling_dev?style=social&logo=x&link=https%3A%2F%2Fx.com%2FScrapling_dev">
</a>
<br/>
<a href="https://pypi.org/project/scrapling/" alt="Supported Python versions">
<img alt="Supported Python versions" src="https://img.shields.io/pypi/pyversions/scrapling.svg"></a>
</p>
<p align="center">
<a href="https://scrapling.readthedocs.io/en/latest/#installation">
Installation
</a>
·
<a href="https://scrapling.readthedocs.io/en/latest/overview/">
Overview
</a>
·
<a href="https://scrapling.readthedocs.io/en/latest/parsing/selection/">
Selection methods
</a>
·
<a href="https://scrapling.readthedocs.io/en/latest/fetching/choosing/">
Choosing a fetcher
</a>
·
<a href="https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup/">
Migrating from Beautifulsoup
</a>
</p>
**Stop fighting anti-bot systems. Stop rewriting selectors after every website update.**
Scrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running.
Built for the modern Web, Scrapling has its own rapid parsing engine and its fetchers to handle all Web Scraping challenges you are facing or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
```python
>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
>> StealthyFetcher.adaptive = True
# Fetch websites' source under the radar!
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
>> print(page.status)
200
>> products = page.css('.product', auto_save=True) # Scrape data that survives website design changes!
>> # Later, if the website structure changes, pass `adaptive=True`
>> products = page.css('.product', adaptive=True) # and Scrapling still finds them!
```
# Sponsors
<!-- sponsors -->
<a href="https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling" target="_blank" title="Evomi is your Swiss Quality Proxy Provider, starting at $0.49/GB"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/evomi.png"></a>
<a href="https://petrosky.io/d4vinci" target="_blank" title="PetroSky delivers cutting-edge VPS hosting."><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/petrosky.png"></a>
<a href="https://www.swiftproxy.net/" target="_blank" title="Unlock Reliable Proxy Services with Swiftproxy!"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/swiftproxy.png"></a>
<a href="https://serpapi.com/?utm_source=scrapling" target="_blank" title="Scrape Google and other search engines with SerpApi"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SerpApi.png"></a>
<a href="https://www.nstproxy.com/?type=flow&utm_source=scrapling" target="_blank" title="One Proxy Service, Infinite Solutions at Unbeatable Prices!"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/NSTproxy.png"></a>
<!-- /sponsors -->
<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suites you!</sub></i>
---
## Key Features
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all levels of Cloudflare's Turnstile with automation easily.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
### Adaptive Scraping & AI Integration
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
- 🔋 **Memory Efficient**: Optimized data structures and lazy loading for a minimal memory footprint.
- ⚡ **Fast JSON Serialization**: 10x faster than the standard library.
- 🏗️ **Battle tested**: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.
### Developer/Web Scraper Friendly Experience
- 🎯 **Interactive Web Scraping Shell**: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
- 🚀 **Use it directly from the Terminal**: Optionally, you can use Scrapling to scrape a URL without writing a single code!
- 🛠️ **Rich Navigation API**: Advanced DOM traversal with parent, sibling, and child navigation methods.
- 🧬 **Enhanced Text Processing**: Built-in regex, cleaning methods, and optimized string operations.
- 📝 **Auto Selector Generation**: Generate robust CSS/XPath selectors for any element.
- 🔌 **Familiar API**: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
- 📘 **Complete Type Coverage**: Full type hints for excellent IDE support and code completion.
### New Session Architecture
Scrapling 0.3 introduces a completely revamped session system:
- **Persistent Sessions**: Maintain cookies, headers, and authentication across multiple requests
- **Automatic Session Management**: Smart session lifecycle handling with proper cleanup
- **Session Inheritance**: All fetchers support both one-off requests and persistent session usage
- **Concurrent Session Support**: Run multiple isolated sessions simultaneously
## Getting Started
### Basic Usage
```python
from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
from scrapling.fetchers import FetcherSession, StealthySession, DynamicSession
# HTTP requests with session support
with FetcherSession(impersonate='chrome') as session: # Use latest version of Chrome's TLS fingerprint
page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
quotes = page.css('.quote .text::text')
# Or use one-off requests
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text')
# Advanced stealth mode (Keep the browser open until you finish)
with StealthySession(headless=True, solve_cloudflare=True) as session:
page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
data = page.css('#padded_content a')
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
data = page.css('#padded_content a')
# Full browser automation (Keep the browser open until you finish)
with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()') # XPath selector if you prefer it
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text')
```
### Advanced Parsing & Navigation
```python
from scrapling.fetchers import Fetcher
# Rich element selection and navigation
page = Fetcher.get('https://quotes.toscrape.com/')
# Get quotes with multiple selection methods
quotes = page.css('.quote') # CSS selector
quotes = page.xpath('//div[@class="quote"]') # XPath
quotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup-style
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # and so on...
# Find element by text content
quotes = page.find_by_text('quote', tag='div')
# Advanced navigation
first_quote = page.css_first('.quote')
quote_text = first_quote.css('.text::text')
quote_text = page.css('.quote').css_first('.text::text') # Chained selectors
quote_text = page.css_first('.quote .text').text # Using `css_first` is faster than `css` if you want the first element
author = first_quote.next_sibling.css('.author::text')
parent_container = first_quote.parent
# Element relationships and similarity
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements()
```
You can use the parser right away if you don't want to fetch websites like below:
```python
from scrapling.parser import Selector
page = Selector("<html>...</html>")
```
And it works precisely the same way!
### Async Session Management Examples
```python
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
page1 = session.get('https://quotes.toscrape.com/')
page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
# Async session usage
async with AsyncStealthySession(max_pages=2) as session:
tasks = []
urls = ['https://example.com/page1', 'https://example.com/page2']
for url in urls:
task = session.fetch(url)
tasks.append(task)
print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)
results = await asyncio.gather(*tasks)
print(session.get_pool_stats())
```
## CLI & Interactive Shell
Scrapling v0.3 includes a powerful command-line interface:
[](https://asciinema.org/a/736339)
```bash
# Launch interactive Web Scraping shell
scrapling shell
# Extract pages to a file directly without programming (Extracts the content inside `body` tag by default)
# If the output file ends with `.txt`, then the text content of the target will be extracted.
# If ended with `.md`, it will be a markdown representation of the HTML content, and `.html` will be the HTML content right away.
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
> There are many additional features, but we want to keep this page short, like the MCP server and the interactive Web Scraping Shell. Check out the full documentation [here](https://scrapling.readthedocs.io/en/latest/)
## Performance Benchmarks
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 deliver exceptional performance improvements across all operations!
### Text Extraction Speed Test (5000 nested elements)
| # | Library | Time (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 1.92 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.036x |
| 3 | Raw Lxml | 2.33 | 1.214x |
| 4 | PyQuery | 20.61 | ~11x |
| 5 | Selectolax | 80.65 | ~42x |
| 6 | BS4 with Lxml | 1283.21 | ~698x |
| 7 | MechanicalSoup | 1304.57 | ~679x |
| 8 | BS4 with html5lib | 3331.96 | ~1735x |
### Element Similarity & Text Search Performance
Scrapling's adaptive element finding capabilities significantly outperform alternatives:
| Library | Time (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 1.87 | 1.0x |
| AutoScraper | 10.24 | 5.476x |
> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
## Installation
Scrapling requires Python 3.10 or higher:
```bash
pip install scrapling
```
Starting with v0.3.2, this installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies.
### Optional Dependencies
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with
```bash
pip install "scrapling[fetchers]"
scrapling install
```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
2. Extra features:
- Install the MCP server feature:
```bash
pip install "scrapling[ai]"
```
- Install shell features (Web Scraping shell and the `extract` command):
```bash
pip install "scrapling[shell]"
```
- Install everything:
```bash
pip install "scrapling[all]"
```
Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
## Contributing
We welcome contributions! Please read our [contributing guidelines](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before getting started.
## Disclaimer
> [!CAUTION]
> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect website terms of service and robots.txt files.
## License
This work is licensed under the BSD-3-Clause License.
## Acknowledgments
This project includes code adapted from:
- Parsel (BSD License)—Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule
## Thanks and References
- [Daijro](https://github.com/daijro)'s brilliant work on [BrowserForge](https://github.com/daijro/browserforge) and [Camoufox](https://github.com/daijro/camoufox)
- [Vinyzu](https://github.com/Vinyzu)'s work on [Botright](https://github.com/Vinyzu/Botright)
- [brotector](https://github.com/kaliiiiiiiiii/brotector) for browser detection bypass techniques
- [fakebrowser](https://github.com/kkoooqq/fakebrowser) for fingerprinting research
- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches) for stealth improvements
---
<div align="center"><small>Designed & crafted with ❤️ by Karim Shoair.</small></div><br>
Raw data
{
"_id": null,
"home_page": "https://github.com/D4Vinci/Scrapling",
"name": "scrapling",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "Karim Shoair <karim.shoair@pm.me>",
"keywords": "web-scraping, scraping, automation, browser-automation, data-extraction, html-parsing, undetectable, playwright, selenium-alternative, web-crawler, browser, crawling",
"author": "Karim Shoair",
"author_email": "Karim Shoair <karim.shoair@pm.me>",
"download_url": "https://files.pythonhosted.org/packages/92/13/b4576b51f31c26c98a87dfacf0c0cee3de83c6106d8b92dde3fbe9432fd2/scrapling-0.3.2.tar.gz",
"platform": null,
"description": "<p align=center>\n <br>\n <a href=\"https://scrapling.readthedocs.io/en/latest/\" target=\"_blank\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/poster.png\" style=\"width: 50%; height: 100%;\"/></a>\n <br>\n <i><code>Easy, effortless Web Scraping as it should be!</code></i>\n</p>\n<p align=\"center\">\n <a href=\"https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml\" alt=\"Tests\">\n <img alt=\"Tests\" src=\"https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg\"></a>\n <a href=\"https://badge.fury.io/py/Scrapling\" alt=\"PyPI version\">\n <img alt=\"PyPI version\" src=\"https://badge.fury.io/py/Scrapling.svg\"></a>\n <a href=\"https://pepy.tech/project/scrapling\" alt=\"PyPI Downloads\">\n <img alt=\"PyPI Downloads\" src=\"https://static.pepy.tech/personalized-badge/scrapling?period=total&units=INTERNATIONAL_SYSTEM&left_color=GRAY&right_color=GREEN&left_text=Downloads\"></a>\n <br/>\n <a href=\"https://discord.gg/EMgGbDceNQ\" alt=\"Discord\" target=\"_blank\">\n <img alt=\"Discord\" src=\"https://img.shields.io/discord/1360786381042880532?style=social&logo=discord&link=https%3A%2F%2Fdiscord.gg%2FEMgGbDceNQ\">\n </a>\n <a href=\"https://x.com/Scrapling_dev\" alt=\"X (formerly Twitter)\">\n <img alt=\"X (formerly Twitter) Follow\" src=\"https://img.shields.io/twitter/follow/Scrapling_dev?style=social&logo=x&link=https%3A%2F%2Fx.com%2FScrapling_dev\">\n </a>\n <br/>\n <a href=\"https://pypi.org/project/scrapling/\" alt=\"Supported Python versions\">\n <img alt=\"Supported Python versions\" src=\"https://img.shields.io/pypi/pyversions/scrapling.svg\"></a>\n</p>\n\n<p align=\"center\">\n <a href=\"https://scrapling.readthedocs.io/en/latest/#installation\">\n Installation\n </a>\n \u00b7\n <a href=\"https://scrapling.readthedocs.io/en/latest/overview/\">\n Overview\n </a>\n \u00b7\n <a href=\"https://scrapling.readthedocs.io/en/latest/parsing/selection/\">\n Selection methods\n </a>\n \u00b7\n <a href=\"https://scrapling.readthedocs.io/en/latest/fetching/choosing/\">\n Choosing a fetcher\n </a>\n \u00b7\n <a href=\"https://scrapling.readthedocs.io/en/latest/tutorials/migrating_from_beautifulsoup/\">\n Migrating from Beautifulsoup\n </a>\n</p>\n\n**Stop fighting anti-bot systems. Stop rewriting selectors after every website update.**\n\nScrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running.\n\nBuilt for the modern Web, Scrapling has its own rapid parsing engine and its fetchers to handle all Web Scraping challenges you are facing or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.\n\n```python\n>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher\n>> StealthyFetcher.adaptive = True\n# Fetch websites' source under the radar!\n>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)\n>> print(page.status)\n200\n>> products = page.css('.product', auto_save=True) # Scrape data that survives website design changes!\n>> # Later, if the website structure changes, pass `adaptive=True`\n>> products = page.css('.product', adaptive=True) # and Scrapling still finds them!\n```\n\n# Sponsors \n\n<!-- sponsors -->\n\n<a href=\"https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling\" target=\"_blank\" title=\"Evomi is your Swiss Quality Proxy Provider, starting at $0.49/GB\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/evomi.png\"></a>\n<a href=\"https://petrosky.io/d4vinci\" target=\"_blank\" title=\"PetroSky delivers cutting-edge VPS hosting.\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/petrosky.png\"></a>\n<a href=\"https://www.swiftproxy.net/\" target=\"_blank\" title=\"Unlock Reliable Proxy Services with Swiftproxy!\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/swiftproxy.png\"></a>\n<a href=\"https://serpapi.com/?utm_source=scrapling\" target=\"_blank\" title=\"Scrape Google and other search engines with SerpApi\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SerpApi.png\"></a>\n<a href=\"https://www.nstproxy.com/?type=flow&utm_source=scrapling\" target=\"_blank\" title=\"One Proxy Service, Infinite Solutions at Unbeatable Prices!\"><img src=\"https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/NSTproxy.png\"></a>\n\n<!-- /sponsors -->\n\n<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suites you!</sub></i>\n\n---\n\n## Key Features\n\n### Advanced Websites Fetching with Session Support\n- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3.\n- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode.\n- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all levels of Cloudflare's Turnstile with automation easily.\n- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.\n- **Async Support**: Complete async support across all fetchers and dedicated async session classes.\n\n### Adaptive Scraping & AI Integration\n- \ud83d\udd04 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.\n- \ud83c\udfaf **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. \n- \ud83d\udd0d **Find Similar Elements**: Automatically locate elements similar to found elements.\n- \ud83e\udd16 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))\n\n### High-Performance & battle-tested Architecture\n- \ud83d\ude80 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.\n- \ud83d\udd0b **Memory Efficient**: Optimized data structures and lazy loading for a minimal memory footprint.\n- \u26a1 **Fast JSON Serialization**: 10x faster than the standard library.\n- \ud83c\udfd7\ufe0f **Battle tested**: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.\n\n### Developer/Web Scraper Friendly Experience\n- \ud83c\udfaf **Interactive Web Scraping Shell**: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.\n- \ud83d\ude80 **Use it directly from the Terminal**: Optionally, you can use Scrapling to scrape a URL without writing a single code!\n- \ud83d\udee0\ufe0f **Rich Navigation API**: Advanced DOM traversal with parent, sibling, and child navigation methods.\n- \ud83e\uddec **Enhanced Text Processing**: Built-in regex, cleaning methods, and optimized string operations.\n- \ud83d\udcdd **Auto Selector Generation**: Generate robust CSS/XPath selectors for any element.\n- \ud83d\udd0c **Familiar API**: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.\n- \ud83d\udcd8 **Complete Type Coverage**: Full type hints for excellent IDE support and code completion.\n\n### New Session Architecture\nScrapling 0.3 introduces a completely revamped session system:\n- **Persistent Sessions**: Maintain cookies, headers, and authentication across multiple requests\n- **Automatic Session Management**: Smart session lifecycle handling with proper cleanup\n- **Session Inheritance**: All fetchers support both one-off requests and persistent session usage\n- **Concurrent Session Support**: Run multiple isolated sessions simultaneously\n\n## Getting Started\n\n### Basic Usage\n```python\nfrom scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher\nfrom scrapling.fetchers import FetcherSession, StealthySession, DynamicSession\n\n# HTTP requests with session support\nwith FetcherSession(impersonate='chrome') as session: # Use latest version of Chrome's TLS fingerprint\n page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)\n quotes = page.css('.quote .text::text')\n\n# Or use one-off requests\npage = Fetcher.get('https://quotes.toscrape.com/')\nquotes = page.css('.quote .text::text')\n\n# Advanced stealth mode (Keep the browser open until you finish)\nwith StealthySession(headless=True, solve_cloudflare=True) as session:\n page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)\n data = page.css('#padded_content a')\n\n# Or use one-off request style, it opens the browser for this request, then closes it after finishing\npage = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')\ndata = page.css('#padded_content a')\n \n# Full browser automation (Keep the browser open until you finish)\nwith DynamicSession(headless=True, disable_resources=False, network_idle=True) as session:\n page = session.fetch('https://quotes.toscrape.com/', load_dom=False)\n data = page.xpath('//span[@class=\"text\"]/text()') # XPath selector if you prefer it\n\n# Or use one-off request style, it opens the browser for this request, then closes it after finishing\npage = DynamicFetcher.fetch('https://quotes.toscrape.com/')\ndata = page.css('.quote .text::text')\n```\n\n### Advanced Parsing & Navigation\n```python\nfrom scrapling.fetchers import Fetcher\n\n# Rich element selection and navigation\npage = Fetcher.get('https://quotes.toscrape.com/')\n\n# Get quotes with multiple selection methods\nquotes = page.css('.quote') # CSS selector\nquotes = page.xpath('//div[@class=\"quote\"]') # XPath\nquotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup-style\n# Same as\nquotes = page.find_all('div', class_='quote')\nquotes = page.find_all(['div'], class_='quote')\nquotes = page.find_all(class_='quote') # and so on...\n# Find element by text content\nquotes = page.find_by_text('quote', tag='div')\n\n# Advanced navigation\nfirst_quote = page.css_first('.quote')\nquote_text = first_quote.css('.text::text')\nquote_text = page.css('.quote').css_first('.text::text') # Chained selectors\nquote_text = page.css_first('.quote .text').text # Using `css_first` is faster than `css` if you want the first element\nauthor = first_quote.next_sibling.css('.author::text')\nparent_container = first_quote.parent\n\n# Element relationships and similarity\nsimilar_elements = first_quote.find_similar()\nbelow_elements = first_quote.below_elements()\n```\nYou can use the parser right away if you don't want to fetch websites like below:\n```python\nfrom scrapling.parser import Selector\n\npage = Selector(\"<html>...</html>\")\n```\nAnd it works precisely the same way!\n\n### Async Session Management Examples\n```python\nimport asyncio\nfrom scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession\n\nasync with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns\n page1 = session.get('https://quotes.toscrape.com/')\n page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')\n\n# Async session usage\nasync with AsyncStealthySession(max_pages=2) as session:\n tasks = []\n urls = ['https://example.com/page1', 'https://example.com/page2']\n \n for url in urls:\n task = session.fetch(url)\n tasks.append(task)\n \n print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)\n results = await asyncio.gather(*tasks)\n print(session.get_pool_stats())\n```\n\n## CLI & Interactive Shell\n\nScrapling v0.3 includes a powerful command-line interface:\n\n[](https://asciinema.org/a/736339)\n\n```bash\n# Launch interactive Web Scraping shell\nscrapling shell\n\n# Extract pages to a file directly without programming (Extracts the content inside `body` tag by default)\n# If the output file ends with `.txt`, then the text content of the target will be extracted.\n# If ended with `.md`, it will be a markdown representation of the HTML content, and `.html` will be the HTML content right away.\nscrapling extract get 'https://example.com' content.md\nscrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'\nscrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless\nscrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare\n```\n\n> [!NOTE]\n> There are many additional features, but we want to keep this page short, like the MCP server and the interactive Web Scraping Shell. Check out the full documentation [here](https://scrapling.readthedocs.io/en/latest/)\n\n## Performance Benchmarks\n\nScrapling isn't just powerful\u2014it's also blazing fast, and the updates since version 0.3 deliver exceptional performance improvements across all operations!\n\n### Text Extraction Speed Test (5000 nested elements)\n\n| # | Library | Time (ms) | vs Scrapling | \n|---|:-----------------:|:---------:|:------------:|\n| 1 | Scrapling | 1.92 | 1.0x |\n| 2 | Parsel/Scrapy | 1.99 | 1.036x |\n| 3 | Raw Lxml | 2.33 | 1.214x |\n| 4 | PyQuery | 20.61 | ~11x |\n| 5 | Selectolax | 80.65 | ~42x |\n| 6 | BS4 with Lxml | 1283.21 | ~698x |\n| 7 | MechanicalSoup | 1304.57 | ~679x |\n| 8 | BS4 with html5lib | 3331.96 | ~1735x |\n\n### Element Similarity & Text Search Performance\n\nScrapling's adaptive element finding capabilities significantly outperform alternatives:\n\n| Library | Time (ms) | vs Scrapling |\n|-------------|:---------:|:------------:|\n| Scrapling | 1.87 | 1.0x |\n| AutoScraper | 10.24 | 5.476x |\n\n\n> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.\n\n## Installation\n\nScrapling requires Python 3.10 or higher:\n\n```bash\npip install scrapling\n```\n\nStarting with v0.3.2, this installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies.\n\n### Optional Dependencies\n\n1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with\n ```bash\n pip install \"scrapling[fetchers]\"\n \n scrapling install\n ```\n\n This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.\n\n2. Extra features:\n - Install the MCP server feature:\n ```bash\n pip install \"scrapling[ai]\"\n ```\n - Install shell features (Web Scraping shell and the `extract` command): \n ```bash\n pip install \"scrapling[shell]\"\n ```\n - Install everything: \n ```bash\n pip install \"scrapling[all]\"\n ```\n Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)\n\n## Contributing\n\nWe welcome contributions! Please read our [contributing guidelines](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before getting started.\n\n## Disclaimer\n\n> [!CAUTION]\n> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect website terms of service and robots.txt files.\n\n## License\n\nThis work is licensed under the BSD-3-Clause License.\n\n## Acknowledgments\n\nThis project includes code adapted from:\n- Parsel (BSD License)\u2014Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule\n\n## Thanks and References\n\n- [Daijro](https://github.com/daijro)'s brilliant work on [BrowserForge](https://github.com/daijro/browserforge) and [Camoufox](https://github.com/daijro/camoufox)\n- [Vinyzu](https://github.com/Vinyzu)'s work on [Botright](https://github.com/Vinyzu/Botright)\n- [brotector](https://github.com/kaliiiiiiiiii/brotector) for browser detection bypass techniques\n- [fakebrowser](https://github.com/kkoooqq/fakebrowser) for fingerprinting research\n- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches) for stealth improvements\n\n---\n<div align=\"center\"><small>Designed & crafted with \u2764\ufe0f by Karim Shoair.</small></div><br>\n",
"bugtrack_url": null,
"license": "BSD 3-Clause License\n \n Copyright (c) 2024, Karim shoair\n \n Redistribution and use in source and binary forms, with or without\n modification, are permitted provided that the following conditions are met:\n \n 1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n \n 2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n \n 3. Neither the name of the copyright holder nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n \n THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\n FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\n OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n ",
"summary": "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!",
"version": "0.3.2",
"project_urls": {
"Bug Tracker": "https://github.com/D4Vinci/Scrapling/issues",
"Documentation": "https://scrapling.readthedocs.io/en/latest/",
"Homepage": "https://github.com/D4Vinci/Scrapling",
"Repository": "https://github.com/D4Vinci/Scrapling"
},
"split_keywords": [
"web-scraping",
" scraping",
" automation",
" browser-automation",
" data-extraction",
" html-parsing",
" undetectable",
" playwright",
" selenium-alternative",
" web-crawler",
" browser",
" crawling"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ae4aa44848b82ca169710c4a944ba7daa49ce0f5d394a3bd8d5afc9bd5c30d24",
"md5": "cb96ea6bf70340250595de0552ac7d47",
"sha256": "ac8bf1b62a11de02dbe04973f8b8900b3f36feee9013d6278b4ededab7f8fbfa"
},
"downloads": -1,
"filename": "scrapling-0.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cb96ea6bf70340250595de0552ac7d47",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 100353,
"upload_time": "2025-09-15T01:29:30",
"upload_time_iso_8601": "2025-09-15T01:29:30.312829Z",
"url": "https://files.pythonhosted.org/packages/ae/4a/a44848b82ca169710c4a944ba7daa49ce0f5d394a3bd8d5afc9bd5c30d24/scrapling-0.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "9213b4576b51f31c26c98a87dfacf0c0cee3de83c6106d8b92dde3fbe9432fd2",
"md5": "4a420ed476a47f646c4df5aeaef9f2cb",
"sha256": "27ba852523d5cc53a96500bba8ac6a459d7928a6a38f805f7c81a3928bf1a5ae"
},
"downloads": -1,
"filename": "scrapling-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "4a420ed476a47f646c4df5aeaef9f2cb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 94218,
"upload_time": "2025-09-15T01:29:32",
"upload_time_iso_8601": "2025-09-15T01:29:32.072472Z",
"url": "https://files.pythonhosted.org/packages/92/13/b4576b51f31c26c98a87dfacf0c0cee3de83c6106d8b92dde3fbe9432fd2/scrapling-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-15 01:29:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "D4Vinci",
"github_project": "Scrapling",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "scrapling"
}
