# 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.
<div align="center">
<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
[](https://github.com/unclecode/crawl4ai/stargazers)
[](https://github.com/unclecode/crawl4ai/network/members)
[](https://badge.fury.io/py/crawl4ai)
[](https://pypi.org/project/crawl4ai/)
[](https://pepy.tech/project/crawl4ai)
<p align="center">
<a href="https://x.com/crawl4ai">
<img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow on X" />
</a>
<a href="https://www.linkedin.com/company/crawl4ai">
<img src="https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" alt="Follow on LinkedIn" />
</a>
<a href="https://discord.gg/jP8KfhDhyN">
<img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join our Discord" />
</a>
</p>
</div>
Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.
[✨ Check out latest update v0.7.0](#-recent-updates)
🎉 **Version 0.7.0 is now available!** The Adaptive Intelligence Update introduces groundbreaking features: Adaptive Crawling that learns website patterns, Virtual Scroll support for infinite pages, intelligent Link Preview with 3-layer scoring, Async URL Seeder for massive discovery, and significant performance improvements. [Read the release notes →](https://docs.crawl4ai.com/blog/release-v0.7.0)
<details>
<summary>🤓 <strong>My Personal Story</strong></summary>
My journey with computers started in childhood when my dad, a computer scientist, introduced me to an Amstrad computer. Those early days sparked a fascination with technology, leading me to pursue computer science and specialize in NLP during my postgraduate studies. It was during this time that I first delved into web crawling, building tools to help researchers organize papers and extract information from publications a challenging yet rewarding experience that honed my skills in data extraction.
Fast forward to 2023, I was working on a tool for a project and needed a crawler to convert a webpage into markdown. While exploring solutions, I found one that claimed to be open-source but required creating an account and generating an API token. Worse, it turned out to be a SaaS model charging $16, and its quality didn’t meet my standards. Frustrated, I realized this was a deeper problem. That frustration turned into turbo anger mode, and I decided to build my own solution. In just a few days, I created Crawl4AI. To my surprise, it went viral, earning thousands of GitHub stars and resonating with a global community.
I made Crawl4AI open-source for two reasons. First, it’s my way of giving back to the open-source community that has supported me throughout my career. Second, I believe data should be accessible to everyone, not locked behind paywalls or monopolized by a few. Open access to data lays the foundation for the democratization of AI, a vision where individuals can train their own models and take ownership of their information. This library is the first step in a larger journey to create the best open-source data extraction and generation tool the world has ever seen, built collaboratively by a passionate community.
Thank you to everyone who has supported this project, used it, and shared feedback. Your encouragement motivates me to dream even bigger. Join us, file issues, submit PRs, or spread the word. Together, we can build a tool that truly empowers people to access their own data and reshape the future of AI.
</details>
## 🧐 Why Crawl4AI?
1. **Built for LLMs**: Creates smart, concise Markdown optimized for RAG and fine-tuning applications.
2. **Lightning Fast**: Delivers results 6x faster with real-time, cost-efficient performance.
3. **Flexible Browser Control**: Offers session management, proxies, and custom hooks for seamless data access.
4. **Heuristic Intelligence**: Uses advanced algorithms for efficient extraction, reducing reliance on costly models.
5. **Open Source & Deployable**: Fully open-source with no API keys—ready for Docker and cloud integration.
6. **Thriving Community**: Actively maintained by a vibrant community and the #1 trending GitHub repository.
## 🚀 Quick Start
1. Install Crawl4AI:
```bash
# Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctor
```
If you encounter any browser-related issues, you can install them manually:
```bash
python -m playwright install --with-deps chromium
```
2. Run a simple web crawl with Python:
```python
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())
```
3. Or use the new command-line interface:
```bash
# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown
# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"
```
## ✨ Features
<details>
<summary>📝 <strong>Markdown Generation</strong></summary>
- 🧹 **Clean Markdown**: Generates clean, structured Markdown with accurate formatting.
- 🎯 **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
- 🔗 **Citations and References**: Converts page links into a numbered reference list with clean citations.
- 🛠️ **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs.
- 📚 **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content.
</details>
<details>
<summary>📊 <strong>Structured Data Extraction</strong></summary>
- 🤖 **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction.
- 🧱 **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
- 🌌 **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction.
- 🔎 **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors.
- 🔧 **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.
</details>
<details>
<summary>🌐 <strong>Browser Integration</strong></summary>
- 🖥️ **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection.
- 🔄 **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
- 👤 **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings.
- 🔒 **Session Management**: Preserve browser states and reuse them for multi-step crawling.
- 🧩 **Proxy Support**: Seamlessly connect to proxies with authentication for secure access.
- ⚙️ **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups.
- 🌍 **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit.
- 📐 **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.
</details>
<details>
<summary>🔎 <strong>Crawling & Scraping</strong></summary>
- 🖼️ **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`.
- 🚀 **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction.
- 📸 **Screenshots**: Capture page screenshots during crawling for debugging or analysis.
- 📂 **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`).
- 🔗 **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content.
- 🛠️ **Customizable Hooks**: Define hooks at every step to customize crawling behavior.
- 💾 **Caching**: Cache data for improved speed and to avoid redundant fetches.
- 📄 **Metadata Extraction**: Retrieve structured metadata from web pages.
- 📡 **IFrame Content Extraction**: Seamless extraction from embedded iframe content.
- 🕵️ **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading.
- 🔄 **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.
</details>
<details>
<summary>🚀 <strong>Deployment</strong></summary>
- 🐳 **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment.
- 🔑 **Secure Authentication**: Built-in JWT token authentication for API security.
- 🔄 **API Gateway**: One-click deployment with secure token authentication for API-based workflows.
- 🌐 **Scalable Architecture**: Designed for mass-scale production and optimized server performance.
- ☁️ **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms.
</details>
<details>
<summary>🎯 <strong>Additional Features</strong></summary>
- 🕶️ **Stealth Mode**: Avoid bot detection by mimicking real users.
- 🏷️ **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata.
- 🔗 **Link Analysis**: Extract and analyze all links for detailed data exploration.
- 🛡️ **Error Handling**: Robust error management for seamless execution.
- 🔐 **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests.
- 📖 **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage.
- 🙌 **Community Recognition**: Acknowledges contributors and pull requests for transparency.
</details>
## Try it Now!
✨ Play around with this [](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
✨ Visit our [Documentation Website](https://docs.crawl4ai.com/)
## Installation 🛠️
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
<details>
<summary>🐍 <strong>Using pip</strong></summary>
Choose the installation option that best fits your needs:
### Basic Installation
For basic web crawling and scraping tasks:
```bash
pip install crawl4ai
crawl4ai-setup # Setup the browser
```
By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.
👉 **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:
1. Through the command line:
```bash
playwright install
```
2. If the above doesn't work, try this more specific command:
```bash
python -m playwright install chromium
```
This second method has proven to be more reliable in some cases.
---
### Installation with Synchronous Version
The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:
```bash
pip install crawl4ai[sync]
```
---
### Development Installation
For contributors who plan to modify the source code:
```bash
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e . # Basic installation in editable mode
```
Install optional features:
```bash
pip install -e ".[torch]" # With PyTorch features
pip install -e ".[transformer]" # With Transformer features
pip install -e ".[cosine]" # With cosine similarity features
pip install -e ".[sync]" # With synchronous crawling (Selenium)
pip install -e ".[all]" # Install all optional features
```
</details>
<details>
<summary>🐳 <strong>Docker Deployment</strong></summary>
> 🚀 **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.
### New Docker Features
The new Docker implementation includes:
- **Browser pooling** with page pre-warming for faster response times
- **Interactive playground** to test and generate request code
- **MCP integration** for direct connection to AI tools like Claude Code
- **Comprehensive API endpoints** including HTML extraction, screenshots, PDF generation, and JavaScript execution
- **Multi-architecture support** with automatic detection (AMD64/ARM64)
- **Optimized resources** with improved memory management
### Getting Started
```bash
# Pull and run the latest release candidate
docker pull unclecode/crawl4ai:0.7.0
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.7.0
# Visit the playground at http://localhost:11235/playground
```
For complete documentation, see our [Docker Deployment Guide](https://docs.crawl4ai.com/core/docker-deployment/).
</details>
---
### Quick Test
Run a quick test (works for both Docker options):
```python
import requests
# Submit a crawl job
response = requests.post(
"http://localhost:11235/crawl",
json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
print("Crawl job submitted successfully.")
if "results" in response.json():
results = response.json()["results"]
print("Crawl job completed. Results:")
for result in results:
print(result)
else:
task_id = response.json()["task_id"]
print(f"Crawl job submitted. Task ID:: {task_id}")
result = requests.get(f"http://localhost:11235/task/{task_id}")
```
For more examples, see our [Docker Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_example.py). For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://docs.crawl4ai.com/basic/docker-deployment/).
</details>
## 🔬 Advanced Usage Examples 🔬
You can check the project structure in the directory [https://github.com/unclecode/crawl4ai/docs/examples](docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
<details>
<summary>📝 <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>
```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
async def main():
browser_config = BrowserConfig(
headless=True,
verbose=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.ENABLED,
markdown_generator=DefaultMarkdownGenerator(
content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
),
# markdown_generator=DefaultMarkdownGenerator(
# content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
# ),
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url="https://docs.micronaut.io/4.7.6/guide/",
config=run_config
)
print(len(result.markdown.raw_markdown))
print(len(result.markdown.fit_markdown))
if __name__ == "__main__":
asyncio.run(main())
```
</details>
<details>
<summary>🖥️ <strong>Executing JavaScript & Extract Structured Data without LLMs</strong></summary>
```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy
import json
async def main():
schema = {
"name": "KidoCode Courses",
"baseSelector": "section.charge-methodology .w-tab-content > div",
"fields": [
{
"name": "section_title",
"selector": "h3.heading-50",
"type": "text",
},
{
"name": "section_description",
"selector": ".charge-content",
"type": "text",
},
{
"name": "course_name",
"selector": ".text-block-93",
"type": "text",
},
{
"name": "course_description",
"selector": ".course-content-text",
"type": "text",
},
{
"name": "course_icon",
"selector": ".image-92",
"type": "attribute",
"attribute": "src"
}
}
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
browser_config = BrowserConfig(
headless=False,
verbose=True
)
run_config = CrawlerRunConfig(
extraction_strategy=extraction_strategy,
js_code=["""(async () => {const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();"""],
cache_mode=CacheMode.BYPASS
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url="https://www.kidocode.com/degrees/technology",
config=run_config
)
companies = json.loads(result.extracted_content)
print(f"Successfully extracted {len(companies)} companies")
print(json.dumps(companies[0], indent=2))
if __name__ == "__main__":
asyncio.run(main())
```
</details>
<details>
<summary>📚 <strong>Extracting Structured Data with LLMs</strong></summary>
```python
import os
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field
class OpenAIModelFee(BaseModel):
model_name: str = Field(..., description="Name of the OpenAI model.")
input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")
async def main():
browser_config = BrowserConfig(verbose=True)
run_config = CrawlerRunConfig(
word_count_threshold=1,
extraction_strategy=LLMExtractionStrategy(
# Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
# provider="ollama/qwen2", api_token="no-token",
llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')),
schema=OpenAIModelFee.schema(),
extraction_type="schema",
instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
Do not miss any models in the entire content. One extracted model JSON format should look like this:
{"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
),
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(
url='https://openai.com/api/pricing/',
config=run_config
)
print(result.extracted_content)
if __name__ == "__main__":
asyncio.run(main())
```
</details>
<details>
<summary>🤖 <strong>Using You own Browser with Custom User Profile</strong></summary>
```python
import os, sys
from pathlib import Path
import asyncio, time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def test_news_crawl():
# Create a persistent user data directory
user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
os.makedirs(user_data_dir, exist_ok=True)
browser_config = BrowserConfig(
verbose=True,
headless=True,
user_data_dir=user_data_dir,
use_persistent_context=True,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS
)
async with AsyncWebCrawler(config=browser_config) as crawler:
url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
result = await crawler.arun(
url,
config=run_config,
magic=True,
)
print(f"Successfully crawled {url}")
print(f"Content length: {len(result.markdown)}")
```
</details>
## ✨ Recent Updates
### Version 0.7.0 Release Highlights - The Adaptive Intelligence Update
- **🧠 Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically:
```python
config = AdaptiveConfig(
confidence_threshold=0.7,
max_history=100,
learning_rate=0.2
)
result = await crawler.arun(
"https://news.example.com",
config=CrawlerRunConfig(adaptive_config=config)
)
# Crawler learns patterns and improves extraction over time
```
- **🌊 Virtual Scroll Support**: Complete content extraction from infinite scroll pages:
```python
scroll_config = VirtualScrollConfig(
container_selector="[data-testid='feed']",
scroll_count=20,
scroll_by="container_height",
wait_after_scroll=1.0
)
result = await crawler.arun(url, config=CrawlerRunConfig(
virtual_scroll_config=scroll_config
))
```
- **🔗 Intelligent Link Analysis**: 3-layer scoring system for smart link prioritization:
```python
link_config = LinkPreviewConfig(
query="machine learning tutorials",
score_threshold=0.3,
concurrent_requests=10
)
result = await crawler.arun(url, config=CrawlerRunConfig(
link_preview_config=link_config,
score_links=True
))
# Links ranked by relevance and quality
```
- **🎣 Async URL Seeder**: Discover thousands of URLs in seconds:
```python
seeder = AsyncUrlSeeder(SeedingConfig(
source="sitemap+cc",
pattern="*/blog/*",
query="python tutorials",
score_threshold=0.4
))
urls = await seeder.discover("https://example.com")
```
- **⚡ Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency
Read the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blog/release-v0.7.0) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).
### Previous Version: 0.6.0 Release Highlights
- **🌎 World-aware Crawling**: Set geolocation, language, and timezone for authentic locale-specific content:
```python
crun_cfg = CrawlerRunConfig(
url="https://browserleaks.com/geo", # test page that shows your location
locale="en-US", # Accept-Language & UI locale
timezone_id="America/Los_Angeles", # JS Date()/Intl timezone
geolocation=GeolocationConfig( # override GPS coords
latitude=34.0522,
longitude=-118.2437,
accuracy=10.0,
)
)
```
- **📊 Table-to-DataFrame Extraction**: Extract HTML tables directly to CSV or pandas DataFrames:
```python
crawler = AsyncWebCrawler(config=browser_config)
await crawler.start()
try:
# Set up scraping parameters
crawl_config = CrawlerRunConfig(
table_score_threshold=8, # Strict table detection
)
# Execute market data extraction
results: List[CrawlResult] = await crawler.arun(
url="https://coinmarketcap.com/?page=1", config=crawl_config
)
# Process results
raw_df = pd.DataFrame()
for result in results:
if result.success and result.media["tables"]:
raw_df = pd.DataFrame(
result.media["tables"][0]["rows"],
columns=result.media["tables"][0]["headers"],
)
break
print(raw_df.head())
finally:
await crawler.stop()
```
- **🚀 Browser Pooling**: Pages launch hot with pre-warmed browser instances for lower latency and memory usage
- **🕸️ Network and Console Capture**: Full traffic logs and MHTML snapshots for debugging:
```python
crawler_config = CrawlerRunConfig(
capture_network=True,
capture_console=True,
mhtml=True
)
```
- **🔌 MCP Integration**: Connect to AI tools like Claude Code through the Model Context Protocol
```bash
# Add Crawl4AI to Claude Code
claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse
```
- **🖥️ Interactive Playground**: Test configurations and generate API requests with the built-in web interface at `http://localhost:11235//playground`
- **🐳 Revamped Docker Deployment**: Streamlined multi-architecture Docker image with improved resource efficiency
- **📱 Multi-stage Build System**: Optimized Dockerfile with platform-specific performance enhancements
### Previous Version: 0.5.0 Major Release Highlights
- **🚀 Deep Crawling System**: Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies
- **⚡ Memory-Adaptive Dispatcher**: Dynamically adjusts concurrency based on system memory
- **🔄 Multiple Crawling Strategies**: Browser-based and lightweight HTTP-only crawlers
- **💻 Command-Line Interface**: New `crwl` CLI provides convenient terminal access
- **👤 Browser Profiler**: Create and manage persistent browser profiles
- **🧠 Crawl4AI Coding Assistant**: AI-powered coding assistant
- **🏎️ LXML Scraping Mode**: Fast HTML parsing using the `lxml` library
- **🌐 Proxy Rotation**: Built-in support for proxy switching
- **🤖 LLM Content Filter**: Intelligent markdown generation using LLMs
- **📄 PDF Processing**: Extract text, images, and metadata from PDF files
Read the full details in our [0.5.0 Release Notes](https://docs.crawl4ai.com/blog/releases/0.5.0.html).
## Version Numbering in Crawl4AI
Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
### Version Numbers Explained
Our version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3)
#### Pre-release Versions
We use different suffixes to indicate development stages:
- `dev` (0.4.3dev1): Development versions, unstable
- `a` (0.4.3a1): Alpha releases, experimental features
- `b` (0.4.3b1): Beta releases, feature complete but needs testing
- `rc` (0.4.3): Release candidates, potential final version
#### Installation
- Regular installation (stable version):
```bash
pip install -U crawl4ai
```
- Install pre-release versions:
```bash
pip install crawl4ai --pre
```
- Install specific version:
```bash
pip install crawl4ai==0.4.3b1
```
#### Why Pre-releases?
We use pre-releases to:
- Test new features in real-world scenarios
- Gather feedback before final releases
- Ensure stability for production users
- Allow early adopters to try new features
For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the `--pre` flag.
## 📖 Documentation & Roadmap
> 🚨 **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
For current documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://docs.crawl4ai.com/).
To check our development plans and upcoming features, visit our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).
<details>
<summary>📈 <strong>Development TODOs</strong></summary>
- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas
- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content
- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
- [ ] 8. Performance Monitor: Real-time insights into crawler operations
- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
- [ ] 10. Sponsorship Program: Structured support system with tiered benefits
- [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials
</details>
## 🤝 Contributing
We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.
I'll help modify the license section with badges. For the halftone effect, here's a version with it:
Here's the updated license section:
## 📄 License & Attribution
This project is licensed under the Apache License 2.0 with a required attribution clause. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
### Attribution Requirements
When using Crawl4AI, you must include one of the following attribution methods:
#### 1. Badge Attribution (Recommended)
Add one of these badges to your README, documentation, or website:
| Theme | Badge |
|-------|-------|
| **Disco Theme (Animated)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/></a> |
| **Night Theme (Dark with Neon)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/></a> |
| **Dark Theme (Classic)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/></a> |
| **Light Theme (Classic)** | <a href="https://github.com/unclecode/crawl4ai"><img src="./docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/></a> |
HTML code for adding the badges:
```html
<!-- Disco Theme (Animated) -->
<a href="https://github.com/unclecode/crawl4ai">
<img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/>
</a>
<!-- Night Theme (Dark with Neon) -->
<a href="https://github.com/unclecode/crawl4ai">
<img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/>
</a>
<!-- Dark Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
<img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/>
</a>
<!-- Light Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
<img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/>
</a>
<!-- Simple Shield Badge -->
<a href="https://github.com/unclecode/crawl4ai">
<img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/>
</a>
```
#### 2. Text Attribution
Add this line to your documentation:
```
This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
```
## 📚 Citation
If you use Crawl4AI in your research or project, please cite:
```bibtex
@software{crawl4ai2024,
author = {UncleCode},
title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
year = {2024},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/unclecode/crawl4ai}},
commit = {Please use the commit hash you're working with}
}
```
Text citation format:
```
UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software].
GitHub. https://github.com/unclecode/crawl4ai
```
## 📧 Contact
For questions, suggestions, or feedback, feel free to reach out:
- GitHub: [unclecode](https://github.com/unclecode)
- Twitter: [@unclecode](https://twitter.com/unclecode)
- Website: [crawl4ai.com](https://crawl4ai.com)
Happy Crawling! 🕸️🚀
## 🗾 Mission
Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.
We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
<details>
<summary>🔑 <strong>Key Opportunities</strong></summary>
- **Data Capitalization**: Transform digital footprints into measurable, valuable assets.
- **Authentic AI Data**: Provide AI systems with real human insights.
- **Shared Economy**: Create a fair data marketplace that benefits data creators.
</details>
<details>
<summary>🚀 <strong>Development Pathway</strong></summary>
1. **Open-Source Tools**: Community-driven platforms for transparent data extraction.
2. **Digital Asset Structuring**: Tools to organize and value digital knowledge.
3. **Ethical Data Marketplace**: A secure, fair platform for exchanging structured data.
For more details, see our [full mission statement](./MISSION.md).
</details>
## Star History
[](https://star-history.com/#unclecode/crawl4ai&Date)
Raw data
{
"_id": null,
"home_page": "https://github.com/unclecode/crawl4ai",
"name": "Crawl4AI",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Unclecode",
"author_email": "Unclecode <unclecode@kidocode.com>",
"download_url": "https://files.pythonhosted.org/packages/ff/cc/d6f48c4b01719b1d8f5f1ad8b1f67569d59db8f566448a8747814bd70e60/crawl4ai-0.7.0.tar.gz",
"platform": null,
"description": "# \ud83d\ude80\ud83e\udd16 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.\n\n<div align=\"center\">\n\n<a href=\"https://trendshift.io/repositories/11716\" target=\"_blank\"><img src=\"https://trendshift.io/api/badge/repositories/11716\" alt=\"unclecode%2Fcrawl4ai | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/></a>\n\n[](https://github.com/unclecode/crawl4ai/stargazers)\n[](https://github.com/unclecode/crawl4ai/network/members)\n\n[](https://badge.fury.io/py/crawl4ai)\n[](https://pypi.org/project/crawl4ai/)\n[](https://pepy.tech/project/crawl4ai)\n\n<p align=\"center\">\n <a href=\"https://x.com/crawl4ai\">\n <img src=\"https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white\" alt=\"Follow on X\" />\n </a>\n <a href=\"https://www.linkedin.com/company/crawl4ai\">\n <img src=\"https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\" alt=\"Follow on LinkedIn\" />\n </a>\n <a href=\"https://discord.gg/jP8KfhDhyN\">\n <img src=\"https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white\" alt=\"Join our Discord\" />\n </a>\n </p>\n</div>\n\nCrawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease. \n\n[\u2728 Check out latest update v0.7.0](#-recent-updates)\n\n\ud83c\udf89 **Version 0.7.0 is now available!** The Adaptive Intelligence Update introduces groundbreaking features: Adaptive Crawling that learns website patterns, Virtual Scroll support for infinite pages, intelligent Link Preview with 3-layer scoring, Async URL Seeder for massive discovery, and significant performance improvements. [Read the release notes \u2192](https://docs.crawl4ai.com/blog/release-v0.7.0)\n\n<details>\n<summary>\ud83e\udd13 <strong>My Personal Story</strong></summary>\n\nMy journey with computers started in childhood when my dad, a computer scientist, introduced me to an Amstrad computer. Those early days sparked a fascination with technology, leading me to pursue computer science and specialize in NLP during my postgraduate studies. It was during this time that I first delved into web crawling, building tools to help researchers organize papers and extract information from publications a challenging yet rewarding experience that honed my skills in data extraction.\n\nFast forward to 2023, I was working on a tool for a project and needed a crawler to convert a webpage into markdown. While exploring solutions, I found one that claimed to be open-source but required creating an account and generating an API token. Worse, it turned out to be a SaaS model charging $16, and its quality didn\u2019t meet my standards. Frustrated, I realized this was a deeper problem. That frustration turned into turbo anger mode, and I decided to build my own solution. In just a few days, I created Crawl4AI. To my surprise, it went viral, earning thousands of GitHub stars and resonating with a global community.\n\nI made Crawl4AI open-source for two reasons. First, it\u2019s my way of giving back to the open-source community that has supported me throughout my career. Second, I believe data should be accessible to everyone, not locked behind paywalls or monopolized by a few. Open access to data lays the foundation for the democratization of AI, a vision where individuals can train their own models and take ownership of their information. This library is the first step in a larger journey to create the best open-source data extraction and generation tool the world has ever seen, built collaboratively by a passionate community.\n\nThank you to everyone who has supported this project, used it, and shared feedback. Your encouragement motivates me to dream even bigger. Join us, file issues, submit PRs, or spread the word. Together, we can build a tool that truly empowers people to access their own data and reshape the future of AI.\n</details>\n\n## \ud83e\uddd0 Why Crawl4AI?\n\n1. **Built for LLMs**: Creates smart, concise Markdown optimized for RAG and fine-tuning applications. \n2. **Lightning Fast**: Delivers results 6x faster with real-time, cost-efficient performance. \n3. **Flexible Browser Control**: Offers session management, proxies, and custom hooks for seamless data access. \n4. **Heuristic Intelligence**: Uses advanced algorithms for efficient extraction, reducing reliance on costly models. \n5. **Open Source & Deployable**: Fully open-source with no API keys\u2014ready for Docker and cloud integration. \n6. **Thriving Community**: Actively maintained by a vibrant community and the #1 trending GitHub repository.\n\n## \ud83d\ude80 Quick Start \n\n1. Install Crawl4AI:\n```bash\n# Install the package\npip install -U crawl4ai\n\n# For pre release versions\npip install crawl4ai --pre\n\n# Run post-installation setup\ncrawl4ai-setup\n\n# Verify your installation\ncrawl4ai-doctor\n```\n\nIf you encounter any browser-related issues, you can install them manually:\n```bash\npython -m playwright install --with-deps chromium\n```\n\n2. Run a simple web crawl with Python:\n```python\nimport asyncio\nfrom crawl4ai import *\n\nasync def main():\n async with AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.nbcnews.com/business\",\n )\n print(result.markdown)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n3. Or use the new command-line interface:\n```bash\n# Basic crawl with markdown output\ncrwl https://www.nbcnews.com/business -o markdown\n\n# Deep crawl with BFS strategy, max 10 pages\ncrwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10\n\n# Use LLM extraction with a specific question\ncrwl https://www.example.com/products -q \"Extract all product prices\"\n```\n\n## \u2728 Features \n\n<details>\n<summary>\ud83d\udcdd <strong>Markdown Generation</strong></summary>\n\n- \ud83e\uddf9 **Clean Markdown**: Generates clean, structured Markdown with accurate formatting.\n- \ud83c\udfaf **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.\n- \ud83d\udd17 **Citations and References**: Converts page links into a numbered reference list with clean citations.\n- \ud83d\udee0\ufe0f **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs.\n- \ud83d\udcda **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content. \n</details>\n\n<details>\n<summary>\ud83d\udcca <strong>Structured Data Extraction</strong></summary>\n\n- \ud83e\udd16 **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction.\n- \ud83e\uddf1 **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.\n- \ud83c\udf0c **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction.\n- \ud83d\udd0e **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors.\n- \ud83d\udd27 **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.\n\n</details>\n\n<details>\n<summary>\ud83c\udf10 <strong>Browser Integration</strong></summary>\n\n- \ud83d\udda5\ufe0f **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection.\n- \ud83d\udd04 **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.\n- \ud83d\udc64 **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings.\n- \ud83d\udd12 **Session Management**: Preserve browser states and reuse them for multi-step crawling.\n- \ud83e\udde9 **Proxy Support**: Seamlessly connect to proxies with authentication for secure access.\n- \u2699\ufe0f **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups.\n- \ud83c\udf0d **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit.\n- \ud83d\udcd0 **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.\n\n</details>\n\n<details>\n<summary>\ud83d\udd0e <strong>Crawling & Scraping</strong></summary>\n\n- \ud83d\uddbc\ufe0f **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`.\n- \ud83d\ude80 **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction.\n- \ud83d\udcf8 **Screenshots**: Capture page screenshots during crawling for debugging or analysis.\n- \ud83d\udcc2 **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`).\n- \ud83d\udd17 **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content.\n- \ud83d\udee0\ufe0f **Customizable Hooks**: Define hooks at every step to customize crawling behavior.\n- \ud83d\udcbe **Caching**: Cache data for improved speed and to avoid redundant fetches.\n- \ud83d\udcc4 **Metadata Extraction**: Retrieve structured metadata from web pages.\n- \ud83d\udce1 **IFrame Content Extraction**: Seamless extraction from embedded iframe content.\n- \ud83d\udd75\ufe0f **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading.\n- \ud83d\udd04 **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.\n\n</details>\n\n<details>\n<summary>\ud83d\ude80 <strong>Deployment</strong></summary>\n\n- \ud83d\udc33 **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment.\n- \ud83d\udd11 **Secure Authentication**: Built-in JWT token authentication for API security.\n- \ud83d\udd04 **API Gateway**: One-click deployment with secure token authentication for API-based workflows.\n- \ud83c\udf10 **Scalable Architecture**: Designed for mass-scale production and optimized server performance.\n- \u2601\ufe0f **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms.\n\n</details>\n\n<details>\n<summary>\ud83c\udfaf <strong>Additional Features</strong></summary>\n\n- \ud83d\udd76\ufe0f **Stealth Mode**: Avoid bot detection by mimicking real users.\n- \ud83c\udff7\ufe0f **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata.\n- \ud83d\udd17 **Link Analysis**: Extract and analyze all links for detailed data exploration.\n- \ud83d\udee1\ufe0f **Error Handling**: Robust error management for seamless execution.\n- \ud83d\udd10 **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests.\n- \ud83d\udcd6 **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage.\n- \ud83d\ude4c **Community Recognition**: Acknowledges contributors and pull requests for transparency.\n\n</details>\n\n## Try it Now!\n\n\u2728 Play around with this [](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)\n\n\u2728 Visit our [Documentation Website](https://docs.crawl4ai.com/)\n\n## Installation \ud83d\udee0\ufe0f\n\nCrawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.\n\n<details>\n<summary>\ud83d\udc0d <strong>Using pip</strong></summary>\n\nChoose the installation option that best fits your needs:\n\n### Basic Installation\n\nFor basic web crawling and scraping tasks:\n\n```bash\npip install crawl4ai\ncrawl4ai-setup # Setup the browser\n```\n\nBy default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.\n\n\ud83d\udc49 **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:\n\n1. Through the command line:\n\n ```bash\n playwright install\n ```\n\n2. If the above doesn't work, try this more specific command:\n\n ```bash\n python -m playwright install chromium\n ```\n\nThis second method has proven to be more reliable in some cases.\n\n---\n\n### Installation with Synchronous Version\n\nThe sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:\n\n```bash\npip install crawl4ai[sync]\n```\n\n---\n\n### Development Installation\n\nFor contributors who plan to modify the source code:\n\n```bash\ngit clone https://github.com/unclecode/crawl4ai.git\ncd crawl4ai\npip install -e . # Basic installation in editable mode\n```\n\nInstall optional features:\n\n```bash\npip install -e \".[torch]\" # With PyTorch features\npip install -e \".[transformer]\" # With Transformer features\npip install -e \".[cosine]\" # With cosine similarity features\npip install -e \".[sync]\" # With synchronous crawling (Selenium)\npip install -e \".[all]\" # Install all optional features\n```\n\n</details>\n\n<details>\n<summary>\ud83d\udc33 <strong>Docker Deployment</strong></summary>\n\n> \ud83d\ude80 **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.\n\n### New Docker Features\n\nThe new Docker implementation includes:\n- **Browser pooling** with page pre-warming for faster response times\n- **Interactive playground** to test and generate request code\n- **MCP integration** for direct connection to AI tools like Claude Code\n- **Comprehensive API endpoints** including HTML extraction, screenshots, PDF generation, and JavaScript execution\n- **Multi-architecture support** with automatic detection (AMD64/ARM64)\n- **Optimized resources** with improved memory management\n\n### Getting Started\n\n```bash\n# Pull and run the latest release candidate\ndocker pull unclecode/crawl4ai:0.7.0\ndocker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:0.7.0\n\n# Visit the playground at http://localhost:11235/playground\n```\n\nFor complete documentation, see our [Docker Deployment Guide](https://docs.crawl4ai.com/core/docker-deployment/).\n\n</details>\n\n---\n\n### Quick Test\n\nRun a quick test (works for both Docker options):\n\n```python\nimport requests\n\n# Submit a crawl job\nresponse = requests.post(\n \"http://localhost:11235/crawl\",\n json={\"urls\": [\"https://example.com\"], \"priority\": 10}\n)\nif response.status_code == 200:\n print(\"Crawl job submitted successfully.\")\n \nif \"results\" in response.json():\n results = response.json()[\"results\"]\n print(\"Crawl job completed. Results:\")\n for result in results:\n print(result)\nelse:\n task_id = response.json()[\"task_id\"]\n print(f\"Crawl job submitted. Task ID:: {task_id}\")\n result = requests.get(f\"http://localhost:11235/task/{task_id}\")\n```\n\nFor more examples, see our [Docker Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_example.py). For advanced configuration, environment variables, and usage examples, see our [Docker Deployment Guide](https://docs.crawl4ai.com/basic/docker-deployment/).\n\n</details>\n\n\n## \ud83d\udd2c Advanced Usage Examples \ud83d\udd2c\n\nYou can check the project structure in the directory [https://github.com/unclecode/crawl4ai/docs/examples](docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.\n\n<details>\n<summary>\ud83d\udcdd <strong>Heuristic Markdown Generation with Clean and Fit Markdown</strong></summary>\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasync def main():\n browser_config = BrowserConfig(\n headless=True, \n verbose=True,\n )\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.48, threshold_type=\"fixed\", min_word_threshold=0)\n ),\n # markdown_generator=DefaultMarkdownGenerator(\n # content_filter=BM25ContentFilter(user_query=\"WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY\", bm25_threshold=1.0)\n # ),\n )\n \n async with AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://docs.micronaut.io/4.7.6/guide/\",\n config=run_config\n )\n print(len(result.markdown.raw_markdown))\n print(len(result.markdown.fit_markdown))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n</details>\n\n<details>\n<summary>\ud83d\udda5\ufe0f <strong>Executing JavaScript & Extract Structured Data without LLMs</strong></summary>\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\nimport json\n\nasync def main():\n schema = {\n \"name\": \"KidoCode Courses\",\n \"baseSelector\": \"section.charge-methodology .w-tab-content > div\",\n \"fields\": [\n {\n \"name\": \"section_title\",\n \"selector\": \"h3.heading-50\",\n \"type\": \"text\",\n },\n {\n \"name\": \"section_description\",\n \"selector\": \".charge-content\",\n \"type\": \"text\",\n },\n {\n \"name\": \"course_name\",\n \"selector\": \".text-block-93\",\n \"type\": \"text\",\n },\n {\n \"name\": \"course_description\",\n \"selector\": \".course-content-text\",\n \"type\": \"text\",\n },\n {\n \"name\": \"course_icon\",\n \"selector\": \".image-92\",\n \"type\": \"attribute\",\n \"attribute\": \"src\"\n }\n }\n}\n\n extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n\n browser_config = BrowserConfig(\n headless=False,\n verbose=True\n )\n run_config = CrawlerRunConfig(\n extraction_strategy=extraction_strategy,\n js_code=[\"\"\"(async () => {const tabs = document.querySelectorAll(\"section.charge-methodology .tabs-menu-3 > div\");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();\"\"\"],\n cache_mode=CacheMode.BYPASS\n )\n \n async with AsyncWebCrawler(config=browser_config) as crawler:\n \n result = await crawler.arun(\n url=\"https://www.kidocode.com/degrees/technology\",\n config=run_config\n )\n\n companies = json.loads(result.extracted_content)\n print(f\"Successfully extracted {len(companies)} companies\")\n print(json.dumps(companies[0], indent=2))\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n</details>\n\n<details>\n<summary>\ud83d\udcda <strong>Extracting Structured Data with LLMs</strong></summary>\n\n```python\nimport os\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\nfrom pydantic import BaseModel, Field\n\nclass OpenAIModelFee(BaseModel):\n model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n\nasync def main():\n browser_config = BrowserConfig(verbose=True)\n run_config = CrawlerRunConfig(\n word_count_threshold=1,\n extraction_strategy=LLMExtractionStrategy(\n # Here you can use any provider that Litellm library supports, for instance: ollama/qwen2\n # provider=\"ollama/qwen2\", api_token=\"no-token\", \n llm_config = LLMConfig(provider=\"openai/gpt-4o\", api_token=os.getenv('OPENAI_API_KEY')), \n schema=OpenAIModelFee.schema(),\n extraction_type=\"schema\",\n instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens. \n Do not miss any models in the entire content. One extracted model JSON format should look like this: \n {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 / 1M tokens\", \"output_fee\": \"US$30.00 / 1M tokens\"}.\"\"\"\n ), \n cache_mode=CacheMode.BYPASS,\n )\n \n async with AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url='https://openai.com/api/pricing/',\n config=run_config\n )\n print(result.extracted_content)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n</details>\n\n<details>\n<summary>\ud83e\udd16 <strong>Using You own Browser with Custom User Profile</strong></summary>\n\n```python\nimport os, sys\nfrom pathlib import Path\nimport asyncio, time\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasync def test_news_crawl():\n # Create a persistent user data directory\n user_data_dir = os.path.join(Path.home(), \".crawl4ai\", \"browser_profile\")\n os.makedirs(user_data_dir, exist_ok=True)\n\n browser_config = BrowserConfig(\n verbose=True,\n headless=True,\n user_data_dir=user_data_dir,\n use_persistent_context=True,\n )\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS\n )\n \n async with AsyncWebCrawler(config=browser_config) as crawler:\n url = \"ADDRESS_OF_A_CHALLENGING_WEBSITE\"\n \n result = await crawler.arun(\n url,\n config=run_config,\n magic=True,\n )\n \n print(f\"Successfully crawled {url}\")\n print(f\"Content length: {len(result.markdown)}\")\n```\n\n</details>\n\n## \u2728 Recent Updates\n\n### Version 0.7.0 Release Highlights - The Adaptive Intelligence Update\n\n- **\ud83e\udde0 Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically:\n ```python\n config = AdaptiveConfig(\n confidence_threshold=0.7,\n max_history=100,\n learning_rate=0.2\n )\n \n result = await crawler.arun(\n \"https://news.example.com\",\n config=CrawlerRunConfig(adaptive_config=config)\n )\n # Crawler learns patterns and improves extraction over time\n ```\n\n- **\ud83c\udf0a Virtual Scroll Support**: Complete content extraction from infinite scroll pages:\n ```python\n scroll_config = VirtualScrollConfig(\n container_selector=\"[data-testid='feed']\",\n scroll_count=20,\n scroll_by=\"container_height\",\n wait_after_scroll=1.0\n )\n \n result = await crawler.arun(url, config=CrawlerRunConfig(\n virtual_scroll_config=scroll_config\n ))\n ```\n\n- **\ud83d\udd17 Intelligent Link Analysis**: 3-layer scoring system for smart link prioritization:\n ```python\n link_config = LinkPreviewConfig(\n query=\"machine learning tutorials\",\n score_threshold=0.3,\n concurrent_requests=10\n )\n \n result = await crawler.arun(url, config=CrawlerRunConfig(\n link_preview_config=link_config,\n score_links=True\n ))\n # Links ranked by relevance and quality\n ```\n\n- **\ud83c\udfa3 Async URL Seeder**: Discover thousands of URLs in seconds:\n ```python\n seeder = AsyncUrlSeeder(SeedingConfig(\n source=\"sitemap+cc\",\n pattern=\"*/blog/*\",\n query=\"python tutorials\",\n score_threshold=0.4\n ))\n \n urls = await seeder.discover(\"https://example.com\")\n ```\n\n- **\u26a1 Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency\n\nRead the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blog/release-v0.7.0) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md).\n\n### Previous Version: 0.6.0 Release Highlights\n\n- **\ud83c\udf0e World-aware Crawling**: Set geolocation, language, and timezone for authentic locale-specific content:\n ```python\n crun_cfg = CrawlerRunConfig(\n url=\"https://browserleaks.com/geo\", # test page that shows your location\n locale=\"en-US\", # Accept-Language & UI locale\n timezone_id=\"America/Los_Angeles\", # JS Date()/Intl timezone\n geolocation=GeolocationConfig( # override GPS coords\n latitude=34.0522,\n longitude=-118.2437,\n accuracy=10.0,\n )\n )\n ```\n\n- **\ud83d\udcca Table-to-DataFrame Extraction**: Extract HTML tables directly to CSV or pandas DataFrames:\n ```python\n crawler = AsyncWebCrawler(config=browser_config)\n await crawler.start()\n\n try:\n # Set up scraping parameters\n crawl_config = CrawlerRunConfig(\n table_score_threshold=8, # Strict table detection\n )\n\n # Execute market data extraction\n results: List[CrawlResult] = await crawler.arun(\n url=\"https://coinmarketcap.com/?page=1\", config=crawl_config\n )\n\n # Process results\n raw_df = pd.DataFrame()\n for result in results:\n if result.success and result.media[\"tables\"]:\n raw_df = pd.DataFrame(\n result.media[\"tables\"][0][\"rows\"],\n columns=result.media[\"tables\"][0][\"headers\"],\n )\n break\n print(raw_df.head())\n\n finally:\n await crawler.stop()\n ```\n\n- **\ud83d\ude80 Browser Pooling**: Pages launch hot with pre-warmed browser instances for lower latency and memory usage\n\n- **\ud83d\udd78\ufe0f Network and Console Capture**: Full traffic logs and MHTML snapshots for debugging:\n ```python\n crawler_config = CrawlerRunConfig(\n capture_network=True,\n capture_console=True,\n mhtml=True\n )\n ```\n\n- **\ud83d\udd0c MCP Integration**: Connect to AI tools like Claude Code through the Model Context Protocol\n ```bash\n # Add Crawl4AI to Claude Code\n claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse\n ```\n\n- **\ud83d\udda5\ufe0f Interactive Playground**: Test configurations and generate API requests with the built-in web interface at `http://localhost:11235//playground`\n\n- **\ud83d\udc33 Revamped Docker Deployment**: Streamlined multi-architecture Docker image with improved resource efficiency\n\n- **\ud83d\udcf1 Multi-stage Build System**: Optimized Dockerfile with platform-specific performance enhancements\n\n\n### Previous Version: 0.5.0 Major Release Highlights\n\n- **\ud83d\ude80 Deep Crawling System**: Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies\n- **\u26a1 Memory-Adaptive Dispatcher**: Dynamically adjusts concurrency based on system memory\n- **\ud83d\udd04 Multiple Crawling Strategies**: Browser-based and lightweight HTTP-only crawlers\n- **\ud83d\udcbb Command-Line Interface**: New `crwl` CLI provides convenient terminal access\n- **\ud83d\udc64 Browser Profiler**: Create and manage persistent browser profiles\n- **\ud83e\udde0 Crawl4AI Coding Assistant**: AI-powered coding assistant\n- **\ud83c\udfce\ufe0f LXML Scraping Mode**: Fast HTML parsing using the `lxml` library\n- **\ud83c\udf10 Proxy Rotation**: Built-in support for proxy switching\n- **\ud83e\udd16 LLM Content Filter**: Intelligent markdown generation using LLMs\n- **\ud83d\udcc4 PDF Processing**: Extract text, images, and metadata from PDF files\n\nRead the full details in our [0.5.0 Release Notes](https://docs.crawl4ai.com/blog/releases/0.5.0.html).\n\n## Version Numbering in Crawl4AI\n\nCrawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.\n\n### Version Numbers Explained\n\nOur version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3)\n\n#### Pre-release Versions\nWe use different suffixes to indicate development stages:\n\n- `dev` (0.4.3dev1): Development versions, unstable\n- `a` (0.4.3a1): Alpha releases, experimental features\n- `b` (0.4.3b1): Beta releases, feature complete but needs testing\n- `rc` (0.4.3): Release candidates, potential final version\n\n#### Installation\n- Regular installation (stable version):\n ```bash\n pip install -U crawl4ai\n ```\n\n- Install pre-release versions:\n ```bash\n pip install crawl4ai --pre\n ```\n\n- Install specific version:\n ```bash\n pip install crawl4ai==0.4.3b1\n ```\n\n#### Why Pre-releases?\nWe use pre-releases to:\n- Test new features in real-world scenarios\n- Gather feedback before final releases\n- Ensure stability for production users\n- Allow early adopters to try new features\n\nFor production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the `--pre` flag.\n\n## \ud83d\udcd6 Documentation & Roadmap \n\n> \ud83d\udea8 **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!\n\nFor current documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://docs.crawl4ai.com/).\n\nTo check our development plans and upcoming features, visit our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).\n\n<details>\n<summary>\ud83d\udcc8 <strong>Development TODOs</strong></summary>\n\n- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction\n- [ ] 1. Question-Based Crawler: Natural language driven web discovery and content extraction\n- [ ] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction\n- [ ] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations\n- [ ] 4. Automated Schema Generator: Convert natural language to extraction schemas\n- [ ] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)\n- [ ] 6. Web Embedding Index: Semantic search infrastructure for crawled content\n- [ ] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance\n- [ ] 8. Performance Monitor: Real-time insights into crawler operations\n- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers\n- [ ] 10. Sponsorship Program: Structured support system with tiered benefits\n- [ ] 11. Educational Content: \"How to Crawl\" video series and interactive tutorials\n\n</details>\n\n## \ud83e\udd1d Contributing \n\nWe welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.\n\nI'll help modify the license section with badges. For the halftone effect, here's a version with it:\n\nHere's the updated license section:\n\n## \ud83d\udcc4 License & Attribution\n\nThis project is licensed under the Apache License 2.0 with a required attribution clause. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.\n\n### Attribution Requirements\nWhen using Crawl4AI, you must include one of the following attribution methods:\n\n#### 1. Badge Attribution (Recommended)\nAdd one of these badges to your README, documentation, or website:\n\n| Theme | Badge |\n|-------|-------|\n| **Disco Theme (Animated)** | <a href=\"https://github.com/unclecode/crawl4ai\"><img src=\"./docs/assets/powered-by-disco.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/></a> |\n| **Night Theme (Dark with Neon)** | <a href=\"https://github.com/unclecode/crawl4ai\"><img src=\"./docs/assets/powered-by-night.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/></a> |\n| **Dark Theme (Classic)** | <a href=\"https://github.com/unclecode/crawl4ai\"><img src=\"./docs/assets/powered-by-dark.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/></a> |\n| **Light Theme (Classic)** | <a href=\"https://github.com/unclecode/crawl4ai\"><img src=\"./docs/assets/powered-by-light.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/></a> |\n \n\nHTML code for adding the badges:\n```html\n<!-- Disco Theme (Animated) -->\n<a href=\"https://github.com/unclecode/crawl4ai\">\n <img src=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/>\n</a>\n\n<!-- Night Theme (Dark with Neon) -->\n<a href=\"https://github.com/unclecode/crawl4ai\">\n <img src=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/>\n</a>\n\n<!-- Dark Theme (Classic) -->\n<a href=\"https://github.com/unclecode/crawl4ai\">\n <img src=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/>\n</a>\n\n<!-- Light Theme (Classic) -->\n<a href=\"https://github.com/unclecode/crawl4ai\">\n <img src=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"/>\n</a>\n\n<!-- Simple Shield Badge -->\n<a href=\"https://github.com/unclecode/crawl4ai\">\n <img src=\"https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square\" alt=\"Powered by Crawl4AI\"/>\n</a>\n```\n\n#### 2. Text Attribution\nAdd this line to your documentation:\n```\nThis project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.\n```\n\n## \ud83d\udcda Citation\n\nIf you use Crawl4AI in your research or project, please cite:\n\n```bibtex\n@software{crawl4ai2024,\n author = {UncleCode},\n title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},\n year = {2024},\n publisher = {GitHub},\n journal = {GitHub Repository},\n howpublished = {\\url{https://github.com/unclecode/crawl4ai}},\n commit = {Please use the commit hash you're working with}\n}\n```\n\nText citation format:\n```\nUncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software]. \nGitHub. https://github.com/unclecode/crawl4ai\n```\n\n## \ud83d\udce7 Contact \n\nFor questions, suggestions, or feedback, feel free to reach out:\n\n- GitHub: [unclecode](https://github.com/unclecode)\n- Twitter: [@unclecode](https://twitter.com/unclecode)\n- Website: [crawl4ai.com](https://crawl4ai.com)\n\nHappy Crawling! \ud83d\udd78\ufe0f\ud83d\ude80\n\n## \ud83d\uddfe Mission\n\nOur mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy. \n\nWe envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.\n\n<details>\n<summary>\ud83d\udd11 <strong>Key Opportunities</strong></summary>\n \n- **Data Capitalization**: Transform digital footprints into measurable, valuable assets. \n- **Authentic AI Data**: Provide AI systems with real human insights. \n- **Shared Economy**: Create a fair data marketplace that benefits data creators. \n\n</details>\n\n<details>\n<summary>\ud83d\ude80 <strong>Development Pathway</strong></summary>\n\n1. **Open-Source Tools**: Community-driven platforms for transparent data extraction. \n2. **Digital Asset Structuring**: Tools to organize and value digital knowledge. \n3. **Ethical Data Marketplace**: A secure, fair platform for exchanging structured data. \n\nFor more details, see our [full mission statement](./MISSION.md).\n</details>\n\n## Star History\n\n[](https://star-history.com/#unclecode/crawl4ai&Date)\n",
"bugtrack_url": null,
"license": null,
"summary": "\ud83d\ude80\ud83e\udd16 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper",
"version": "0.7.0",
"project_urls": {
"Homepage": "https://github.com/unclecode/crawl4ai"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "777593ddcc340b49d8bf6c663b827cc4f46c321cde8fa3923e73d329179603b5",
"md5": "3695cfa42ed1f4e91d854597295951ae",
"sha256": "33a6569555d0c23eb3d4acb2bb7f4c9c77af1c0a03db262530fa8e48c4ea3754"
},
"downloads": -1,
"filename": "crawl4ai-0.7.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3695cfa42ed1f4e91d854597295951ae",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 392072,
"upload_time": "2025-07-12T11:18:19",
"upload_time_iso_8601": "2025-07-12T11:18:19.694734Z",
"url": "https://files.pythonhosted.org/packages/77/75/93ddcc340b49d8bf6c663b827cc4f46c321cde8fa3923e73d329179603b5/crawl4ai-0.7.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ffccd6f48c4b01719b1d8f5f1ad8b1f67569d59db8f566448a8747814bd70e60",
"md5": "790ba8d60c2ea26c952d5b915cee5e24",
"sha256": "c6925341c8358ef8b30793fa3864b1f9002f9b24dbc5e661401eb78cfa762f57"
},
"downloads": -1,
"filename": "crawl4ai-0.7.0.tar.gz",
"has_sig": false,
"md5_digest": "790ba8d60c2ea26c952d5b915cee5e24",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 397687,
"upload_time": "2025-07-12T11:18:21",
"upload_time_iso_8601": "2025-07-12T11:18:21.626155Z",
"url": "https://files.pythonhosted.org/packages/ff/cc/d6f48c4b01719b1d8f5f1ad8b1f67569d59db8f566448a8747814bd70e60/crawl4ai-0.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-12 11:18:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "unclecode",
"github_project": "crawl4ai",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiosqlite",
"specs": [
[
"~=",
"0.20"
]
]
},
{
"name": "lxml",
"specs": [
[
"~=",
"5.3"
]
]
},
{
"name": "litellm",
"specs": [
[
">=",
"1.53.1"
]
]
},
{
"name": "numpy",
"specs": [
[
"<",
"3"
],
[
">=",
"1.26.0"
]
]
},
{
"name": "pillow",
"specs": [
[
">=",
"10.4"
]
]
},
{
"name": "playwright",
"specs": [
[
">=",
"1.49.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"~=",
"1.0"
]
]
},
{
"name": "requests",
"specs": [
[
"~=",
"2.26"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
"~=",
"4.12"
]
]
},
{
"name": "tf-playwright-stealth",
"specs": [
[
">=",
"1.1.0"
]
]
},
{
"name": "xxhash",
"specs": [
[
"~=",
"3.4"
]
]
},
{
"name": "rank-bm25",
"specs": [
[
"~=",
"0.2"
]
]
},
{
"name": "aiofiles",
"specs": [
[
">=",
"24.1.0"
]
]
},
{
"name": "colorama",
"specs": [
[
"~=",
"0.4"
]
]
},
{
"name": "snowballstemmer",
"specs": [
[
"~=",
"2.2"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"2.10"
]
]
},
{
"name": "pyOpenSSL",
"specs": [
[
">=",
"24.3.0"
]
]
},
{
"name": "psutil",
"specs": [
[
">=",
"6.1.1"
]
]
},
{
"name": "nltk",
"specs": [
[
">=",
"3.9.1"
]
]
},
{
"name": "rich",
"specs": [
[
">=",
"13.9.4"
]
]
},
{
"name": "cssselect",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "chardet",
"specs": [
[
">=",
"5.2.0"
]
]
},
{
"name": "brotli",
"specs": [
[
">=",
"1.1.0"
]
]
},
{
"name": "httpx",
"specs": [
[
">=",
"0.27.2"
]
]
},
{
"name": "sentence-transformers",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "alphashape",
"specs": [
[
">=",
"1.3.1"
]
]
},
{
"name": "shapely",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "fake-useragent",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "pdf2image",
"specs": [
[
">=",
"1.17.0"
]
]
},
{
"name": "PyPDF2",
"specs": [
[
">=",
"3.0.1"
]
]
}
],
"lcname": "crawl4ai"
}
JSON
