# OpenIntel
A comprehensive Python package for detecting and analyzing technology stacks used by websites. Perfect for competitive intelligence, lead generation, and market research.
## Features
- **Comprehensive Detection**: Identifies 50+ technologies across multiple categories
- **Fast Async Processing**: Concurrent analysis of multiple websites
- **Detailed Analysis**: Confidence scoring and evidence collection
- **Trend Analysis**: Track technology adoption over time
- **CLI Interface**: Easy command-line usage
- **API Ready**: Use as a library in your Python projects
## Categories Detected
- Frontend Frameworks (React, Vue.js, Angular)
- JavaScript Libraries (jQuery, Lodash)
- CSS Frameworks (Bootstrap, Tailwind CSS)
- Analytics (Google Analytics, Facebook Pixel)
- CDNs (Cloudflare, Amazon CloudFront)
- CMS (WordPress, Drupal)
- E-commerce (Shopify, WooCommerce)
- Backend Technologies (Apache, Nginx, Express.js)
- And many more...
## Installation
```bash
pip install openintel
```
## Quick Start
### Command Line Usage
```bash
# Analyze a single website
openintel example.com
# Analyze multiple websites from file
openintel --file urls.txt --format json --output results.json
# Analyze with custom timeout
openintel example.com --timeout 60
```
### Python API Usage
```python
import asyncio
from openintel import TechStackDetector, TechStackAnalyzer
async def main():
# Basic detection
async with TechStackDetector() as detector:
result = await detector.detect("https://example.com")
print(f"Found {len(result.technologies)} technologies")
for tech in result.technologies:
print(f"- {tech.name} ({tech.confidence:.2f})")
# Advanced analysis
analyzer = TechStackAnalyzer()
analyzer.add_result(result)
# Get technology scoring
scores = analyzer.score_tech_stack(result)
print(f"Overall tech stack score: {scores['overall_score']}/10")
asyncio.run(main())
```
## API Reference
### TechStackDetector
Main detection class with async context manager support.
```python
async with TechStackDetector(timeout=30) as detector:
result = await detector.detect(url)
```
### DetectionResult
Contains comprehensive analysis results:
```python
class DetectionResult:
url: str
timestamp: datetime
technologies: List[DetectedTech]
total_score: float
response_time: float
errors: List[str]
raw_data: Dict[str, Any]
```
### TechStackAnalyzer
Advanced analysis and trend tracking:
```python
analyzer = TechStackAnalyzer()
trends = analyzer.get_technology_trends(days=30)
scores = analyzer.score_tech_stack(result)
```
## Detection Methods
The detector uses multiple techniques:
1. **HTML Pattern Matching**: Searches for framework-specific patterns
2. **JavaScript Analysis**: Identifies libraries and frameworks
3. **HTTP Headers**: Analyzes server and security headers
4. **Meta Tags**: Extracts generator and other meta information
5. **Script Sources**: Identifies external libraries and CDNs
6. **DNS Analysis**: Checks for hosting and CDN providers
## Example Output
```json
{
"url": "https://example.com",
"timestamp": "2024-01-15T10:30:00",
"technologies": [
{
"name": "React",
"category": "frontend_framework",
"confidence": 0.95,
"version": "18.2.0",
"evidence": ["HTML pattern: _react", "Script: react.min.js"]
},
{
"name": "Cloudflare",
"category": "cdn",
"confidence": 0.98,
"evidence": ["Header server: cloudflare"]
}
],
"total_score": 15.7,
"response_time": 2.34
}
```
## Advanced Features
### Batch Processing
```python
urls = ["site1.com", "site2.com", "site3.com"]
async with TechStackDetector() as detector:
tasks = [detector.detect(url) for url in urls]
results = await asyncio.gather(*tasks)
```
### Custom Signatures
Extend detection capabilities by adding custom technology signatures:
```python
from openintel.models import TechSignature, TechCategory
custom_sig = TechSignature(
name="Custom Framework",
category=TechCategory.FRONTEND_FRAMEWORK,
patterns={"html": [r"custom-framework"]},
confidence_score=0.9
)
```
### Trend Analysis
```python
analyzer = TechStackAnalyzer()
for result in results:
analyzer.add_result(result)
trends = analyzer.get_technology_trends(days=30)
print(f"Most popular: {trends['top_technologies'][0]}")
```
## Performance
- **Concurrent Processing**: Analyze multiple sites simultaneously
- **Async Operations**: Non-blocking I/O for maximum efficiency
- **Intelligent Caching**: Reduces redundant requests
- **Configurable Timeouts**: Prevent hanging requests
## Use Cases
- **Competitive Intelligence**: Analyze competitor tech stacks
- **Lead Generation**: Identify prospects using specific technologies
- **Market Research**: Track technology adoption trends
- **Security Auditing**: Identify outdated or vulnerable technologies
- **Sales Intelligence**: Tailor pitches based on prospect's tech stack
## Contributing
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request
## License
MIT License - see LICENSE file for details.
## Support
- GitHub Issues: Report bugs and request features
- Documentation: Full API documentation available
- Examples: Check the examples/ directory
## Changelog
### v1.0.0
- Initial release
- 50+ technology signatures
- Async detection engine
- CLI interface
- Trend analysis capabilities
Raw data
{
"_id": null,
"home_page": null,
"name": "openintel",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "competitive-intelligence, mixrank, technology-detection, web-scraping",
"author": null,
"author_email": "Devanshu Singh <devanshukumar45@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/83/14/5ea618b392febb03630bcd04194a827f47c2954d84476601b72dc157e52b/openintel-1.0.0.tar.gz",
"platform": null,
"description": "# OpenIntel\n\nA comprehensive Python package for detecting and analyzing technology stacks used by websites. Perfect for competitive intelligence, lead generation, and market research.\n\n## Features\n\n- **Comprehensive Detection**: Identifies 50+ technologies across multiple categories\n- **Fast Async Processing**: Concurrent analysis of multiple websites\n- **Detailed Analysis**: Confidence scoring and evidence collection\n- **Trend Analysis**: Track technology adoption over time\n- **CLI Interface**: Easy command-line usage\n- **API Ready**: Use as a library in your Python projects\n\n## Categories Detected\n\n- Frontend Frameworks (React, Vue.js, Angular)\n- JavaScript Libraries (jQuery, Lodash)\n- CSS Frameworks (Bootstrap, Tailwind CSS)\n- Analytics (Google Analytics, Facebook Pixel)\n- CDNs (Cloudflare, Amazon CloudFront)\n- CMS (WordPress, Drupal)\n- E-commerce (Shopify, WooCommerce)\n- Backend Technologies (Apache, Nginx, Express.js)\n- And many more...\n\n## Installation\n\n```bash\npip install openintel\n```\n\n## Quick Start\n\n### Command Line Usage\n\n```bash\n# Analyze a single website\nopenintel example.com\n\n# Analyze multiple websites from file\nopenintel --file urls.txt --format json --output results.json\n\n# Analyze with custom timeout\nopenintel example.com --timeout 60\n```\n\n### Python API Usage\n\n```python\nimport asyncio\nfrom openintel import TechStackDetector, TechStackAnalyzer\n\nasync def main():\n # Basic detection\n async with TechStackDetector() as detector:\n result = await detector.detect(\"https://example.com\")\n \n print(f\"Found {len(result.technologies)} technologies\")\n for tech in result.technologies:\n print(f\"- {tech.name} ({tech.confidence:.2f})\")\n \n # Advanced analysis\n analyzer = TechStackAnalyzer()\n analyzer.add_result(result)\n \n # Get technology scoring\n scores = analyzer.score_tech_stack(result)\n print(f\"Overall tech stack score: {scores['overall_score']}/10\")\n\nasyncio.run(main())\n```\n\n## API Reference\n\n### TechStackDetector\n\nMain detection class with async context manager support.\n\n```python\nasync with TechStackDetector(timeout=30) as detector:\n result = await detector.detect(url)\n```\n\n### DetectionResult\n\nContains comprehensive analysis results:\n\n```python\nclass DetectionResult:\n url: str\n timestamp: datetime\n technologies: List[DetectedTech]\n total_score: float\n response_time: float\n errors: List[str]\n raw_data: Dict[str, Any]\n```\n\n### TechStackAnalyzer\n\nAdvanced analysis and trend tracking:\n\n```python\nanalyzer = TechStackAnalyzer()\ntrends = analyzer.get_technology_trends(days=30)\nscores = analyzer.score_tech_stack(result)\n```\n\n## Detection Methods\n\nThe detector uses multiple techniques:\n\n1. **HTML Pattern Matching**: Searches for framework-specific patterns\n2. **JavaScript Analysis**: Identifies libraries and frameworks\n3. **HTTP Headers**: Analyzes server and security headers\n4. **Meta Tags**: Extracts generator and other meta information\n5. **Script Sources**: Identifies external libraries and CDNs\n6. **DNS Analysis**: Checks for hosting and CDN providers\n\n## Example Output\n\n```json\n{\n \"url\": \"https://example.com\",\n \"timestamp\": \"2024-01-15T10:30:00\",\n \"technologies\": [\n {\n \"name\": \"React\",\n \"category\": \"frontend_framework\",\n \"confidence\": 0.95,\n \"version\": \"18.2.0\",\n \"evidence\": [\"HTML pattern: _react\", \"Script: react.min.js\"]\n },\n {\n \"name\": \"Cloudflare\",\n \"category\": \"cdn\",\n \"confidence\": 0.98,\n \"evidence\": [\"Header server: cloudflare\"]\n }\n ],\n \"total_score\": 15.7,\n \"response_time\": 2.34\n}\n```\n\n## Advanced Features\n\n### Batch Processing\n\n```python\nurls = [\"site1.com\", \"site2.com\", \"site3.com\"]\nasync with TechStackDetector() as detector:\n tasks = [detector.detect(url) for url in urls]\n results = await asyncio.gather(*tasks)\n```\n\n### Custom Signatures\n\nExtend detection capabilities by adding custom technology signatures:\n\n```python\nfrom openintel.models import TechSignature, TechCategory\n\ncustom_sig = TechSignature(\n name=\"Custom Framework\",\n category=TechCategory.FRONTEND_FRAMEWORK,\n patterns={\"html\": [r\"custom-framework\"]},\n confidence_score=0.9\n)\n```\n\n### Trend Analysis\n\n```python\nanalyzer = TechStackAnalyzer()\nfor result in results:\n analyzer.add_result(result)\n\ntrends = analyzer.get_technology_trends(days=30)\nprint(f\"Most popular: {trends['top_technologies'][0]}\")\n```\n\n## Performance\n\n- **Concurrent Processing**: Analyze multiple sites simultaneously\n- **Async Operations**: Non-blocking I/O for maximum efficiency\n- **Intelligent Caching**: Reduces redundant requests\n- **Configurable Timeouts**: Prevent hanging requests\n\n## Use Cases\n\n- **Competitive Intelligence**: Analyze competitor tech stacks\n- **Lead Generation**: Identify prospects using specific technologies\n- **Market Research**: Track technology adoption trends\n- **Security Auditing**: Identify outdated or vulnerable technologies\n- **Sales Intelligence**: Tailor pitches based on prospect's tech stack\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Ensure all tests pass\n5. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Support\n\n- GitHub Issues: Report bugs and request features\n- Documentation: Full API documentation available\n- Examples: Check the examples/ directory\n\n## Changelog\n\n### v1.0.0\n- Initial release\n- 50+ technology signatures\n- Async detection engine\n- CLI interface\n- Trend analysis capabilities",
"bugtrack_url": null,
"license": null,
"summary": "A comprehensive tool to detect technology stacks used by websites",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/Devanshusisodiya/openintel",
"Issues": "https://github.com/Devanshusisodiya/openintel/issues",
"Repository": "https://github.com/Devanshusisodiya/openintel"
},
"split_keywords": [
"competitive-intelligence",
" mixrank",
" technology-detection",
" web-scraping"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "e8bf8808a7472404badd50e5512fa82c8a9c22f54d68aca5a06df0d54da19717",
"md5": "108729b2adb76a5100b09365b7c1be36",
"sha256": "4aa02b6ab40a20cdca04e37efcfd9180c84344f66ae6e756ec90977f9b9a4f1e"
},
"downloads": -1,
"filename": "openintel-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "108729b2adb76a5100b09365b7c1be36",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 14373,
"upload_time": "2025-07-26T20:03:29",
"upload_time_iso_8601": "2025-07-26T20:03:29.092410Z",
"url": "https://files.pythonhosted.org/packages/e8/bf/8808a7472404badd50e5512fa82c8a9c22f54d68aca5a06df0d54da19717/openintel-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "83145ea618b392febb03630bcd04194a827f47c2954d84476601b72dc157e52b",
"md5": "f3aa17f8c294eddb91563d9100e9f79c",
"sha256": "03cb3196056ffad9b06c67a97205023117a2e10a686d61f15fadd25c845f9716"
},
"downloads": -1,
"filename": "openintel-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "f3aa17f8c294eddb91563d9100e9f79c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 154600,
"upload_time": "2025-07-26T20:03:32",
"upload_time_iso_8601": "2025-07-26T20:03:32.115390Z",
"url": "https://files.pythonhosted.org/packages/83/14/5ea618b392febb03630bcd04194a827f47c2954d84476601b72dc157e52b/openintel-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-26 20:03:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Devanshusisodiya",
"github_project": "openintel",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "openintel"
}
JSON
