<a href="https://webtranspose.com">
<img alt="Web Transpose. Simple APIs to get data from the internet." src="img/web-transpose-cover.png" width="100%" />
<h1 align="center">Web Transpose</h1>
<p align="center"><b>
Web Crawler & AI Web Scraper APIs for building new web experiences.
</b></p>
</a>
```bash
pip install webtranspose
```
<h4 align="center">
<a href="https://twitter.com/mikegeecmu">
<img src="https://img.shields.io/twitter/follow/mikegeecmu?style=flat&label=%40mikegeecmu&logo=twitter&color=0bf&logoColor=fff" alt="X" />
</a>
<a href="https://github.com/mike-gee/webtranspose/blob/master/LICENSE.rst">
<img src="https://img.shields.io/badge/LICENSE-GNU%20AGPLv3-blue" alt="License" />
</a>
<a href="https://github.com/mikegeecmu/webtranspose/blob/main/LICENSE">
<img src="https://img.shields.io/badge/docs-Web%20Transpose-blue" alt="License" />
</a>
</h4>
<p align="center">
<a href="#introduction"><strong>Introduction</strong></a> ·
<a href="#installation"><strong>Installation</strong></a> ·
<a href="https://docs.webtranspose.com"><strong>Docs</strong></a>
</p>
<br/>
## Introduction
In the near future, **nobody will open websites**. Instead, we will be directly served the information we are seeking. New web experiences will combine the information from many websites into a single, unified experience.
**Web Transpose** is a collection of API tools that allow building these new web experiences simple.
- [Webᵀ Crawl: Distributed Web Crawler](#crawl)
- [Webᵀ Scrape: AI Web Scraper](#scrape)
### Crawl
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
crawl = webt.Crawl(
"https://www.example.com",
max_pages=100,
render_js=True,
)
await crawl.crawl() # crawl.queue_crawl() for async
```
## Scrape
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
schema = {
"Merchant Name": "string",
"Title of Product": "string",
"Product Photo URL": "string",
}
scraper = webt.Scraper(
schema,
render_js=True,
)
out_json = scraper.scrape("https://www.example.com")
```
## Web Search (AI SERP API)
```python
import webtranspose as webt
import os
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR WEBT API KEY"
results = webt.search("what caused the fourth great ninja war?")
# results.keys()
# ['results']
# AI Filter
results = webt.search_filter("Paul Graham's Blog")
# results.keys()
# ['results', 'filtered_results']
```
## Installation
Non-Python Users: [📄 API Docs](https://docs.webtranspose.com).
This repo contains a local **lite** installation of Web Transpose. This is a good option if you want to run Web Transpose locally on your machine for quick use cases.
```shell
pip install webtranspose
```
However, if you wish to leverage the full tools of Web Transpose and use in production, you should add your API key to add the **full** version.
```python
os.environ["WEBTRANSPOSE_API_KEY"] = "YOUR_API_KEY_HERE"
```
## Enterprise Support
Web Transpose serves enterprises small and large. We partner with companies for the long term with hands-on support and custom solutions.
Please email me directly at mike@webtranspose.com for enquiries.
Raw data
{
"_id": null,
"home_page": "https://github.com/mike-gee/webtranspose",
"name": "webtranspose",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.12",
"maintainer_email": "",
"keywords": "webtranspose",
"author": "Mike Gee",
"author_email": "mike@webtranspose.com",
"download_url": "https://files.pythonhosted.org/packages/87/30/9768eb37025c99c4e77cc9199d77b262df060d449605dafac204eb1642cc/webtranspose-0.3.2.tar.gz",
"platform": null,
"description": "<a href=\"https://webtranspose.com\">\n <img alt=\"Web Transpose. Simple APIs to get data from the internet.\" src=\"img/web-transpose-cover.png\" width=\"100%\" />\n <h1 align=\"center\">Web Transpose</h1>\n <p align=\"center\"><b>\n Web Crawler & AI Web Scraper APIs for building new web experiences.\n </b></p>\n</a>\n\n```bash\npip install webtranspose\n```\n\n<h4 align=\"center\">\n <a href=\"https://twitter.com/mikegeecmu\">\n <img src=\"https://img.shields.io/twitter/follow/mikegeecmu?style=flat&label=%40mikegeecmu&logo=twitter&color=0bf&logoColor=fff\" alt=\"X\" />\n </a>\n <a href=\"https://github.com/mike-gee/webtranspose/blob/master/LICENSE.rst\">\n <img src=\"https://img.shields.io/badge/LICENSE-GNU%20AGPLv3-blue\" alt=\"License\" />\n </a>\n <a href=\"https://github.com/mikegeecmu/webtranspose/blob/main/LICENSE\">\n <img src=\"https://img.shields.io/badge/docs-Web%20Transpose-blue\" alt=\"License\" />\n </a>\n</h4>\n\n\n<p align=\"center\">\n <a href=\"#introduction\"><strong>Introduction</strong></a> \u00b7\n <a href=\"#installation\"><strong>Installation</strong></a> \u00b7\n <a href=\"https://docs.webtranspose.com\"><strong>Docs</strong></a>\n</p>\n<br/>\n\n## Introduction\n\nIn the near future, **nobody will open websites**. Instead, we will be directly served the information we are seeking. New web experiences will combine the information from many websites into a single, unified experience.\n\n**Web Transpose** is a collection of API tools that allow building these new web experiences simple.\n\n- [Web\u1d40 Crawl: Distributed Web Crawler](#crawl)\n- [Web\u1d40 Scrape: AI Web Scraper](#scrape)\n\n\n### Crawl\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\ncrawl = webt.Crawl(\n \"https://www.example.com\",\n max_pages=100,\n render_js=True,\n)\nawait crawl.crawl() # crawl.queue_crawl() for async\n```\n\n## Scrape\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\nschema = {\n \"Merchant Name\": \"string\",\n \"Title of Product\": \"string\",\n \"Product Photo URL\": \"string\",\n}\n\nscraper = webt.Scraper(\n schema, \n render_js=True, \n)\nout_json = scraper.scrape(\"https://www.example.com\")\n```\n\n## Web Search (AI SERP API)\n\n```python\nimport webtranspose as webt\n\nimport os\nos.environ['WEBTRANSPOSE_API_KEY'] = \"YOUR WEBT API KEY\"\n\nresults = webt.search(\"what caused the fourth great ninja war?\")\n# results.keys()\n# ['results']\n\n# AI Filter\nresults = webt.search_filter(\"Paul Graham's Blog\")\n# results.keys()\n# ['results', 'filtered_results']\n```\n\n\n## Installation\n\nNon-Python Users: [\ud83d\udcc4 API Docs](https://docs.webtranspose.com).\n\nThis repo contains a local **lite** installation of Web Transpose. This is a good option if you want to run Web Transpose locally on your machine for quick use cases. \n\n```shell\npip install webtranspose\n```\n\nHowever, if you wish to leverage the full tools of Web Transpose and use in production, you should add your API key to add the **full** version.\n\n```python\nos.environ[\"WEBTRANSPOSE_API_KEY\"] = \"YOUR_API_KEY_HERE\"\n```\n\n\n## Enterprise Support\n\nWeb Transpose serves enterprises small and large. We partner with companies for the long term with hands-on support and custom solutions.\n\nPlease email me directly at mike@webtranspose.com for enquiries.\n",
"bugtrack_url": null,
"license": "",
"summary": "Reliable APIs for the website data",
"version": "0.3.2",
"project_urls": {
"Bug Tracker": "https://github.com/mike-gee/webtranspose/issues",
"Documentation": "https://docs.webtranspose.com",
"Homepage": "https://github.com/mike-gee/webtranspose",
"Repository": "https://github.com/mike-gee/webtranspose"
},
"split_keywords": [
"webtranspose"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "19c0df78f771722417090ed3735b5d2ed29542b139dde00b1e2ca972beb5b9e5",
"md5": "cc62c0b05b5bfef1a91bed817715aa40",
"sha256": "1e86ac0cc183d4f3607d6a99f8849aaba6162acc7cf912fdc367fe0d3da59cf2"
},
"downloads": -1,
"filename": "webtranspose-0.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cc62c0b05b5bfef1a91bed817715aa40",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<3.12",
"size": 26235,
"upload_time": "2024-01-18T00:35:57",
"upload_time_iso_8601": "2024-01-18T00:35:57.934474Z",
"url": "https://files.pythonhosted.org/packages/19/c0/df78f771722417090ed3735b5d2ed29542b139dde00b1e2ca972beb5b9e5/webtranspose-0.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "87309768eb37025c99c4e77cc9199d77b262df060d449605dafac204eb1642cc",
"md5": "40d382618df7a4bc004b3651a116af9c",
"sha256": "91395fa591944ff2c5899a3890c6765204c99bb61e1065677b5d9f21bf85b50f"
},
"downloads": -1,
"filename": "webtranspose-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "40d382618df7a4bc004b3651a116af9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.12",
"size": 25669,
"upload_time": "2024-01-18T00:36:00",
"upload_time_iso_8601": "2024-01-18T00:36:00.579348Z",
"url": "https://files.pythonhosted.org/packages/87/30/9768eb37025c99c4e77cc9199d77b262df060d449605dafac204eb1642cc/webtranspose-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-18 00:36:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mike-gee",
"github_project": "webtranspose",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "webtranspose"
}
JSON
