# web_utils.py Library: Providing a Unified Solution for Web Requests
[English Documentation](README.md) | [中文文档](README_CN.md)
**Note:** If there is a conflict between the Chinese and English documents, please prioritize the English document.
## Introduction
In the digital age, data has never been more important. Every day, developers, data scientists, and business analysts are striving to retrieve, analyze, and leverage data from the web. However, there is currently a wide variety of web request libraries, each with its unique interface and usage methods. This not only makes learning and switching costly but also introduces complexity and instability into the code.
The birth of the **WebFetcher** library stems from the desire for a unified, efficient, and straightforward solution for web requests. Whether you are using requests, urllib, http.client, aiohttp, or Selenium under the hood, **WebFetcher** provides you with a unified and concise interface. This means you can freely switch backend technologies without altering any core code, ensuring code readability and maintainability.
But this is just our first step. We believe that community strength is essential to making this library truly influential and widely used. Whether you are a beginner or an experienced developer, we sincerely invite you to participate in this project. You can contribute not only by submitting code and improving features but also by sharing your usage experience and suggestions to help us. Only through cooperation can **WebFetcher** truly become an essential tool in every developer's toolbox.
If you are passionate about web development and have an unwavering pursuit of innovation and excellence, please join us! Let's work together to create a powerful, flexible, and user-friendly web request library that serves developers worldwide.
## Installation
You can install the package using pip:
`pip install webmix`
`pip3 install webmix`
Alternatively, you can clone the repository from GitHub and install it manually:
`git clone https://github.com/ng-fukgin/webmix`
`cd webmix`
`python setup.py install`
## Using the WebFetcher Class
### Using `requests` as the Backend
```
from webmix.web_utils import WebFetcher
fetcher = WebFetcher(backend='requests')
response = fetcher.get("https://example.com")
print(response)
```
### Using `urllib` as the Backend
```
fetcher = WebFetcher(backend='urllib')
response = fetcher.get("https://example.com")
print(response)
```
### Using `selenium` as the Backend
```
fetcher = WebFetcher(backend='selenium')
response = fetcher.get("https://example.com")
print(response)
```
### Selenium Helper Methods
```
fetcher.scroll_to_bottom()
fetcher.fill_form_by_id("myFormId", "Some Value")
fetcher.click_button_by_id("myButtonId")
script_output = fetcher.execute_script("return document.title;")
print(script_output)
fetcher.close() # close the `WebDriver` session
```
Note that using Selenium may require additional setup, such as installing the appropriate browser driver.
### Additional Methods with Selenium
The WebFetcher class provides a host of helper methods to facilitate interaction with web elements when using the Selenium backend:
* `find_element_by_id`: Locate an element by its ID.
* `find_element_by_name`: Locate an element by its name attribute.
* `find_element_by_xpath`: Locate an element using XPath.
* `wait_for_element_visibility`: Wait for an element to be visible.
* `switch_to_frame`: Switch the focus to a particular iframe or frame.
* `switch_to_default_content`: Switch focus back to the main page content.
* `switch_to_window`: Switch focus to a different browser window or tab.
* `accept_alert`: Accept the currently displayed alert box.
* `dismiss_alert`: Dismiss the currently displayed alert box.
* `maximize_window`: Maximize the browser window.
* `set_window_size`: Set the dimensions of the browser window.
### Other Request Methods
WebFetcher also supports making HTTP requests with a variety of backends:
* `_request_with_requests`: Execute an HTTP request using the requests library.
* `_request_with_urllib`: Execute an HTTP request using the urllib library.
* `_request_with_http_client`: Execute an HTTP request using the http.client library.
* `_request_with_aiohttp`: Execute an asynchronous HTTP request using the aiohttp library.
### Support for Other Libraries
#### http.client Backend
```
fetcher = WebFetcher(backend='http_client')
response = fetcher.get("https://example.com")
print(response)
```
#### aiohttp Backend (Asynchronous)
```
import asyncio
async def fetch_data():
fetcher = WebFetcher(backend='aiohttp')
response = await fetcher.get("https://example.com")
print(response)
asyncio.run(fetch_data())
```
## Version History
| Version | Description |
|---------|-------------|
| 0.0.4 | Initial release with basic functionalities working as expected. |
| 0.0.5 | Introduced support for `http.client` and `aiohttp` backends. Extended the list of methods available for Selenium integration. |
Raw data
{
"_id": null,
"home_page": "https://github.com/ng-fukgin/webmix",
"name": "webmix",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "webmix, \u7f51\u9875\u6293\u53d6, \u81ea\u52a8\u5316, requests, urllib, selenium, web scraping, automation",
"author": "wfj",
"author_email": "wfj.0000@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c8/72/4033df2eccd845527c8cf7a16050a3e11f23af0f4774d162508bd9f95ff6/webmix-0.0.10.tar.gz",
"platform": "any",
"description": "\r\n\r\n# web_utils.py Library: Providing a Unified Solution for Web Requests\r\n\r\n[English Documentation](README.md) | [\u4e2d\u6587\u6587\u6863](README_CN.md)\r\n\r\n**Note:** If there is a conflict between the Chinese and English documents, please prioritize the English document.\r\n\r\n## Introduction\r\n\r\nIn the digital age, data has never been more important. Every day, developers, data scientists, and business analysts are striving to retrieve, analyze, and leverage data from the web. However, there is currently a wide variety of web request libraries, each with its unique interface and usage methods. This not only makes learning and switching costly but also introduces complexity and instability into the code.\r\n\r\nThe birth of the **WebFetcher** library stems from the desire for a unified, efficient, and straightforward solution for web requests. Whether you are using requests, urllib, http.client, aiohttp, or Selenium under the hood, **WebFetcher** provides you with a unified and concise interface. This means you can freely switch backend technologies without altering any core code, ensuring code readability and maintainability.\r\n\r\nBut this is just our first step. We believe that community strength is essential to making this library truly influential and widely used. Whether you are a beginner or an experienced developer, we sincerely invite you to participate in this project. You can contribute not only by submitting code and improving features but also by sharing your usage experience and suggestions to help us. Only through cooperation can **WebFetcher** truly become an essential tool in every developer's toolbox.\r\n\r\nIf you are passionate about web development and have an unwavering pursuit of innovation and excellence, please join us! Let's work together to create a powerful, flexible, and user-friendly web request library that serves developers worldwide.\r\n\r\n## Installation\r\n\r\nYou can install the package using pip:\r\n\r\n`pip install webmix` \r\n`pip3 install webmix`\r\n\r\nAlternatively, you can clone the repository from GitHub and install it manually:\r\n\r\n`git clone https://github.com/ng-fukgin/webmix` \r\n`cd webmix` \r\n`python setup.py install`\r\n\r\n## Using the WebFetcher Class\r\n\r\n### Using `requests` as the Backend\r\n```\r\nfrom webmix.web_utils import WebFetcher\r\n\r\nfetcher = WebFetcher(backend='requests')\r\nresponse = fetcher.get(\"https://example.com\")\r\nprint(response)\r\n``` \r\n\r\n### Using `urllib` as the Backend\r\n```\r\nfetcher = WebFetcher(backend='urllib')\r\nresponse = fetcher.get(\"https://example.com\")\r\nprint(response)\r\n ``` \r\n\r\n### Using `selenium` as the Backend\r\n```\r\nfetcher = WebFetcher(backend='selenium')\r\nresponse = fetcher.get(\"https://example.com\")\r\nprint(response)\r\n ``` \r\n\r\n### Selenium Helper Methods\r\n```\r\nfetcher.scroll_to_bottom()\r\nfetcher.fill_form_by_id(\"myFormId\", \"Some Value\")\r\nfetcher.click_button_by_id(\"myButtonId\")\r\nscript_output = fetcher.execute_script(\"return document.title;\")\r\nprint(script_output)\r\n\r\nfetcher.close() # close the `WebDriver` session\r\n``` \r\n\r\nNote that using Selenium may require additional setup, such as installing the appropriate browser driver.\r\n\r\n### Additional Methods with Selenium\r\n\r\nThe WebFetcher class provides a host of helper methods to facilitate interaction with web elements when using the Selenium backend:\r\n\r\n* `find_element_by_id`: Locate an element by its ID.\r\n* `find_element_by_name`: Locate an element by its name attribute.\r\n* `find_element_by_xpath`: Locate an element using XPath.\r\n* `wait_for_element_visibility`: Wait for an element to be visible.\r\n* `switch_to_frame`: Switch the focus to a particular iframe or frame.\r\n* `switch_to_default_content`: Switch focus back to the main page content.\r\n* `switch_to_window`: Switch focus to a different browser window or tab.\r\n* `accept_alert`: Accept the currently displayed alert box.\r\n* `dismiss_alert`: Dismiss the currently displayed alert box.\r\n* `maximize_window`: Maximize the browser window.\r\n* `set_window_size`: Set the dimensions of the browser window.\r\n\r\n### Other Request Methods\r\n\r\nWebFetcher also supports making HTTP requests with a variety of backends:\r\n\r\n* `_request_with_requests`: Execute an HTTP request using the requests library.\r\n* `_request_with_urllib`: Execute an HTTP request using the urllib library.\r\n* `_request_with_http_client`: Execute an HTTP request using the http.client library.\r\n* `_request_with_aiohttp`: Execute an asynchronous HTTP request using the aiohttp library.\r\n\r\n### Support for Other Libraries\r\n\r\n#### http.client Backend\r\n\r\n```\r\n\r\nfetcher = WebFetcher(backend='http_client')\r\nresponse = fetcher.get(\"https://example.com\")\r\nprint(response)\r\n```\r\n\r\n#### aiohttp Backend (Asynchronous)\r\n\r\n```\r\n\r\nimport asyncio\r\n\r\nasync def fetch_data():\r\n fetcher = WebFetcher(backend='aiohttp')\r\n response = await fetcher.get(\"https://example.com\")\r\n print(response)\r\n\r\nasyncio.run(fetch_data())\r\n```\r\n\r\n## Version History\r\n\r\n| Version | Description |\r\n|---------|-------------|\r\n| 0.0.4 | Initial release with basic functionalities working as expected. |\r\n| 0.0.5 | Introduced support for `http.client` and `aiohttp` backends. Extended the list of methods available for Selenium integration. |\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A set of utilities for web scraping and automation, supporting multiple backends: requests, urllib, and selenium.",
"version": "0.0.10",
"project_urls": {
"Bug Tracker": "https://github.com/ng-fukgin/webmix/issues",
"Documentation": "https://github.com/ng-fukgin/webmix/wiki",
"Homepage": "https://github.com/ng-fukgin/webmix",
"Source Code": "https://github.com/ng-fukgin/webmix"
},
"split_keywords": [
"webmix",
" \u7f51\u9875\u6293\u53d6",
" \u81ea\u52a8\u5316",
" requests",
" urllib",
" selenium",
" web scraping",
" automation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2b2c02f21af8afc43a0dc1f23808ececde9fdfb7473235b85a9f8fd938086224",
"md5": "c3a6b3b37efea423e222bc0d54b301c6",
"sha256": "178324ac1969c602b889286079e6d3f2f6a0482af8f55e8549ea5b0d3dfca0f4"
},
"downloads": -1,
"filename": "webmix-0.0.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c3a6b3b37efea423e222bc0d54b301c6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6652,
"upload_time": "2024-09-05T02:40:26",
"upload_time_iso_8601": "2024-09-05T02:40:26.621100Z",
"url": "https://files.pythonhosted.org/packages/2b/2c/02f21af8afc43a0dc1f23808ececde9fdfb7473235b85a9f8fd938086224/webmix-0.0.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c8724033df2eccd845527c8cf7a16050a3e11f23af0f4774d162508bd9f95ff6",
"md5": "14a7520a16cd3872ff442deab3321a40",
"sha256": "ab75c2e1c140cbe2ed41e56d2346446f4ca76057c3617d57d503eafb57aa71e3"
},
"downloads": -1,
"filename": "webmix-0.0.10.tar.gz",
"has_sig": false,
"md5_digest": "14a7520a16cd3872ff442deab3321a40",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 8816,
"upload_time": "2024-09-05T02:40:28",
"upload_time_iso_8601": "2024-09-05T02:40:28.009583Z",
"url": "https://files.pythonhosted.org/packages/c8/72/4033df2eccd845527c8cf7a16050a3e11f23af0f4774d162508bd9f95ff6/webmix-0.0.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-05 02:40:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ng-fukgin",
"github_project": "webmix",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "webmix"
}
JSON
