<h1 align="center">request-boost </h1>
<p align="center">
<a href="https://pypi.org/project/request-boost/">
<img src="https://github.com/singhsidhukuldeep/request-boost/raw/main/img/benchmark.png" width="1080" alt="Go to https://pypi.org/project/request-boost/">
</a>
</p>
<p align="center">
<b>A better requests and urllib package.</b></b><br> A simple package for hitting multiple URLs and performing GET/POST requests in parallel.<br>
<a href="https://pypi.org/project/request-boost/"><img src="https://img.shields.io/pypi/pyversions/request-boost" alt="Go to https://pypi.org/project/request-boost/"/></a>
<a href="https://pypi.org/project/request-boost/"><img src="https://img.shields.io/pypi/v/request-boost" alt="Go to https://pypi.org/project/request-boost/"/></a>
<a href="https://pypi.org/project/request-boost/"><img src="https://img.shields.io/pypi/status/request-boost" alt="Go to https://pypi.org/project/request-boost/"/></a>
<!-- <a href="https://pypi.org/project/request-boost/"><img src="https://img.shields.io/pypi/format/request-boost" alt="Go to https://pypi.org/project/request-boost/"/></a> -->
<!-- <a href="https://lgtm.com/projects/g/singhsidhukuldeep/request-boost/context:python"><img alt="Language grade: Python" src="https://img.shields.io/lgtm/grade/python/g/singhsidhukuldeep/request-boost.svg?logo=lgtm&logoWidth=18"/></a> -->
<a href="https://pypistats.org/packages/request-boost"><img src="https://img.shields.io/pypi/dm/request-boost"/></a>
<!-- <img src="https://visitor-badge.glitch.me/badge?page_id=request_boost" alt="Go to https://pypi.org/project/request-boost/"/> -->
<img src="https://static.pepy.tech/personalized-badge/request-boost?period=total&units=none&left_color=black&right_color=brightgreen&left_text=Total%20Downloads" alt="Go to https://pypi.org/project/request-boost/"/>
</p>
## Setup
```shell
pip install request-boost
```
**Important:** *Virtual Environment is recommended*
## Usage
```python
# Sample data
number_of_sample_urls = 1000
urls = [ f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls) ]
headers = [{'sample_header':test_no} for test_no in range(number_of_sample_urls)]
```
```python
from request_boost import boosted_requests
results = boosted_requests(urls=urls)
print(results)
```
```python
from request_boost import boosted_requests
results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers, verbose=True, parse_json=True)
print(results)
```
```python
from request_boost import boosted_requests
# Config
number_of_sample_urls = 100
# Sample data
urls = [f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls)]
post_urls = [f'https://postman-echo.com/post' for test_no in range(number_of_sample_urls)]
headers = [{'sample_header': test_no} for test_no in range(number_of_sample_urls)]
data = [{'sample_data': test_no} for test_no in range(number_of_sample_urls)] # Required for POST requests,
#For POST request data can be just list of empty dict but not NONE
simple_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=None, verbose=False, parse_json=True)
header_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers, parse_json=True)
post_results = boosted_requests(urls=post_urls, no_workers=16, max_tries=5, timeout=5, headers=headers, data=data, verbose=True, parse_json=True)
```
## Documentation
<p align="center">
<a href="https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg">
<img src="https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg" width="70%" alt="Go to https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg">
</a>
</p>
```python
boosted_requests(
urls,
no_workers=32,
max_tries=5,
after_max_tries="assert",
timeout=10,
headers=None,
data=None,
verbose=True,
parse_json=True,
)
Get data from APIs in parallel by creating workers that process in the background
:param urls: list of URLS
:param no_workers: maximum number of parallel processes {Default::32}
:param max_tries: Maximum number of tries before failing for a specific URL {Default::5}
:param after_max_tries: What to do if not successfull after "max_tries" for a specific URL,
one of {"assert", "break"} {Default::assert}
:param timeout: Waiting time per request {Default::10}
:param headers: Headers if any for the URL requests
:param data: data if any for the URL requests (Wherever not None a POST request is made)
:param verbose: Show progress [True or False] {Default::True}
:param parse_json: Parse response to json [True or False] {Default::True}
:return: List of response for each API (order is maintained)
```
## Credits
### Maintained by
***Kuldeep Singh Sidhu***
Github: [github/singhsidhukuldeep](https://github.com/singhsidhukuldeep)
`https://github.com/singhsidhukuldeep`
Website: [Kuldeep Singh Sidhu (Website)](http://kuldeepsinghsidhu.com)
`http://kuldeepsinghsidhu.com`
LinkedIn: [Kuldeep Singh Sidhu (LinkedIn)](https://www.linkedin.com/in/singhsidhukuldeep/)
`https://www.linkedin.com/in/singhsidhukuldeep/`
### Contributors
<a href="https://github.com/singhsidhukuldeep/request-boost/graphs/contributors">
<img src="https://contrib.rocks/image?repo=singhsidhukuldeep/request-boost" />
</a>
The full list of all the contributors is available [here](https://github.com/singhsidhukuldeep/request-boost/graphs/contributors)
[](http://kuldeepsinghsidhu.com)
#### TODO:: Want to contribute?
- [ ] Make the backend used (urllib/request library) an option that the user can choose, by default we use urllib
- [ ] For parallel processing add options for multi-processing along with multi-threading
- [ ] Set-up tests for edge cases and changes verification
- [ ] Set-up CI/CD pipleine (possibly using GitHub actions) to publish changes to PyPi
- [ ] Improeve the doc-strings documentation to add more explanantion and examples
- [ ] Add a message queue to deploy the service across machines
- [ ] Add option to run URL requests with Self signed certificate `verify=False`
- [ ] Add option to supress warnings
- [ ] Add progess bars from tqdm and ascii
- [ ] Add option to add session and auth
## Say Thanks
If this helped you in any way, it would be great if you could share it with others.
## Steps for publishing to `pypi` [This is just for me, Maybe!]
- `pip3 install setuptools twine`
- Go to project folder
- `python3 setup.py sdist`
- `twine upload --repository-url https://upload.pypi.org/legacy/ dist/*`
OR
Go to your project folder and:
```shell
pip3 install setuptools twine
python3 setup.py sdist
twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
```
## Current Status
[](https://github.com/singhsidhukuldeep/request-boost/issues)
[](https://github.com/singhsidhukuldeep/request-boost/graphs/contributors)
[](https://github.com/singhsidhukuldeep/request-boost/network/members)
[](https://github.com/singhsidhukuldeep/request-boost/stargazers)
[](https://github.com/singhsidhukuldeep/request-boost/watchers)
[](https://github.com/singhsidhukuldeep/request-boost/branches)
[](https://www.gnu.org/licenses/agpl-3.0)
[](https://www.python.org/)
[](https://github.com/singhsidhukuldeep/request-boost)
[](https://github.com/singhsidhukuldeep?tab=followers)
CHANGE LOG
==============================
> LATEST
0.8 (06/06/2023)
------------------------
- added a new :param `after_max_tries`: What to do if not successfull after `"max_tries"` for a specific URL,
one of {`"assert"`, `"break"`} {Default::`assert`}
> PAST
0.1 (16/09/2021)
------------------------
- Released first version(stable) of request-boost.
0.2 (16/09/2021)
------------------------
- Fixed header issue
- Added POST functionality
0.3.1 (17/09/2021)
------------------------
- Added Verbose
0.4 (17/09/2021)
------------------------
- Added Verbose timing
- Improved performance
0.5 (16/10/2021)
------------------------
- Added Encoding fix
- Improved Readability
0.6 (29/12/2021)
------------------------
- Added option to not parse the JSON
- Improved documentation
0.7 (29/12/2021)
------------------------
- Fixed timeout key error issue, to return `None` for timeouts
- Improved documentation
Read more at https://github.com/singhsidhukuldeep/request-boost
Raw data
{
"_id": null,
"home_page": "https://github.com/singhsidhukuldeep/request-boost",
"name": "request-boost",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "request boost urllib parallel fast REST API",
"author": "Kuldeep Singh Sidhu",
"author_email": "singhsidhukuldeep@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/32/0c/917b744b0dfb4449b8e431147f182ce0f1bbd39ad7083ddfd8a4a7b12b62/request-boost-0.8.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">request-boost </h1>\n\n<p align=\"center\">\n \n <a href=\"https://pypi.org/project/request-boost/\">\n <img src=\"https://github.com/singhsidhukuldeep/request-boost/raw/main/img/benchmark.png\" width=\"1080\" alt=\"Go to https://pypi.org/project/request-boost/\">\n </a>\n</p>\n\n<p align=\"center\">\n<b>A better requests and urllib package.</b></b><br> A simple package for hitting multiple URLs and performing GET/POST requests in parallel.<br>\n<a href=\"https://pypi.org/project/request-boost/\"><img src=\"https://img.shields.io/pypi/pyversions/request-boost\" alt=\"Go to https://pypi.org/project/request-boost/\"/></a>\n<a href=\"https://pypi.org/project/request-boost/\"><img src=\"https://img.shields.io/pypi/v/request-boost\" alt=\"Go to https://pypi.org/project/request-boost/\"/></a>\n<a href=\"https://pypi.org/project/request-boost/\"><img src=\"https://img.shields.io/pypi/status/request-boost\" alt=\"Go to https://pypi.org/project/request-boost/\"/></a>\n<!-- <a href=\"https://pypi.org/project/request-boost/\"><img src=\"https://img.shields.io/pypi/format/request-boost\" alt=\"Go to https://pypi.org/project/request-boost/\"/></a> -->\n<!-- <a href=\"https://lgtm.com/projects/g/singhsidhukuldeep/request-boost/context:python\"><img alt=\"Language grade: Python\" src=\"https://img.shields.io/lgtm/grade/python/g/singhsidhukuldeep/request-boost.svg?logo=lgtm&logoWidth=18\"/></a> -->\n<a href=\"https://pypistats.org/packages/request-boost\"><img src=\"https://img.shields.io/pypi/dm/request-boost\"/></a>\n<!-- <img src=\"https://visitor-badge.glitch.me/badge?page_id=request_boost\" alt=\"Go to https://pypi.org/project/request-boost/\"/> -->\n<img src=\"https://static.pepy.tech/personalized-badge/request-boost?period=total&units=none&left_color=black&right_color=brightgreen&left_text=Total%20Downloads\" alt=\"Go to https://pypi.org/project/request-boost/\"/>\n</p>\n\n## Setup\n\n```shell\npip install request-boost\n```\n\n**Important:** *Virtual Environment is recommended*\n\n## Usage\n\n```python\n# Sample data\nnumber_of_sample_urls = 1000\nurls = [ f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls) ]\nheaders = [{'sample_header':test_no} for test_no in range(number_of_sample_urls)]\n```\n\n```python\nfrom request_boost import boosted_requests\n\nresults = boosted_requests(urls=urls)\nprint(results)\n```\n\n```python\nfrom request_boost import boosted_requests\n\nresults = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers, verbose=True, parse_json=True)\nprint(results)\n```\n\n```python\nfrom request_boost import boosted_requests\n\n# Config\nnumber_of_sample_urls = 100\n\n# Sample data\nurls = [f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls)]\npost_urls = [f'https://postman-echo.com/post' for test_no in range(number_of_sample_urls)]\nheaders = [{'sample_header': test_no} for test_no in range(number_of_sample_urls)]\ndata = [{'sample_data': test_no} for test_no in range(number_of_sample_urls)] # Required for POST requests, \n#For POST request data can be just list of empty dict but not NONE\n\nsimple_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=None, verbose=False, parse_json=True)\nheader_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers, parse_json=True)\npost_results = boosted_requests(urls=post_urls, no_workers=16, max_tries=5, timeout=5, headers=headers, data=data, verbose=True, parse_json=True)\n```\n\n## Documentation\n\n<p align=\"center\">\n <a href=\"https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg\">\n <img src=\"https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg\" width=\"70%\" alt=\"Go to https://github.com/singhsidhukuldeep/request-boost/raw/main/img/structure.jpg\">\n </a>\n</p>\n\n```python\nboosted_requests(\n urls,\n no_workers=32,\n max_tries=5,\n after_max_tries=\"assert\",\n timeout=10,\n headers=None,\n data=None,\n verbose=True,\n parse_json=True,\n)\n\nGet data from APIs in parallel by creating workers that process in the background\n :param urls: list of URLS\n :param no_workers: maximum number of parallel processes {Default::32}\n :param max_tries: Maximum number of tries before failing for a specific URL {Default::5}\n :param after_max_tries: What to do if not successfull after \"max_tries\" for a specific URL, \n one of {\"assert\", \"break\"} {Default::assert}\n :param timeout: Waiting time per request {Default::10}\n :param headers: Headers if any for the URL requests\n :param data: data if any for the URL requests (Wherever not None a POST request is made)\n :param verbose: Show progress [True or False] {Default::True}\n :param parse_json: Parse response to json [True or False] {Default::True}\n :return: List of response for each API (order is maintained)\n```\n\n## Credits\n\n### Maintained by\n\n***Kuldeep Singh Sidhu*** \n\nGithub: [github/singhsidhukuldeep](https://github.com/singhsidhukuldeep)\n`https://github.com/singhsidhukuldeep`\n\nWebsite: [Kuldeep Singh Sidhu (Website)](http://kuldeepsinghsidhu.com)\n`http://kuldeepsinghsidhu.com`\n\nLinkedIn: [Kuldeep Singh Sidhu (LinkedIn)](https://www.linkedin.com/in/singhsidhukuldeep/)\n`https://www.linkedin.com/in/singhsidhukuldeep/`\n\n### Contributors\n\n<a href=\"https://github.com/singhsidhukuldeep/request-boost/graphs/contributors\">\n <img src=\"https://contrib.rocks/image?repo=singhsidhukuldeep/request-boost\" />\n</a>\n\n The full list of all the contributors is available [here](https://github.com/singhsidhukuldeep/request-boost/graphs/contributors)\n\n\n[](http://kuldeepsinghsidhu.com)\n\n#### TODO:: Want to contribute?\n\n- [ ] Make the backend used (urllib/request library) an option that the user can choose, by default we use urllib\n- [ ] For parallel processing add options for multi-processing along with multi-threading\n- [ ] Set-up tests for edge cases and changes verification\n- [ ] Set-up CI/CD pipleine (possibly using GitHub actions) to publish changes to PyPi\n- [ ] Improeve the doc-strings documentation to add more explanantion and examples\n- [ ] Add a message queue to deploy the service across machines\n- [ ] Add option to run URL requests with Self signed certificate `verify=False`\n- [ ] Add option to supress warnings\n- [ ] Add progess bars from tqdm and ascii\n- [ ] Add option to add session and auth\n\n## Say Thanks\n\n If this helped you in any way, it would be great if you could share it with others.\n\n## Steps for publishing to `pypi` [This is just for me, Maybe!]\n\n- `pip3 install setuptools twine`\n- Go to project folder\n- `python3 setup.py sdist`\n- `twine upload --repository-url https://upload.pypi.org/legacy/ dist/*`\n\nOR\n\nGo to your project folder and:\n```shell\npip3 install setuptools twine\n\npython3 setup.py sdist\ntwine upload --repository-url https://upload.pypi.org/legacy/ dist/*\n```\n\n## Current Status\n\n[](https://github.com/singhsidhukuldeep/request-boost/issues)\n[](https://github.com/singhsidhukuldeep/request-boost/graphs/contributors)\n[](https://github.com/singhsidhukuldeep/request-boost/network/members)\n[](https://github.com/singhsidhukuldeep/request-boost/stargazers)\n[](https://github.com/singhsidhukuldeep/request-boost/watchers)\n[](https://github.com/singhsidhukuldeep/request-boost/branches)\n\n[](https://www.gnu.org/licenses/agpl-3.0)\n[](https://www.python.org/)\n[](https://github.com/singhsidhukuldeep/request-boost)\n[](https://github.com/singhsidhukuldeep?tab=followers)\n\n\n\nCHANGE LOG\n==============================\n\n> LATEST\n\n0.8 (06/06/2023)\n------------------------\n- added a new :param `after_max_tries`: What to do if not successfull after `\"max_tries\"` for a specific URL, \n one of {`\"assert\"`, `\"break\"`} {Default::`assert`}\n\n\n> PAST\n\n0.1 (16/09/2021)\n------------------------\n- Released first version(stable) of request-boost.\n\n0.2 (16/09/2021)\n------------------------\n- Fixed header issue\n- Added POST functionality\n\n0.3.1 (17/09/2021)\n------------------------\n- Added Verbose\n\n0.4 (17/09/2021)\n------------------------\n- Added Verbose timing\n- Improved performance\n\n0.5 (16/10/2021)\n------------------------\n- Added Encoding fix\n- Improved Readability\n\n0.6 (29/12/2021)\n------------------------\n- Added option to not parse the JSON\n- Improved documentation\n\n0.7 (29/12/2021)\n------------------------\n- Fixed timeout key error issue, to return `None` for timeouts\n- Improved documentation\n\n\n\nRead more at https://github.com/singhsidhukuldeep/request-boost\n",
"bugtrack_url": null,
"license": "",
"summary": "A better requests and urllib. A simple package for hitting multiple URLs and performing GET/POST requests in parallel.",
"version": "0.8",
"project_urls": {
"Homepage": "https://github.com/singhsidhukuldeep/request-boost"
},
"split_keywords": [
"request",
"boost",
"urllib",
"parallel",
"fast",
"rest",
"api"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "320c917b744b0dfb4449b8e431147f182ce0f1bbd39ad7083ddfd8a4a7b12b62",
"md5": "5bb7992ec5de1884f314ba5bb2172bfb",
"sha256": "1d0d9b89776f119414b1cab1ae462978810645559c16956f7ae06b579b3bbbd8"
},
"downloads": -1,
"filename": "request-boost-0.8.tar.gz",
"has_sig": false,
"md5_digest": "5bb7992ec5de1884f314ba5bb2172bfb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 19395,
"upload_time": "2023-06-06T06:56:27",
"upload_time_iso_8601": "2023-06-06T06:56:27.351204Z",
"url": "https://files.pythonhosted.org/packages/32/0c/917b744b0dfb4449b8e431147f182ce0f1bbd39ad7083ddfd8a4a7b12b62/request-boost-0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-06 06:56:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "singhsidhukuldeep",
"github_project": "request-boost",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "request-boost"
}
JSON
