# Resilient Caller
A Python package that provides a customizable wrapper to retry function calls with custom logic. This package was developed to address the need for executing numerous requests with similar, yet slightly different, exception handling. The wrapper reduces the need to write multiple while loops and try/except blocks for each request.
The wrapper can be implemented for any function, not just requests-related functions. The module also includes a Python requests implementation with auto proxy formatting from a string, by simply passing the proxy as a string to the send_request function. Async functions support as well.
The resilient caller supports the following keyword arguments (kwargs) for the wrapper. Note that these kwargs will not be passed to the wrapped function:
- `conditions`: A dictionary specifying the actions to take for a given outcome.
- `conditions_criteria`: The criteria to use when checking the `conditions`.
- `exceptions`: A dictionary specifying the actions to take for a given exception or 'all' for non-handled or all exceptions.
- `retries`: The maximum number of times to retry the function (disabled by default).
- `delay`: The number of seconds to sleep between retries.
- `on_retry`: A callback function to execute on retry, for example, a log function.
In the module, a Python requests implementation is provided with automatic proxy formatting from a string.
Please refer to the usage examples below and the examples folder in the repository for more information on how to use the resilient caller.
## Installation
Install the package using pip:
```
pip install resilient_caller
```
## Examples
### Quick simple and easy web scraping monitor
In this example, we will create a `SimpleScraper` class that monitors [glizzykingdreko's medium blog](https://medium.com/@glizzykingdreko) for new articles. By defining the start method with the `@resilient_call()` decorator and passing `delay=5` and `exceptions={"all": RETRY_EVENT}` when calling it, we ensure that the scraper handles all exceptions with a retry and adds a delay of 5 seconds between each request.
```python
from bs4 import BeautifulSoup
from requests import Session
from typing import List
from time import sleep
from resilenter_caller import resilient_call, RETRY_EVENT
class SimpleScraper:
def __init__(self):
self.session, self.articles = Session(), []
# This is making the function run through a while loop with a delay of 5 seconds
# and handling any exception with a retry event.
self.start(delay=5, exceptions={"all": RETRY_EVENT})
@resilient_call()
def start(self) -> None:
data = self.load_api_data()
self.load_response_details(data)
def load_response_details(self, response: str) -> List[str]:
data = BeautifulSoup(response, "html.parser")
new_articles = [
[d.find("h2").text, d.find("a").get("href")]
for d in data.find_all("article")
if d.find("h2").text not in self.articles
]
for article in new_articles:
name, url = article
print(f"New glizzykingdreko's article on medium \"{name}\"! Check it out at {url}.")
self.articles.append(name)
return new_articles
def load_api_data(self) -> str:
url = "https://medium.com/@glizzykingdreko"
return self.session.get(url).text
if __name__ == "__main__":
SimpleScraper()
```
### Web scraping with retry, custom handling and handling for all exceptions
In this example, we will use the send_request() function provided in the module to perform
web scraping. We will also use all the available options to customize the handling of
different HTTP response codes and handle all exceptions.
```python
from bs4 import BeautifulSoup
from resilenter_caller import send_request, RETRY_EVENT
def handle_success(response):
print(f"Request successful, status code: {response.status_code}")
soup = BeautifulSoup(response.text, "html.parser")
title = soup.find("title")
print(f"Page title: {title.string}")
return response
def handle_not_found(response):
print(f"Page not found, status code: {response.status_code}")
return RETRY_EVENT
def handle_server_error(response):
print(f"Server error, status code: {response.status_code}")
return RETRY_EVENT
def handle_all_exceptions(exception):
print(f"An exception occurred: {type(exception).__name__} - {exception}")
return RETRY_EVENT
if __name__ == "__main__":
response = send_request(
"https://www.example.com",
retries=3,
delay=2,
conditions={200: handle_success, 404: handle_not_found, 500: handle_server_error},
exceptions={"all": handle_all_exceptions},
on_retry=lambda tries: print(f"Retry {tries}")
)
```
### File processing with retry and custom handling
In this example, we will use the resilient_call() decorator to implement a function
that processes a file and retries the operation in case of failure. We will also
use all the available options to customize the handling of different file sizes.
```python
import os
from resilenter_caller import resilient_call, RETRY_EVENT
def process_large_file(file_path):
print(f"Processing large file: {file_path}")
return RETRY_EVENT
def process_small_file(file_path):
print(f"Processing small file: {file_path}")
return RETRY_EVENT
def process_valid_file(file_path):
print(f"Processing valid file: {file_path}")
return file_path
@resilient_call()
def process_file(file_path):
file_size = os.path.getsize(file_path)
return file_size
if __name__ == "__main__":
processed_file = process_file(
"example.txt",
retries=5,
delay=2,
conditions={-1: process_large_file, 1: process_small_file},
conditions_criteria=lambda file_size: -1 if file_size > 1000000 else 1 if file_size < 1000 else 0,
on_retry=lambda tries: print(f"Retry {tries}")
)
```
### Asynchronous API call with rate limiting retry
In this example, we will use the resilient_call() decorator to implement an
asynchronous function that makes an API call and retries the call in case of failure or rate limiting.
```python
import aiohttp, asyncio
from resilenter_caller import resilient_call, RETRY_EVENT
async def handle_rate_limit(e):
print(f"Rate limited: {e}")
return RETRY_EVENT
@resilient_call()
async def async_api_call(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
if response.status == 429:
raise Exception("Rate limited")
data = await response.json()
return data
async def main():
await async_api_call(
"https://httpbin.org/status/429",
retries=3,
delay=5,
exceptions={Exception: handle_rate_limit}
)
if __name__ == "__main__":
asyncio.run(main())
```
### Custom backoff strategy (exponential backoff)
In this example, we will create a function that will randomly fail with a 30% chance.
If it fails, we will retry the function with an exponential backoff strategy.
```python
import logging
import random
from resilenter_caller import resilient_call, RETRY_EVENT
# Set level as debug to get full logs
logging.basicConfig(level=logging.DEBUG)
def exponential_backoff(tries):
return 2 ** (tries - 1) + random.uniform(0, 1)
# If 2 arguments are passed to an exception function
# the second argument will be the number of tries
# (same thing for a condition)
def print_exception(exception, tries):
print(f"Exception: {exception} (try {tries})")
return RETRY_EVENT
@resilient_call()
def example_function():
random_num = random.random()
if random_num < 0.7:
print("Failed, retrying...")
raise ValueError("Random number too low")
else:
print("Success!")
return "Successful response"
if __name__ == '__main__':
result = example_function(
retries=5,
on_retry=exponential_backoff,
exceptions={ValueError: print_exception}
)
print("Result:", result)
```
### Pass the number of tries to the action function
In this example, by using a function that takes 2 arguments, we
can pass the number of tries to the action function.
```python
import random
from resilenter_caller import resilient_call, RETRY_EVENT
def some_condition(response):
return response == "Retry"
def handle_response(response, tries):
if tries < 3 and some_condition(response):
return RETRY_EVENT
else:
return response
# We set the max execution time to 10 seconds
@resilient_call(max_elapsed_time=10)
def example_function():
random_num = random.random()
if random_num < 0.6:
print("Returning 'Retry'")
return "Retry"
else:
print("Success!")
return "Successful response"
if __name__ == '__main__':
result = example_function(
# With 'all' we can handle all the
# possible responses or exceptions
conditions={'all': handle_response},
)
print("Result:", result)
```
These examples demonstrate the versatility and usefulness of the Resilient Caller module. This module can be applied to a wide range of use cases, from web scraping and file processing to API calls and custom backoff strategies. Make sure to explore the examples folder in the repository for even more examples and use cases with other parameters and configurations.
## Personal Thoughts
I hope this module will help many developers save time and make their code more efficient. Please feel free to contact me for any help or suggestions via [Email](mailto:glizzykingdreko@protonmail.com) or [Twitter](https://mobile.twitter.com/glizzykingdreko). Don't forget to follow me on GitHub and Medium for more exciting content and updates. I appreciate your feedback and contributions to the project.
## Contributing
I welcome contributions to the Resilient Caller project! To contribute, please follow these steps:
- Fork the repository on GitHub.
- Create a new branch with a descriptive name.
- Make your changes, add new features, or fix bugs.
- Write tests to ensure that your changes work as expected.
- Update the documentation and examples to reflect your changes.
- Commit your changes and create a pull request.
Please make sure to follow the existing code style and provide clear, concise commit messages. If you have any questions, feel free to open an issue, and we'll be happy to help.
## License
This project is licensed under the MIT [License](LICENSE). See the LICENSE file for more details.
## My links
- [Project repository](https://github.com/glizzykingdreko/resilient_caller)
- [GitHub](https://github.com/glizzykingdreko)
- [Twitter](https://mobile.twitter.com/glizzykingdreko)
- [Medium](https://medium.com/@glizzykingdreko)
- [Email](mailto:glizzykingdreko@protonmail.com)
Raw data
{
"_id": null,
"home_page": "https://github.com/glizzykingdreko/resilient_caller",
"name": "resilient-caller",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "glizzykingdreko",
"author_email": "glizzykingdreko@protonmail.com",
"download_url": "https://files.pythonhosted.org/packages/8a/a3/2c7dba5075dc706b84f6bc4636cf84f9b3a61471600c5863947598a9593f/resilient_caller-0.2.1.tar.gz",
"platform": null,
"description": "# Resilient Caller\n\nA Python package that provides a customizable wrapper to retry function calls with custom logic. This package was developed to address the need for executing numerous requests with similar, yet slightly different, exception handling. The wrapper reduces the need to write multiple while loops and try/except blocks for each request.\n\nThe wrapper can be implemented for any function, not just requests-related functions. The module also includes a Python requests implementation with auto proxy formatting from a string, by simply passing the proxy as a string to the send_request function. Async functions support as well.\n\nThe resilient caller supports the following keyword arguments (kwargs) for the wrapper. Note that these kwargs will not be passed to the wrapped function:\n\n- `conditions`: A dictionary specifying the actions to take for a given outcome.\n- `conditions_criteria`: The criteria to use when checking the `conditions`.\n- `exceptions`: A dictionary specifying the actions to take for a given exception or 'all' for non-handled or all exceptions.\n- `retries`: The maximum number of times to retry the function (disabled by default).\n- `delay`: The number of seconds to sleep between retries.\n- `on_retry`: A callback function to execute on retry, for example, a log function.\nIn the module, a Python requests implementation is provided with automatic proxy formatting from a string.\n\nPlease refer to the usage examples below and the examples folder in the repository for more information on how to use the resilient caller.\n \n## Installation\nInstall the package using pip:\n```\npip install resilient_caller\n```\n\n## Examples\n\n### Quick simple and easy web scraping monitor\nIn this example, we will create a `SimpleScraper` class that monitors [glizzykingdreko's medium blog](https://medium.com/@glizzykingdreko) for new articles. By defining the start method with the `@resilient_call()` decorator and passing `delay=5` and `exceptions={\"all\": RETRY_EVENT}` when calling it, we ensure that the scraper handles all exceptions with a retry and adds a delay of 5 seconds between each request.\n```python\nfrom bs4 import BeautifulSoup\nfrom requests import Session\nfrom typing import List\nfrom time import sleep\nfrom resilenter_caller import resilient_call, RETRY_EVENT\n\nclass SimpleScraper:\n def __init__(self):\n self.session, self.articles = Session(), []\n # This is making the function run through a while loop with a delay of 5 seconds\n # and handling any exception with a retry event.\n self.start(delay=5, exceptions={\"all\": RETRY_EVENT})\n\n @resilient_call()\n def start(self) -> None:\n data = self.load_api_data()\n self.load_response_details(data)\n\n def load_response_details(self, response: str) -> List[str]:\n data = BeautifulSoup(response, \"html.parser\")\n new_articles = [\n [d.find(\"h2\").text, d.find(\"a\").get(\"href\")] \n for d in data.find_all(\"article\") \n if d.find(\"h2\").text not in self.articles\n ]\n for article in new_articles:\n name, url = article\n print(f\"New glizzykingdreko's article on medium \\\"{name}\\\"! Check it out at {url}.\")\n self.articles.append(name)\n return new_articles\n\n def load_api_data(self) -> str:\n url = \"https://medium.com/@glizzykingdreko\"\n return self.session.get(url).text\n\nif __name__ == \"__main__\":\n SimpleScraper()\n\n```\n### Web scraping with retry, custom handling and handling for all exceptions\nIn this example, we will use the send_request() function provided in the module to perform \nweb scraping. We will also use all the available options to customize the handling of \ndifferent HTTP response codes and handle all exceptions.\n```python\nfrom bs4 import BeautifulSoup\nfrom resilenter_caller import send_request, RETRY_EVENT\n\ndef handle_success(response):\n print(f\"Request successful, status code: {response.status_code}\")\n soup = BeautifulSoup(response.text, \"html.parser\")\n title = soup.find(\"title\")\n print(f\"Page title: {title.string}\")\n return response\n\ndef handle_not_found(response):\n print(f\"Page not found, status code: {response.status_code}\")\n return RETRY_EVENT\n\ndef handle_server_error(response):\n print(f\"Server error, status code: {response.status_code}\")\n return RETRY_EVENT\n\ndef handle_all_exceptions(exception):\n print(f\"An exception occurred: {type(exception).__name__} - {exception}\")\n return RETRY_EVENT\n\nif __name__ == \"__main__\":\n response = send_request(\n \"https://www.example.com\",\n retries=3,\n delay=2,\n conditions={200: handle_success, 404: handle_not_found, 500: handle_server_error},\n exceptions={\"all\": handle_all_exceptions},\n on_retry=lambda tries: print(f\"Retry {tries}\")\n )\n```\n\n### File processing with retry and custom handling\nIn this example, we will use the resilient_call() decorator to implement a function \nthat processes a file and retries the operation in case of failure. We will also \nuse all the available options to customize the handling of different file sizes.\n\n```python\nimport os\nfrom resilenter_caller import resilient_call, RETRY_EVENT\n\ndef process_large_file(file_path):\n print(f\"Processing large file: {file_path}\")\n return RETRY_EVENT\n\ndef process_small_file(file_path):\n print(f\"Processing small file: {file_path}\")\n return RETRY_EVENT\n\ndef process_valid_file(file_path):\n print(f\"Processing valid file: {file_path}\")\n return file_path\n\n@resilient_call()\ndef process_file(file_path):\n file_size = os.path.getsize(file_path)\n return file_size\n\nif __name__ == \"__main__\":\n processed_file = process_file(\n \"example.txt\",\n retries=5,\n delay=2,\n conditions={-1: process_large_file, 1: process_small_file},\n conditions_criteria=lambda file_size: -1 if file_size > 1000000 else 1 if file_size < 1000 else 0,\n on_retry=lambda tries: print(f\"Retry {tries}\")\n )\n```\n\n### Asynchronous API call with rate limiting retry\nIn this example, we will use the resilient_call() decorator to implement an \nasynchronous function that makes an API call and retries the call in case of failure or rate limiting.\n```python\nimport aiohttp, asyncio\nfrom resilenter_caller import resilient_call, RETRY_EVENT\n\nasync def handle_rate_limit(e):\n print(f\"Rate limited: {e}\")\n return RETRY_EVENT\n\n@resilient_call()\nasync def async_api_call(url):\n async with aiohttp.ClientSession() as session:\n async with session.get(url) as response:\n if response.status == 429:\n raise Exception(\"Rate limited\")\n data = await response.json()\n return data\n\nasync def main():\n await async_api_call(\n \"https://httpbin.org/status/429\",\n retries=3, \n delay=5,\n exceptions={Exception: handle_rate_limit}\n )\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n### Custom backoff strategy (exponential backoff)\nIn this example, we will create a function that will randomly fail with a 30% chance.\nIf it fails, we will retry the function with an exponential backoff strategy.\n```python\nimport logging\nimport random\nfrom resilenter_caller import resilient_call, RETRY_EVENT\n\n# Set level as debug to get full logs\nlogging.basicConfig(level=logging.DEBUG)\n\ndef exponential_backoff(tries):\n return 2 ** (tries - 1) + random.uniform(0, 1)\n\n# If 2 arguments are passed to an exception function\n# the second argument will be the number of tries\n# (same thing for a condition)\ndef print_exception(exception, tries):\n print(f\"Exception: {exception} (try {tries})\")\n return RETRY_EVENT\n\n@resilient_call()\ndef example_function():\n random_num = random.random()\n if random_num < 0.7:\n print(\"Failed, retrying...\")\n raise ValueError(\"Random number too low\")\n else:\n print(\"Success!\")\n return \"Successful response\"\n\nif __name__ == '__main__':\n result = example_function(\n retries=5, \n on_retry=exponential_backoff,\n exceptions={ValueError: print_exception}\n )\n print(\"Result:\", result)\n```\n\n### Pass the number of tries to the action function\nIn this example, by using a function that takes 2 arguments, we \ncan pass the number of tries to the action function.\n```python\nimport random\nfrom resilenter_caller import resilient_call, RETRY_EVENT\n\ndef some_condition(response):\n return response == \"Retry\"\n\ndef handle_response(response, tries):\n if tries < 3 and some_condition(response):\n return RETRY_EVENT\n else:\n return response\n\n# We set the max execution time to 10 seconds\n@resilient_call(max_elapsed_time=10)\ndef example_function():\n random_num = random.random()\n if random_num < 0.6:\n print(\"Returning 'Retry'\")\n return \"Retry\"\n else:\n print(\"Success!\")\n return \"Successful response\"\n\nif __name__ == '__main__':\n result = example_function(\n # With 'all' we can handle all the\n # possible responses or exceptions\n conditions={'all': handle_response},\n )\n print(\"Result:\", result)\n```\nThese examples demonstrate the versatility and usefulness of the Resilient Caller module. This module can be applied to a wide range of use cases, from web scraping and file processing to API calls and custom backoff strategies. Make sure to explore the examples folder in the repository for even more examples and use cases with other parameters and configurations.\n\n## Personal Thoughts\n\nI hope this module will help many developers save time and make their code more efficient. Please feel free to contact me for any help or suggestions via [Email](mailto:glizzykingdreko@protonmail.com) or [Twitter](https://mobile.twitter.com/glizzykingdreko). Don't forget to follow me on GitHub and Medium for more exciting content and updates. I appreciate your feedback and contributions to the project.\n\n## Contributing\nI welcome contributions to the Resilient Caller project! To contribute, please follow these steps:\n\n- Fork the repository on GitHub.\n- Create a new branch with a descriptive name.\n- Make your changes, add new features, or fix bugs.\n- Write tests to ensure that your changes work as expected.\n- Update the documentation and examples to reflect your changes.\n- Commit your changes and create a pull request.\nPlease make sure to follow the existing code style and provide clear, concise commit messages. If you have any questions, feel free to open an issue, and we'll be happy to help.\n\n## License\nThis project is licensed under the MIT [License](LICENSE). See the LICENSE file for more details.\n\n## My links\n- [Project repository](https://github.com/glizzykingdreko/resilient_caller)\n- [GitHub](https://github.com/glizzykingdreko)\n- [Twitter](https://mobile.twitter.com/glizzykingdreko)\n- [Medium](https://medium.com/@glizzykingdreko)\n- [Email](mailto:glizzykingdreko@protonmail.com)\n",
"bugtrack_url": null,
"license": "",
"summary": "A Python package to retry function calls with custom logic and handling.",
"version": "0.2.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ce5b2c89355d6d5bdb37dd41a55d1020c74f66d9fb1b871b674f9a9cd8b8225f",
"md5": "f7515e622ce5f53149335fa179c25797",
"sha256": "32b73e14a7d23fc0baa8fbc3a61fb26c28b59b6f62bba7d79d1ec39da212d304"
},
"downloads": -1,
"filename": "resilient_caller-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f7515e622ce5f53149335fa179c25797",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 11891,
"upload_time": "2023-03-29T23:29:21",
"upload_time_iso_8601": "2023-03-29T23:29:21.393363Z",
"url": "https://files.pythonhosted.org/packages/ce/5b/2c89355d6d5bdb37dd41a55d1020c74f66d9fb1b871b674f9a9cd8b8225f/resilient_caller-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8aa32c7dba5075dc706b84f6bc4636cf84f9b3a61471600c5863947598a9593f",
"md5": "bc3416d8de6722b504a80fe8e15d5e01",
"sha256": "ded59a4592dfa2a4995e42e5b03be65eed7a68b21302da9a7500a57f157e3d0e"
},
"downloads": -1,
"filename": "resilient_caller-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "bc3416d8de6722b504a80fe8e15d5e01",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 11148,
"upload_time": "2023-03-29T23:29:24",
"upload_time_iso_8601": "2023-03-29T23:29:24.085494Z",
"url": "https://files.pythonhosted.org/packages/8a/a3/2c7dba5075dc706b84f6bc4636cf84f9b3a61471600c5863947598a9593f/resilient_caller-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-29 23:29:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "glizzykingdreko",
"github_project": "resilient_caller",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "resilient-caller"
}
JSON
