# fetch-cache
🚧 警告：这是一个"能用就行"的项目！
💡 如果发现 bug，那一定是特性！
🔧 代码写得不够优雅？随时欢迎 PR！
🎯 目标是：能用 > 好用 > 很好用
一个支持多种缓存后端的 HTTP 客户端库，基于 httpx 开发，提供同步和异步支持,本项目是自用代码整理后的开源版本，功能可能不够完善，但核心功能已经可以正常使用。 欢迎根据实际需求修改代码，如果对你有帮助，请点个星！。
## 特性
- 基于 httpx 的现代 HTTP 客户端
- 支持同步和异步请求
- 内置多种缓存后端:
- 内存缓存 (默认)
- 文件缓存
- Redis 缓存
- SQL 缓存 (支持 MySQL、PostgreSQL、SQLite、MariaDB)
- MongoDB 缓存 (实验性)
- Django 缓存 (实验性)
- 灵活的缓存配置
## 安装
```bash
pip install fetch_cache
```
## 基础使用
### 同步客户端
```python
from fetch_cache import HTTPClient
# 创建客户端实例
client = HTTPClient(
base_url="https://api.example.com",
headers={"Authorization": "Bearer token"},
cache_type="memory", # 默认使用内存缓存
cache_ttl=3600, # 缓存有效期(秒)
)
# 发送请求
response = client.request(
method="GET",
endpoint="/users",
params={"page": 1}
)
# 不使用缓存
response = client.request(
method="GET",
endpoint="/users",
no_cache=True
)
# 关闭客户端
client.close()
```
### 异步客户端
```python
import asyncio
from fetch_cache import AsyncHTTPClient
async def main():
# 创建异步客户端实例
client = AsyncHTTPClient(
base_url="https://api.example.com",
headers={"Authorization": "Bearer token"},
cache_type="redis",
cache_config={
"host": "localhost",
"port": 6379,
"db": 0
},
cache_ttl=3600
)
# 发送异步请求
response = await client.request(
method="GET",
endpoint="/users",
params={"page": 1}
)
# 关闭客户端
await client.close()
asyncio.run(main())
```
## 缓存配置
### 内存缓存 (默认)
最简单的缓存实现,数据存储在内存中,程序重启后缓存会丢失。
### 文件缓存
将缓存数据以 JSON 格式存储在本地文件系统中。
### Redis 缓存
使用 Redis 作为缓存后端,需要先安装 redis 包:
```bash
pip install redis
```
配置示例:
```python
client = HTTPClient(
base_url="https://api.example.com",
cache_type="redis",
cache_config={
"host": "localhost",
"port": 6379,
"db": 0,
"password": None,
"socket_timeout": 5,
"connection_pool": {
"max_connections": 10
}
},
cache_ttl=3600
)
```
### SQL 缓存
支持 MySQL、PostgreSQL、SQLite 和 MariaDB,需要先安装对应的数据库驱动:
```bash
# MySQL
pip install pymysql
# PostgreSQL
pip install psycopg2-binary
# MariaDB
pip install mariadb
# SQLite (Python 内置)
```
配置示例:
```python
client = HTTPClient(
base_url="https://api.example.com",
cache_type="mysql", # 或 postgresql、sqlite、mariadb
cache_config={
"engine_url": "mysql+pymysql://user:pass@localhost/dbname",
"table_name": "http_cache", # 可选,默认为 http_cache
"pool_size": 5,
"max_overflow": 10,
"pool_timeout": 30,
"pool_recycle": 1800
},
cache_ttl=3600
)
```
### 扩展
```python
from fetch_cache import HTTPClient
from datetime import datetime
import time
class FC(HTTPClient):
def field_list(self, params: dict = None):
"""获取字段列表"""
response = self.get("field_list", params=params, no_cache=True)
return {str(i["id"]): i["name"] for i in response["data"]}
http_big_client = FC(
base_url=f"http://11111.11.11/kong/user/",
headers={"Authorization": "bearer xxxx"},
cache_type="file",
cache_config={"cache_dir": "./cache"},
cache_ttl=10,
)
if __name__ == "__main__":
resp = http_big_client.field_list()
print(resp)
resp = http_big_client.field_list()
print(resp)
```
各数据库的 engine_url 格式:
- MySQL: `mysql+pymysql://user:pass@localhost/dbname`
- PostgreSQL: `postgresql+psycopg2://user:pass@localhost/dbname`
- SQLite: `sqlite:///path/to/cache.db`
- MariaDB: `mariadb+mariadb://user:pass@localhost/dbname`
Raw data
{
"_id": null,
"home_page": "https://github.com/xiwen-haochi/fetch_cache",
"name": "fetch_cache",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "http, cache, httpx, db, sql, redis, mongo, django",
"author": "'fkl'",
"author_email": "18654198191@163.com",
"download_url": "https://files.pythonhosted.org/packages/b5/25/2b5f84149e1ea3c2388b6887b64cb574b9d38074a891d242c7ec49e39337/fetch_cache-0.1.3.tar.gz",
"platform": null,
"description": "# fetch-cache\n\ud83d\udea7 \u8b66\u544a\uff1a\u8fd9\u662f\u4e00\u4e2a\"\u80fd\u7528\u5c31\u884c\"\u7684\u9879\u76ee\uff01\n\n\ud83d\udca1 \u5982\u679c\u53d1\u73b0 bug\uff0c\u90a3\u4e00\u5b9a\u662f\u7279\u6027\uff01\n\n\ud83d\udd27 \u4ee3\u7801\u5199\u5f97\u4e0d\u591f\u4f18\u96c5\uff1f\u968f\u65f6\u6b22\u8fce PR\uff01\n\n\ud83c\udfaf \u76ee\u6807\u662f\uff1a\u80fd\u7528 > \u597d\u7528 > \u5f88\u597d\u7528\n\n\n\n\u4e00\u4e2a\u652f\u6301\u591a\u79cd\u7f13\u5b58\u540e\u7aef\u7684 HTTP \u5ba2\u6237\u7aef\u5e93\uff0c\u57fa\u4e8e httpx \u5f00\u53d1\uff0c\u63d0\u4f9b\u540c\u6b65\u548c\u5f02\u6b65\u652f\u6301,\u672c\u9879\u76ee\u662f\u81ea\u7528\u4ee3\u7801\u6574\u7406\u540e\u7684\u5f00\u6e90\u7248\u672c\uff0c\u529f\u80fd\u53ef\u80fd\u4e0d\u591f\u5b8c\u5584\uff0c\u4f46\u6838\u5fc3\u529f\u80fd\u5df2\u7ecf\u53ef\u4ee5\u6b63\u5e38\u4f7f\u7528\u3002 \u6b22\u8fce\u6839\u636e\u5b9e\u9645\u9700\u6c42\u4fee\u6539\u4ee3\u7801\uff0c\u5982\u679c\u5bf9\u4f60\u6709\u5e2e\u52a9\uff0c\u8bf7\u70b9\u4e2a\u661f\uff01\u3002\n\n## \u7279\u6027\n\n- \u57fa\u4e8e httpx \u7684\u73b0\u4ee3 HTTP \u5ba2\u6237\u7aef\n- \u652f\u6301\u540c\u6b65\u548c\u5f02\u6b65\u8bf7\u6c42\n- \u5185\u7f6e\u591a\u79cd\u7f13\u5b58\u540e\u7aef:\n - \u5185\u5b58\u7f13\u5b58 (\u9ed8\u8ba4)\n - \u6587\u4ef6\u7f13\u5b58\n - Redis \u7f13\u5b58\n - SQL \u7f13\u5b58 (\u652f\u6301 MySQL\u3001PostgreSQL\u3001SQLite\u3001MariaDB)\n - MongoDB \u7f13\u5b58 (\u5b9e\u9a8c\u6027)\n - Django \u7f13\u5b58 (\u5b9e\u9a8c\u6027)\n- \u7075\u6d3b\u7684\u7f13\u5b58\u914d\u7f6e\n\n\n## \u5b89\u88c5\n\n```bash\npip install fetch_cache\n```\n\n## \u57fa\u7840\u4f7f\u7528\n\n### \u540c\u6b65\u5ba2\u6237\u7aef\n\n```python\nfrom fetch_cache import HTTPClient\n\n# \u521b\u5efa\u5ba2\u6237\u7aef\u5b9e\u4f8b\nclient = HTTPClient(\n base_url=\"https://api.example.com\",\n headers={\"Authorization\": \"Bearer token\"},\n cache_type=\"memory\", # \u9ed8\u8ba4\u4f7f\u7528\u5185\u5b58\u7f13\u5b58\n cache_ttl=3600, # \u7f13\u5b58\u6709\u6548\u671f(\u79d2)\n)\n\n# \u53d1\u9001\u8bf7\u6c42\nresponse = client.request(\n method=\"GET\",\n endpoint=\"/users\",\n params={\"page\": 1}\n)\n\n# \u4e0d\u4f7f\u7528\u7f13\u5b58\nresponse = client.request(\n method=\"GET\",\n endpoint=\"/users\",\n no_cache=True\n)\n\n# \u5173\u95ed\u5ba2\u6237\u7aef\nclient.close()\n```\n\n### \u5f02\u6b65\u5ba2\u6237\u7aef\n\n```python\nimport asyncio\nfrom fetch_cache import AsyncHTTPClient\n\nasync def main():\n # \u521b\u5efa\u5f02\u6b65\u5ba2\u6237\u7aef\u5b9e\u4f8b\n client = AsyncHTTPClient(\n base_url=\"https://api.example.com\",\n headers={\"Authorization\": \"Bearer token\"},\n cache_type=\"redis\",\n cache_config={\n \"host\": \"localhost\",\n \"port\": 6379,\n \"db\": 0\n },\n cache_ttl=3600\n )\n\n # \u53d1\u9001\u5f02\u6b65\u8bf7\u6c42\n response = await client.request(\n method=\"GET\",\n endpoint=\"/users\",\n params={\"page\": 1}\n )\n\n # \u5173\u95ed\u5ba2\u6237\u7aef\n await client.close()\n\nasyncio.run(main())\n```\n\n## \u7f13\u5b58\u914d\u7f6e\n\n### \u5185\u5b58\u7f13\u5b58 (\u9ed8\u8ba4)\n\n\u6700\u7b80\u5355\u7684\u7f13\u5b58\u5b9e\u73b0,\u6570\u636e\u5b58\u50a8\u5728\u5185\u5b58\u4e2d,\u7a0b\u5e8f\u91cd\u542f\u540e\u7f13\u5b58\u4f1a\u4e22\u5931\u3002\n\n### \u6587\u4ef6\u7f13\u5b58\n\n\u5c06\u7f13\u5b58\u6570\u636e\u4ee5 JSON \u683c\u5f0f\u5b58\u50a8\u5728\u672c\u5730\u6587\u4ef6\u7cfb\u7edf\u4e2d\u3002\n\n### Redis \u7f13\u5b58\n\n\u4f7f\u7528 Redis \u4f5c\u4e3a\u7f13\u5b58\u540e\u7aef,\u9700\u8981\u5148\u5b89\u88c5 redis \u5305:\n\n```bash\npip install redis\n```\n\n\u914d\u7f6e\u793a\u4f8b:\n\n```python\nclient = HTTPClient(\n base_url=\"https://api.example.com\",\n cache_type=\"redis\",\n cache_config={\n \"host\": \"localhost\",\n \"port\": 6379,\n \"db\": 0,\n \"password\": None,\n \"socket_timeout\": 5,\n \"connection_pool\": {\n \"max_connections\": 10\n }\n },\n cache_ttl=3600\n)\n```\n\n### SQL \u7f13\u5b58\n\n\u652f\u6301 MySQL\u3001PostgreSQL\u3001SQLite \u548c MariaDB,\u9700\u8981\u5148\u5b89\u88c5\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u9a71\u52a8:\n\n```bash\n# MySQL\npip install pymysql\n\n# PostgreSQL \npip install psycopg2-binary\n\n# MariaDB\npip install mariadb\n\n# SQLite (Python \u5185\u7f6e)\n```\n\n\u914d\u7f6e\u793a\u4f8b:\n\n```python\nclient = HTTPClient(\n base_url=\"https://api.example.com\",\n cache_type=\"mysql\", # \u6216 postgresql\u3001sqlite\u3001mariadb\n cache_config={\n \"engine_url\": \"mysql+pymysql://user:pass@localhost/dbname\",\n \"table_name\": \"http_cache\", # \u53ef\u9009,\u9ed8\u8ba4\u4e3a http_cache\n \"pool_size\": 5,\n \"max_overflow\": 10,\n \"pool_timeout\": 30,\n \"pool_recycle\": 1800\n },\n cache_ttl=3600\n)\n```\n### \u6269\u5c55\n```python\nfrom fetch_cache import HTTPClient\nfrom datetime import datetime\nimport time\n\n\n\nclass FC(HTTPClient):\n def field_list(self, params: dict = None):\n \"\"\"\u83b7\u53d6\u5b57\u6bb5\u5217\u8868\"\"\"\n response = self.get(\"field_list\", params=params, no_cache=True)\n return {str(i[\"id\"]): i[\"name\"] for i in response[\"data\"]}\n\n\nhttp_big_client = FC(\n base_url=f\"http://11111.11.11/kong/user/\",\n headers={\"Authorization\": \"bearer xxxx\"},\n cache_type=\"file\",\n cache_config={\"cache_dir\": \"./cache\"},\n cache_ttl=10,\n)\n\n\nif __name__ == \"__main__\":\n resp = http_big_client.field_list()\n print(resp)\n resp = http_big_client.field_list()\n print(resp)\n\n```\n\u5404\u6570\u636e\u5e93\u7684 engine_url \u683c\u5f0f:\n\n- MySQL: `mysql+pymysql://user:pass@localhost/dbname`\n- PostgreSQL: `postgresql+psycopg2://user:pass@localhost/dbname`\n- SQLite: `sqlite:///path/to/cache.db`\n- MariaDB: `mariadb+mariadb://user:pass@localhost/dbname`\n\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An HTTP client with multiple cache backends to optimize repeated request performance",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/xiwen-haochi/fetch_cache",
"Repository": "https://github.com/xiwen-haochi/fetch_cache"
},
"split_keywords": [
"http",
" cache",
" httpx",
" db",
" sql",
" redis",
" mongo",
" django"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2c40b8705073149ee34b5438f5af5936be8c9069e43848172a3db585c92dad8b",
"md5": "e5f41b908ab85c8639fc4889b4481623",
"sha256": "e3ada7f457e312143174d83b8fe934e3d35162ee118b6b5b1f690c9f095aea7f"
},
"downloads": -1,
"filename": "fetch_cache-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e5f41b908ab85c8639fc4889b4481623",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16156,
"upload_time": "2024-11-22T05:58:50",
"upload_time_iso_8601": "2024-11-22T05:58:50.827344Z",
"url": "https://files.pythonhosted.org/packages/2c/40/b8705073149ee34b5438f5af5936be8c9069e43848172a3db585c92dad8b/fetch_cache-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b5252b5f84149e1ea3c2388b6887b64cb574b9d38074a891d242c7ec49e39337",
"md5": "90096f52fface36143a480e2cf98b7f8",
"sha256": "e95ce0b5c1af7d5f8a0dfa2b7660ca6a9886de1db2cdd7fcedcad457708b4758"
},
"downloads": -1,
"filename": "fetch_cache-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "90096f52fface36143a480e2cf98b7f8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 20264,
"upload_time": "2024-11-22T05:58:51",
"upload_time_iso_8601": "2024-11-22T05:58:51.977404Z",
"url": "https://files.pythonhosted.org/packages/b5/25/2b5f84149e1ea3c2388b6887b64cb574b9d38074a891d242c7ec49e39337/fetch_cache-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-22 05:58:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xiwen-haochi",
"github_project": "fetch_cache",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "fetch_cache"
}
JSON
