# sparrow-python
[](https://pypi.org/project/sparrow-python)
[](https://www.python.org/)
[](LICENSE)
---
## TODO
- [ ] prompt调试页面
- [ ] 相关配置指定支持：prompt后端地址；模型参数配置；
- [ ]
- [ ] 添加测试按钮，模型选项，模型配置
- [ ] 原生git下载支持
- [ ]
- [X] streamlit 多模态chat input: https://github.com/streamlit/streamlit/issues/7409
- [ ] from .cv.image.image_processor import messages_preprocess 添加是否对网络url替换为base64的控制；添加对video切帧的支持
- [ ] https://github.com/hiyouga/LLaMA-Factory/blob/main/src/llamafactory/chat/vllm_engine.py#L99
识别下面链接的滚动截图：
https://sjh.baidu.com/site/dzfmws.cn/da721a31-476d-42ed-aad1-81c2dc3a66a3
vllm 异步推理示例：
```python
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List
import uvicorn
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
import torch
# Define request data model
class RequestData(BaseModel):
prompts: List[str]
max_tokens: int = 2048
temperature: float = 0.7
# Initialize FastAPI app
app = FastAPI()
# Determine device
device = "cuda" if torch.cuda.is_available() else "cpu"
# Initialize AsyncLLMEngine
engine_args = AsyncEngineArgs(
model="your-model-name", # Replace with your model name
dtype="bfloat16",
gpu_memory_utilization=0.8,
max_model_len=4096,
trust_remote_code=True
)
llm_engine = AsyncLLMEngine.from_engine_args(engine_args)
# Define the inference endpoint
@app.post("/predict")
async def generate_text(data: RequestData):
sampling_params = SamplingParams(
max_tokens=data.max_tokens,
temperature=data.temperature
)
request_id = "unique_request_id" # Generate a unique request ID
results_generator = llm_engine.generate(data.prompts, sampling_params, request_id)
final_output = None
async for request_output in results_generator:
final_output = request_output
assert final_output is not None
text_outputs = [output.text for output in final_output.outputs]
return {"responses": text_outputs}
# Run the server
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
```
## 待添加脚本
## Install
```bash
pip install sparrow-python
# Or dev version
pip install sparrow-python[dev]
# Or
pip install -e .
# Or
pip install -e .[dev]
```
## Usage
### Multiprocessing SyncManager
Open server first:
```bash
$ spr start-server
```
The defualt port `50001`.
(Process1) productor:
```python
from sparrow.multiprocess.client import Client
client = Client(port=50001)
client.update_dict({'a': 1, 'b': 2})
```
(Process2) consumer:
```python
from sparrow.multiprocess.client import Client
client = Client(port=50001)
print(client.get_dict_data())
>> > {'a': 1, 'b': 2}
```
### Common tools
- **Kill process by port**
```bash
$ spr kill {port}
```
- **pack & unpack**
support archive format: "zip", "tar", "gztar", "bztar", or "xztar".
```bash
$ spr pack pack_dir
```
```bash
$ spr unpack filename extract_dir
```
- **Scaffold**
```bash
$ spr create awosome-project
```
### Some useful functions
> `sparrow.relp`
> Relative path, which is used to read or save files more easily.
> `sparrow.performance.MeasureTime`
> For measuring time (including gpu time)
> `sparrow.performance.get_process_memory`
> Get the memory size occupied by the process
> `sparrow.performance.get_virtual_memory`
> Get virtual machine memory information
> `sparrow.add_env_path`
> Add python environment variable (use relative file path)
Raw data
{
"_id": null,
"home_page": null,
"name": "sparrow-python",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "Machine Learning, cli, cv, nlp",
"author": null,
"author_email": "kunyuan <beidongjiedeguang@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/60/2b/97a5557fe034f163c47157bd1b0c5e91a759792a7522a3c1a26247bbb928/sparrow_python-0.4.2.tar.gz",
"platform": null,
"description": "# sparrow-python\n\n[](https://pypi.org/project/sparrow-python)\n[](https://www.python.org/)\n[](LICENSE)\n\n---\n\n## TODO\n\n- [ ] prompt\u8c03\u8bd5\u9875\u9762\n- [ ] \u76f8\u5173\u914d\u7f6e\u6307\u5b9a\u652f\u6301\uff1aprompt\u540e\u7aef\u5730\u5740\uff1b\u6a21\u578b\u53c2\u6570\u914d\u7f6e\uff1b\n- [ ] \n- [ ] \u6dfb\u52a0\u6d4b\u8bd5\u6309\u94ae\uff0c\u6a21\u578b\u9009\u9879\uff0c\u6a21\u578b\u914d\u7f6e\n- [ ] \u539f\u751fgit\u4e0b\u8f7d\u652f\u6301\n- [ ]\n- [X] streamlit \u591a\u6a21\u6001chat input: https://github.com/streamlit/streamlit/issues/7409\n- [ ] from .cv.image.image_processor import messages_preprocess \u6dfb\u52a0\u662f\u5426\u5bf9\u7f51\u7edcurl\u66ff\u6362\u4e3abase64\u7684\u63a7\u5236\uff1b\u6dfb\u52a0\u5bf9video\u5207\u5e27\u7684\u652f\u6301\n- [ ] https://github.com/hiyouga/LLaMA-Factory/blob/main/src/llamafactory/chat/vllm_engine.py#L99\n\n\u8bc6\u522b\u4e0b\u9762\u94fe\u63a5\u7684\u6eda\u52a8\u622a\u56fe\uff1a\nhttps://sjh.baidu.com/site/dzfmws.cn/da721a31-476d-42ed-aad1-81c2dc3a66a3\n\nvllm \u5f02\u6b65\u63a8\u7406\u793a\u4f8b\uff1a\n\n```python\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel\nfrom typing import List\nimport uvicorn\nfrom vllm.engine.arg_utils import AsyncEngineArgs\nfrom vllm.engine.async_llm_engine import AsyncLLMEngine\nfrom vllm.sampling_params import SamplingParams\nimport torch\n\n# Define request data model\nclass RequestData(BaseModel):\n prompts: List[str]\n max_tokens: int = 2048\n temperature: float = 0.7\n\n# Initialize FastAPI app\napp = FastAPI()\n\n# Determine device\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n# Initialize AsyncLLMEngine\nengine_args = AsyncEngineArgs(\n model=\"your-model-name\", # Replace with your model name\n dtype=\"bfloat16\",\n gpu_memory_utilization=0.8,\n max_model_len=4096,\n trust_remote_code=True\n)\nllm_engine = AsyncLLMEngine.from_engine_args(engine_args)\n\n# Define the inference endpoint\n@app.post(\"/predict\")\nasync def generate_text(data: RequestData):\n sampling_params = SamplingParams(\n max_tokens=data.max_tokens,\n temperature=data.temperature\n )\n request_id = \"unique_request_id\" # Generate a unique request ID\n results_generator = llm_engine.generate(data.prompts, sampling_params, request_id)\n \n final_output = None\n async for request_output in results_generator:\n final_output = request_output\n \n assert final_output is not None\n text_outputs = [output.text for output in final_output.outputs]\n return {\"responses\": text_outputs}\n\n# Run the server\nif __name__ == \"__main__\":\n uvicorn.run(app, host=\"0.0.0.0\", port=8000)\n\n```\n\n## \u5f85\u6dfb\u52a0\u811a\u672c\n\n## Install\n\n```bash\npip install sparrow-python\n# Or dev version\npip install sparrow-python[dev]\n# Or\npip install -e .\n# Or\npip install -e .[dev]\n```\n\n## Usage\n\n### Multiprocessing SyncManager\n\nOpen server first:\n\n```bash\n$ spr start-server\n```\n\nThe defualt port `50001`.\n\n(Process1) productor:\n\n```python\nfrom sparrow.multiprocess.client import Client\n\nclient = Client(port=50001)\nclient.update_dict({'a': 1, 'b': 2})\n```\n\n(Process2) consumer:\n\n```python\nfrom sparrow.multiprocess.client import Client\n\nclient = Client(port=50001)\nprint(client.get_dict_data())\n\n>> > {'a': 1, 'b': 2}\n```\n\n### Common tools\n\n- **Kill process by port**\n\n```bash\n$ spr kill {port}\n```\n\n- **pack & unpack**\n support archive format: \"zip\", \"tar\", \"gztar\", \"bztar\", or \"xztar\".\n\n```bash\n$ spr pack pack_dir\n```\n\n```bash\n$ spr unpack filename extract_dir\n```\n\n- **Scaffold**\n\n```bash\n$ spr create awosome-project\n```\n\n### Some useful functions\n\n> `sparrow.relp`\n> Relative path, which is used to read or save files more easily.\n\n> `sparrow.performance.MeasureTime`\n> For measuring time (including gpu time)\n\n> `sparrow.performance.get_process_memory`\n> Get the memory size occupied by the process\n\n> `sparrow.performance.get_virtual_memory`\n> Get virtual machine memory information\n\n> `sparrow.add_env_path`\n> Add python environment variable (use relative file path)\n",
"bugtrack_url": null,
"license": null,
"summary": null,
"version": "0.4.2",
"project_urls": {
"Issues": "https://github.com/beidongjiedeguang/sparrow/issues",
"Source": "https://github.com/beidongjiedeguang/sparrow",
"documentation": "https://github.com/beidongjiedeguang/sparrow#sparrow_python",
"homepage": "https://github.com/beidongjiedeguang/sparrow",
"repository": "https://github.com/beidongjiedeguang/sparrow"
},
"split_keywords": [
"machine learning",
" cli",
" cv",
" nlp"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "79ee21718854138087e524a19f074f6bd6c6feb26795dcae3d881b21e6b32e80",
"md5": "6064ec6d728b44df0ed2dbde83986273",
"sha256": "523b242b15e794bf5aa56932e071bf77fd822c4a1263ba37962d529c1458f7ac"
},
"downloads": -1,
"filename": "sparrow_python-0.4.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6064ec6d728b44df0ed2dbde83986273",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 931723,
"upload_time": "2025-03-03T05:40:47",
"upload_time_iso_8601": "2025-03-03T05:40:47.966219Z",
"url": "https://files.pythonhosted.org/packages/79/ee/21718854138087e524a19f074f6bd6c6feb26795dcae3d881b21e6b32e80/sparrow_python-0.4.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "602b97a5557fe034f163c47157bd1b0c5e91a759792a7522a3c1a26247bbb928",
"md5": "9fc172289e86c11ea3a42d8015ad4aca",
"sha256": "8ad14e12ab3646da20d01b96dba99bbaf221a1bd80991df263bf1558d8615c25"
},
"downloads": -1,
"filename": "sparrow_python-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "9fc172289e86c11ea3a42d8015ad4aca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 885488,
"upload_time": "2025-03-03T05:40:50",
"upload_time_iso_8601": "2025-03-03T05:40:50.519972Z",
"url": "https://files.pythonhosted.org/packages/60/2b/97a5557fe034f163c47157bd1b0c5e91a759792a7522a3c1a26247bbb928/sparrow_python-0.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-03-03 05:40:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "beidongjiedeguang",
"github_project": "sparrow",
"github_not_found": true,
"lcname": "sparrow-python"
}
JSON
