# OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
<div align="center">
[[Website]](https://os-copilot.github.io/)
[[Arxiv]](https://arxiv.org/abs/2402.07456)
[[PDF]](https://arxiv.org/pdf/2402.07456.pdf)
<!-- [[Tweet]](https://twitter.com/DrJimFan/status/1662115266933972993?s=20) -->
[](https://github.com/OS-Copilot/FRIDAY/blob/main/LICENSE)
[](https://github.com/OS-Copilot/FRIDAY-front)
<p align="center">
<img src='pic/demo.png' width="100%">
</p>
</div>
## π Overview
- **OS-Copilot** is a pioneering conceptual framework for building generalist computer agents on Linux and MacOS, which provides a unified interface for app interactions in the heterogeneous OS ecosystem.
<p align="center">
<img src='pic/framework.png' width="75%">
</p>
- Leveraging OS-Copilot, we built **FRIDAY**, a self-improving AI assistant capable of solving general computer tasks.
<p align="center">
<img src='pic/FRIDAY.png' width="75%">
</p>
## β‘οΈ Quickstart
1. **Clone the GitHub Repository:**
```
git clone https://github.com/OS-Copilot/FRIDAY.git
```
2. **Set Up Python Environment:** Ensure you have a version 3.10 or higher Python environment. You can create and
activate this environment using the following commands, replacing `FRIDAY_env` with your preferred environment
name:
```
conda create -n FRIDAY_env python=3.10 -y
conda activate FRIDAY_env
```
3. **Install Dependencies:** Move into the `FRIDAY` directory and install the necessary dependencies by running:
```
cd FRIDAY
pip install -r requirements.txt
```
4. **Set OpenAI API Key:** Configure your OpenAI API key in [.env](.env) and select the model you wish to use.
5. **Execute Your Task:** Run the following command to start FRIDAY. Replace `[query]` with your task as needed. By default, the task is *"Move the text files containing the word 'agent' from the folder named 'document' to the path 'working_dir/agent'"*. If the task requires using related files, you can use `--query_file_path [file_path]`.
```
python run.py --query [query]
```
\* FRIDAY currently only supports single-round conversation.
## π οΈ FRIDAY-Gizmos
We maintain an open-source library of toolkits for FRIDAY, which includes tools that can be directly utilized within FRIDAY.
For a detailed list of tools, please see [FRIDAY-Gizmos](https://github.com/OS-Copilot/FRIDAY-Gizmos). The usage methods are as follows:
1. Find the tool you want to use in [FRIDAY-Gizmos](https://github.com/OS-Copilot/FRIDAY-Gizmos) and download its tool code.
2. Add the tool to FRIDAY's toolkit:
```shell
python friday/core/action_manager.py --add --tool_name [tool_name] --tool_path [tool_path]
```
3. If you wish to remove a tool, you can run:
```shell
python friday/core/action_manager.py --delete --tool_name [tool_name]
```
## π» User Interface (UI)
**Enhance Your Experience with Our Intuitive Frontend!** This interface is crafted for effortless control of your agents. For more details, visit [FRIDAY Frontend](https://github.com/OS-Copilot/FRIDAY-front).
## β¨ Deploy your own API tools with FastAPI
All FastAPIs are underοΌ [friday/api](friday/api)
1. **Prepare your FastAPI file:** Create a new api folder under [friday/api](friday/api) and put your FastAPi python files under that folder.
2. **Import your FastAPI in API server:** Import your apis in [friday/core/api_server.py](friday/core/api_server.py)οΌ
```python
import os
from fastapi import FastAPI
from friday.core.server_config import ConfigManager
app = FastAPI()
from friday.api.bing.bing_service import router as bing_router
#[TODO] Import your own api here
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
class LoggingMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
print(f"Incoming request: {request.method} {request.url}")
try:
response = await call_next(request)
except Exception as e:
print(f"Request error: {str(e)}")
raise e from None
else:
print(f"Outgoing response: {response.status_code}")
return response
app.add_middleware(LoggingMiddleware)
# Create a dictionary that maps service names to their routers
services = {
"bing": bing_router,
# [TODO] Add your api router here
}
server_list = [
"bing",
# [TODO] Add your api's service name here.
]
# Include only the routers for the services listed in server_list
for service in server_list:
if service in services:
app.include_router(services[service])
# proxy_manager = ConfigManager()
# proxy_manager.apply_proxies()
if __name__ == "__main__":
import uvicorn
# you can change your port anyway
uvicorn.run(app, host="0.0.0.0", port=8079)
```
3. **Run API server:**
Run the server in localhost,or deploy it on your web server:
```
python api_server.py
```
4. **Update API documentation:**
Update the API documentation located in [friday/core/openapi.json](friday/core/openapi.json). After launching the API server, you can access the current OpenAPI documentation at `http://localhost:8079/openapi.json`.
Ensure to thoroughly update each API's summary in the documentation to clearly explain its functionality and usage. This is crucial as FRIDAY relies on these descriptions to understand the purpose of each API.
For example:
```json
{
"openapi": "3.1.0",
"info": {
"title": "FastAPI",
"version": "0.1.0"
},
"paths": {
"/tools/audio2text": {
"post": {
// [TODO] change the summary to describe the usage of your api.
"summary": "A tool that converts audio to natural language text",
"operationId": "audio2text_tools_audio2text_post",
"requestBody": {
"content": {
"multipart/form-data": {
"schema": {
"$ref": "#/components/schemas/Body_audio2text_tools_audio2text_post"
}
}
},
"required": true
},
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
},
},
"components": {
"schemas": {
"Body_audio2text_tools_audio2text_post": {
"properties": {
"file": {
"type": "string",
"format": "binary",
"title": "File"
}
},
"type": "object",
"required": [
"file"
],
"title": "Body_audio2text_tools_audio2text_post"
},
}
}
}
```
5. **Change the base url of tool_request_util.py:** FRIDAY utilizes the script located at [friday/core/tool_request_util.py](friday/core/tool_request_util.py) to interface with your API tools. After deploying your APIs, make sure to update the base URL in this file to match your API server's URL.
```python
import requests
class ToolRequestUtil:
def __init__(self):
self.session = requests.session()
self.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
# [TODO] Change the base url
self.base_url = "http://localhost:8079"
def request(self, api_path, method, params=None, files=None, content_type="application/json"):
"""
:param api_path: the path of the api
:param method: get/post
:param params: the params of the api, can be None
:param files: files to be uploaded, can be None
:param content_type: the content_type of api, e.g., application/json, multipart/form-data, can be None
:return: the return of the api
"""
url = self.base_url + api_path
try:
if method.lower() == "get":
if content_type == "application/json":
result = self.session.get(url=url, json=params, headers=self.headers, timeout=60).json()
else:
result = self.session.get(url=url, params=params, headers=self.headers, timeout=60).json()
elif method.lower() == "post":
if content_type == "multipart/form-data":
result = self.session.post(url=url, files=files, data=params, headers=self.headers).json()
elif content_type == "application/json":
result = self.session.post(url=url, json=params, headers=self.headers).json()
else:
result = self.session.post(url=url, data=params, headers=self.headers).json()
else:
print("request method error!")
return None
return result
except Exception as e:
print("http request error: %s" % e)
return None
```
<!-- ## π¨βπ»β Contributors
<a href="">
<img src="" />
</a>
Made with [contrib.rocks](https://contrib.rocks). -->
## π‘ Disclaimer
OS-Copilot is provided "as is" without warranty of any kind. Users assume full responsibility for any risks associated with its use, including **potential data loss** or **changes to system settings**. The developers of OS-Copilot are not liable for any damages or losses resulting from its use. Users must ensure their actions comply with applicable laws and regulations.
## π« Community
Join our community to connect with other agent enthusiasts, share your tools and demos, and collaborate on exciting initiatives. You can find us on [Slack](https://join.slack.com/t/slack-ped8294/shared_invite/zt-2cqebow90-soac9UFKGZ2RcUy8PqjZrA).
## π Citation
```
@misc{wu2024oscopilot,
title={OS-Copilot: Towards Generalist Computer Agents with Self-Improvement},
author={Zhiyong Wu and Chengcheng Han and Zichen Ding and Zhenmin Weng and Zhoumianze Liu and Shunyu Yao and Tao Yu and Lingpeng Kong},
year={2024},
eprint={2402.07456},
archivePrefix={arXiv},
primaryClass={cs.AI}
}
```
## π¬ Contact
If you have any inquiries, suggestions, or wish to contact us for any reason, we warmly invite you to email us at wuzhiyong@pjlab.org.cn.
Raw data
{
"_id": null,
"home_page": "https://github.com/OS-Copilot/FRIDAY",
"name": "friday-agent",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "AI,LLMs,Large Language Models,Agent,OS,Operating System",
"author": "Zhiyong Wu and Chengcheng Han and Zichen Ding and Zhenmin Weng and Zhoumianze Liu and Shunyu Yao and Tao Yu and Lingpeng Kong",
"author_email": "wuzhiyong@pjlab.org.cn, hccngu@163.com",
"download_url": "https://files.pythonhosted.org/packages/1c/be/1101b3373d8b89dcf87a8c646c3728843ea42f3e9aba32e5d338f8e2755c/friday-agent-0.1.0.tar.gz",
"platform": null,
"description": "# OS-Copilot: Towards Generalist Computer Agents with Self-Improvement\n\n<div align=\"center\">\n\n[[Website]](https://os-copilot.github.io/)\n[[Arxiv]](https://arxiv.org/abs/2402.07456)\n[[PDF]](https://arxiv.org/pdf/2402.07456.pdf)\n<!-- [[Tweet]](https://twitter.com/DrJimFan/status/1662115266933972993?s=20) -->\n\n[](https://github.com/OS-Copilot/FRIDAY/blob/main/LICENSE)\n\n[](https://github.com/OS-Copilot/FRIDAY-front)\n\n\n\n<p align=\"center\">\n <img src='pic/demo.png' width=\"100%\">\n</p>\n\n</div>\n\n## \ud83d\udcd6 Overview\n\n- **OS-Copilot** is a pioneering conceptual framework for building generalist computer agents on Linux and MacOS, which provides a unified interface for app interactions in the heterogeneous OS ecosystem.\n \n<p align=\"center\">\n <img src='pic/framework.png' width=\"75%\">\n</p>\n\n- Leveraging OS-Copilot, we built **FRIDAY**, a self-improving AI assistant capable of solving general computer tasks.\n\n<p align=\"center\">\n <img src='pic/FRIDAY.png' width=\"75%\">\n</p>\n\n## \u26a1\ufe0f Quickstart\n\n1. **Clone the GitHub Repository:** \n\n ```\n git clone https://github.com/OS-Copilot/FRIDAY.git\n ```\n\n2. **Set Up Python Environment:** Ensure you have a version 3.10 or higher Python environment. You can create and\n activate this environment using the following commands, replacing `FRIDAY_env` with your preferred environment\n name:\n\n ```\n conda create -n FRIDAY_env python=3.10 -y\n conda activate FRIDAY_env\n ```\n\n3. **Install Dependencies:** Move into the `FRIDAY` directory and install the necessary dependencies by running:\n\n ```\n cd FRIDAY\n pip install -r requirements.txt\n ```\n\n4. **Set OpenAI API Key:** Configure your OpenAI API key in [.env](.env) and select the model you wish to use.\n\n5. **Execute Your Task:** Run the following command to start FRIDAY. Replace `[query]` with your task as needed. By default, the task is *\"Move the text files containing the word 'agent' from the folder named 'document' to the path 'working_dir/agent'\"*. If the task requires using related files, you can use `--query_file_path [file_path]`.\n ```\n python run.py --query [query]\n ```\n\n\\* FRIDAY currently only supports single-round conversation.\n\n## \ud83d\udee0\ufe0f FRIDAY-Gizmos\nWe maintain an open-source library of toolkits for FRIDAY, which includes tools that can be directly utilized within FRIDAY.\nFor a detailed list of tools, please see [FRIDAY-Gizmos](https://github.com/OS-Copilot/FRIDAY-Gizmos). The usage methods are as follows:\n\n1. Find the tool you want to use in [FRIDAY-Gizmos](https://github.com/OS-Copilot/FRIDAY-Gizmos) and download its tool code.\n2. Add the tool to FRIDAY's toolkit:\n```shell\npython friday/core/action_manager.py --add --tool_name [tool_name] --tool_path [tool_path]\n```\n3. If you wish to remove a tool, you can run:\n```shell\npython friday/core/action_manager.py --delete --tool_name [tool_name]\n```\n\n## \ud83d\udcbb User Interface (UI)\n\n**Enhance Your Experience with Our Intuitive Frontend!** This interface is crafted for effortless control of your agents. For more details, visit [FRIDAY Frontend](https://github.com/OS-Copilot/FRIDAY-front).\n\n## \u2728 Deploy your own API tools with FastAPI\nAll FastAPIs are under\uff1a [friday/api](friday/api)\n1. **Prepare your FastAPI file:** Create a new api folder under [friday/api](friday/api) and put your FastAPi python files under that folder.\n2. **Import your FastAPI in API server:** Import your apis in [friday/core/api_server.py](friday/core/api_server.py)\uff1a\n```python\nimport os\n\nfrom fastapi import FastAPI\nfrom friday.core.server_config import ConfigManager\n\napp = FastAPI()\n\n\nfrom friday.api.bing.bing_service import router as bing_router\n#[TODO] Import your own api here\n\n\nfrom starlette.middleware.base import BaseHTTPMiddleware\nfrom starlette.requests import Request\n\n\nclass LoggingMiddleware(BaseHTTPMiddleware):\n async def dispatch(self, request: Request, call_next):\n print(f\"Incoming request: {request.method} {request.url}\")\n try:\n response = await call_next(request)\n except Exception as e:\n print(f\"Request error: {str(e)}\")\n raise e from None\n else:\n print(f\"Outgoing response: {response.status_code}\")\n return response\n\n\napp.add_middleware(LoggingMiddleware)\n\n# Create a dictionary that maps service names to their routers\nservices = {\n \"bing\": bing_router,\n # [TODO] Add your api router here\n\n}\n\nserver_list = [\n \"bing\",\n # [TODO] Add your api's service name here.\n]\n\n# Include only the routers for the services listed in server_list\nfor service in server_list:\n if service in services:\n app.include_router(services[service])\n\n# proxy_manager = ConfigManager()\n# proxy_manager.apply_proxies()\n\nif __name__ == \"__main__\":\n import uvicorn\n # you can change your port anyway\n uvicorn.run(app, host=\"0.0.0.0\", port=8079)\n```\n3. **Run API server:**\nRun the server in localhost,or deploy it on your web server:\n```\npython api_server.py\n```\n4. **Update API documentation:** \n\nUpdate the API documentation located in [friday/core/openapi.json](friday/core/openapi.json). After launching the API server, you can access the current OpenAPI documentation at `http://localhost:8079/openapi.json`.\n\nEnsure to thoroughly update each API's summary in the documentation to clearly explain its functionality and usage. This is crucial as FRIDAY relies on these descriptions to understand the purpose of each API.\n\nFor example:\n```json\n{\n \"openapi\": \"3.1.0\",\n \"info\": {\n \"title\": \"FastAPI\",\n \"version\": \"0.1.0\"\n },\n \"paths\": { \n \"/tools/audio2text\": {\n \"post\": {\n // [TODO] change the summary to describe the usage of your api.\n \"summary\": \"A tool that converts audio to natural language text\",\n \"operationId\": \"audio2text_tools_audio2text_post\",\n \"requestBody\": {\n \"content\": {\n \"multipart/form-data\": {\n \"schema\": {\n \"$ref\": \"#/components/schemas/Body_audio2text_tools_audio2text_post\"\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"description\": \"Successful Response\",\n \"content\": {\n \"application/json\": {\n \"schema\": {}\n }\n }\n },\n \"422\": {\n \"description\": \"Validation Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"$ref\": \"#/components/schemas/HTTPValidationError\"\n }\n }\n }\n }\n }\n }\n },\n \n },\n \"components\": {\n \"schemas\": {\n \"Body_audio2text_tools_audio2text_post\": {\n \"properties\": {\n \"file\": {\n \"type\": \"string\",\n \"format\": \"binary\",\n \"title\": \"File\"\n }\n },\n \"type\": \"object\",\n \"required\": [\n \"file\"\n ],\n \"title\": \"Body_audio2text_tools_audio2text_post\"\n },\n \n \n }\n }\n}\n```\n\n5. **Change the base url of tool_request_util.py:** FRIDAY utilizes the script located at [friday/core/tool_request_util.py](friday/core/tool_request_util.py) to interface with your API tools. After deploying your APIs, make sure to update the base URL in this file to match your API server's URL.\n```python\nimport requests\nclass ToolRequestUtil:\n def __init__(self):\n self.session = requests.session()\n self.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML like Gecko) Chrome/52.0.2743.116 Safari/537.36'}\n # [TODO] Change the base url\n self.base_url = \"http://localhost:8079\"\n\n def request(self, api_path, method, params=None, files=None, content_type=\"application/json\"):\n \"\"\"\n :param api_path: the path of the api\n :param method: get/post\n :param params: the params of the api, can be None\n :param files: files to be uploaded, can be None\n :param content_type: the content_type of api, e.g., application/json, multipart/form-data, can be None\n :return: the return of the api\n \"\"\"\n url = self.base_url + api_path\n try:\n if method.lower() == \"get\":\n if content_type == \"application/json\":\n result = self.session.get(url=url, json=params, headers=self.headers, timeout=60).json()\n else: \n result = self.session.get(url=url, params=params, headers=self.headers, timeout=60).json()\n elif method.lower() == \"post\":\n if content_type == \"multipart/form-data\":\n result = self.session.post(url=url, files=files, data=params, headers=self.headers).json()\n elif content_type == \"application/json\":\n result = self.session.post(url=url, json=params, headers=self.headers).json()\n else:\n result = self.session.post(url=url, data=params, headers=self.headers).json()\n else:\n print(\"request method error!\")\n return None\n return result\n except Exception as e:\n print(\"http request error: %s\" % e)\n return None\n```\n<!-- ## \ud83d\udc68\u200d\ud83d\udcbb\u200d Contributors\n\n<a href=\"\">\n <img src=\"\" />\n</a>\n\nMade with [contrib.rocks](https://contrib.rocks). -->\n\n## \ud83d\udee1 Disclaimer\n\nOS-Copilot is provided \"as is\" without warranty of any kind. Users assume full responsibility for any risks associated with its use, including **potential data loss** or **changes to system settings**. The developers of OS-Copilot are not liable for any damages or losses resulting from its use. Users must ensure their actions comply with applicable laws and regulations.\n\n\n## \ud83c\udfeb Community\n\nJoin our community to connect with other agent enthusiasts, share your tools and demos, and collaborate on exciting initiatives. You can find us on [Slack](https://join.slack.com/t/slack-ped8294/shared_invite/zt-2cqebow90-soac9UFKGZ2RcUy8PqjZrA).\n\n\n## \ud83d\udd0e Citation\n\n```\n@misc{wu2024oscopilot,\n title={OS-Copilot: Towards Generalist Computer Agents with Self-Improvement}, \n author={Zhiyong Wu and Chengcheng Han and Zichen Ding and Zhenmin Weng and Zhoumianze Liu and Shunyu Yao and Tao Yu and Lingpeng Kong},\n year={2024},\n eprint={2402.07456},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}\n```\n\n\n## \ud83d\udcec Contact\n\nIf you have any inquiries, suggestions, or wish to contact us for any reason, we warmly invite you to email us at wuzhiyong@pjlab.org.cn.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An self-improving embodied conversational agent seamlessly integrated into the operating system to automate our daily tasks.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/OS-Copilot/FRIDAY"
},
"split_keywords": [
"ai",
"llms",
"large language models",
"agent",
"os",
"operating system"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "50b88e4146588690991e7e5363281b107dc0b580c55c66e64ad9e90b287eebee",
"md5": "f49c6cc9d8e064498acc40f2565c8d78",
"sha256": "b578c3e642d2b8bff512d717d8b671226c2607abfbb9b8760088e4f9afb0fd82"
},
"downloads": -1,
"filename": "friday_agent-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f49c6cc9d8e064498acc40f2565c8d78",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 83997,
"upload_time": "2024-03-07T08:29:27",
"upload_time_iso_8601": "2024-03-07T08:29:27.319605Z",
"url": "https://files.pythonhosted.org/packages/50/b8/8e4146588690991e7e5363281b107dc0b580c55c66e64ad9e90b287eebee/friday_agent-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1cbe1101b3373d8b89dcf87a8c646c3728843ea42f3e9aba32e5d338f8e2755c",
"md5": "b4cf61b915a22862c1a4427b3a55c50b",
"sha256": "b810f3eb0b5ede9b9e3baf809d8ae078394d48ab570a78ff60a7a280e4fcffe6"
},
"downloads": -1,
"filename": "friday-agent-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "b4cf61b915a22862c1a4427b3a55c50b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 72742,
"upload_time": "2024-03-07T08:29:29",
"upload_time_iso_8601": "2024-03-07T08:29:29.859459Z",
"url": "https://files.pythonhosted.org/packages/1c/be/1101b3373d8b89dcf87a8c646c3728843ea42f3e9aba32e5d338f8e2755c/friday-agent-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-07 08:29:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "OS-Copilot",
"github_project": "FRIDAY",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "friday-agent"
}
JSON
