| Name | ai01 JSON |
| Version |
0.2.16
JSON |
| download |
| home_page | None |
| Summary | AI Agent SDK on Huddle01 dRTC Network |
| upload_time | 2025-01-21 09:31:13 |
| maintainer | None |
| docs_url | None |
| author | Om Gupta |
| requires_python | <4.0,>=3.12 |
| license | None |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<!-- Begin Banner -->
<a href="https://huddle01.com"><p align="center">
<img height=100 src="https://user-images.githubusercontent.com/34184939/167926050-f1e2db1d-49ae-4b51-bd98-68e09b45c5b1.svg"/>
</p></a>
<p align="center">
<strong>The dRTC Infra for AI.</strong>
</p>
<!-- End Banner -->
# Overview 🚀
This repository aims to bridge the gap between Artificial Intelligence (AI) and Real-Time Communication (RTC) technologies. It provides a set of examples and tutorials to help developers integrate AI into their WebRTC applications.
# Features 🎯
- **Agents**: AI-powered agents that can be integrated into WebRTC applications, such as chatbots, voicebots, and video bots.
- **LLM Models**: API for integrating Large Language Models (LLM) into WebRTC applications, such as Realtime API, Text-to-Speech, and Speech-to-Text.
# Quick Start 🚀
To install the core Agents library:
```bash
pip install ai01
```
Internally the Library uses `huddle01-ai` package to interact with the dRTC Network and build WebRTC Connections.
# Basic Usage 📝
```python
import asyncio
from ai01.agent import Agent, AgentOptions
from ai01.providers.openai import AudioTrack
from ai01.rtc import RTCOptions, Role, HuddleClientOptions
# Configure RTC options
rtc_options = RTCOptions(
api_key="your_huddle_api_key",
project_id="your_project_id",
room_id="your_room_id",
role=Role.HOST,
metadata={"displayName": "AI Agent"}
)
# Initialize Agent
agent = Agent(
options=AgentOptions(
rtc_options=rtc_options,
audio_track=AudioTrack()
)
)
```
# Module Documentation 📖
The core module can be broken down into the following submodules:
- **`ai01.agent`**: Core module for creating AI agents that can be integrated into WebRTC applications, each agent can be considered as a separate entity that can be connected to a dRTC room, and can interact with other agents and users.
- **`ai01.providers`**: Module for integrating AI providers into the core module, such as OpenAI, Google Cloud, and Microsoft Azure, and also exposes API to integrate custom AI providers.
- **`ai01.rtc`**: Module for integrating Real-Time Communication (RTC) technologies into the core module, such as Huddle, Twilio, and Agora, and also exposes API to integrate custom RTC providers, Each Agent in itself is connected to an RTC room.
Anything can be built with mixing and matching different agents with different models in any pattern which is suitable for the application.
# Agent Module
The Agent module provides the core functionality for creating AI agents that can participate in real-time communication rooms.
An Agent is a high-level entity that can:
- Connect to dRTC network
- Interact with AI models
- Process media streams
- Handle room events
## Agent Methods
- **`join()`**
Join Method is used to join the dRTC Network and establish a websocket connection, upon successful connection the agent is assigned a `room`
using which agent can setup necessary event listeners before calling `connect` method.
```python
from ai01.agent import Agent, AgentOptions
from ai01.rtc import RoomEvents
room = await agent.join()
@room.on(RoomEvent.RoomJoined)
def on_room_joined():
print("Room Joined")
```
> Note: The `join` method is an async method and should be awaited before calling any other method.
- **`connect(`)**
Connect Method is used to establish a WebRTC Connection with the room, upon successful connection the agent can start sending and receiving media streams.
```python
await agent.connect()
```
### Events
The Agent module exposes a set of events that can be used to handle room events
```python
from ai01.rtc import RoomEvents, RoomEventsData
@room.on(RoomEvents.NewPeerJoined)
def on_room_joined(data: RoomEventsData.NewPeerJoined):
print("New Peer Joined the Room",data['remote_peer_id'])
```
The following events are supported:
- `RoomJoined`: Triggered when the agent successfully joins the room, which means the agent is successfully connected to the dRTC network.
- `RoomJoinFailed`: Triggered when the agent fails to join the room, which means the agent is not connected to the dRTC network.
- `RoomClosed`: Triggered when the room is closed, which means the room is no longer available.
- `RoomConnecting`: Triggered when the agent successfully connects to the room, which means the agent is connected to the room.
- `NewPeerJoined`: Triggered when a new peer joins the room, which means a new peer is connected to the room.
- `PeerLeft`: Triggered when a peer leaves the room, which means a peer is disconnected from the room.
- `RemoteProducerAdded`: Triggered when a remote producer is added to the room, which means a remote producer is added to the room.
- `RemoteProducerRemoved`: Triggered when a remote producer is removed from the room, which means a remote producer is removed from the room.
- `NewConsumerAdded`: Triggered when a new consumer is added to the room, which means a new consumer is added to the room.
- `ConsumerClosed`: Triggered when a consumer is closed, which means a consumer is closed.
- `ConsumerPaused`: Triggered when a consumer is paused, which means a consumer is paused.
- `ConsumerResumed`: Triggered when a consumer is resumed, which means a consumer is resumed.
# Providers Module
The Providers module provides a set of APIs for integrating AI providers into the core module.
Currently, the following providers are supported:
- `OpenAI`:
1. Realtime API -> ALPHA
2. Speech-to-Text -> ALPHA
3. Text-to-Speech -> TODO
- `Gemini`: TODO
- `Anthropic`: TODO
- `grok`: TODO
## OpenAI Provider
The OpenAI provider provides an API for integrating OpenAI models into the core module.
> Note: Currently, the OpenAI Realtime API is only available as provider.
> Issues are open for adding more providers and models, feel free to open a PR or Issue for adding more providers and models.
### Realtime API
Realtime API is a most advanced model by Open_AI which provides direct voice-to-voice communication with the model, which makes it very powerful
and the response time is the fastest among all the models.
```python
from ai01.providers.openai.realtime import RealTimeModel, RealTimeModelOptions
openai_api_key = os.getenv("OPENAI_API_KEY")
llm = RealTimeModel(
agent=agent,
options=RealTimeModelOptions(
oai_api_key=openai_api_key,
instructions=bot_prompt,
),
)
await llm.connect()
```
This model is still in Beta, but we actively working on it to make it more stable and reliable, and also adding more features to it.
## Methods
**`connect()`**: Connect method is used to establish a connection with the OpenAI Realtime API, upon successful connection the agent can start sending and receiving media streams.
```python
await llm.connect()
```
Upon successful connection, the agent can start sending and receiving media streams, to and from the OpenAI Realtime API.
## Events
Right now the Model pushes all the streams to the `audio_track` of the agent, and the agent can process the audio stream as per the requirement.
# RTC Module
The Core RTC Module is built on-top of the `huddle01` python package, for detailed documentation refer to the documentaion of the `huddle01` package
on [pypi](https://pypi.org/project/huddle01/)
# Contributing 🤝
> This Repository is under active development, and we are actively looking for contributors to help us build this project.
If you are interested in contributing to this project, please refer to the [Contributing Guidelines](https://github.com/Huddle01/huddle01-ai/blob/main/CONTRIBUTING.md).
Keep checking for Latest Issues and PRs, and feel free to open an Issue or PR for any feature or bug.
You can also join the Huddle01 Discord Community for any queries or discussions.
Discord: [Huddle01 Discord](https://discord.gg/huddle01)
# Setting Up the Project Locally 🛠️
To set up the project locally, follow the steps mentioned in the [Setup Guide](https://github.com/Huddle01/huddle01-ai/blob/main/setup.md)
#
Raw data
{
"_id": null,
"home_page": null,
"name": "ai01",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": null,
"author": "Om Gupta",
"author_email": "omgupta0720@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c9/0e/31b8387a0be264c8d6564a708dd487908ce6c09719763068eae062566c7e/ai01-0.2.16.tar.gz",
"platform": null,
"description": "<!-- Begin Banner -->\n<a href=\"https://huddle01.com\"><p align=\"center\">\n<img height=100 src=\"https://user-images.githubusercontent.com/34184939/167926050-f1e2db1d-49ae-4b51-bd98-68e09b45c5b1.svg\"/>\n\n</p></a>\n<p align=\"center\">\n <strong>The dRTC Infra for AI.</strong>\n</p>\n\n<!-- End Banner -->\n\n# Overview \ud83d\ude80\nThis repository aims to bridge the gap between Artificial Intelligence (AI) and Real-Time Communication (RTC) technologies. It provides a set of examples and tutorials to help developers integrate AI into their WebRTC applications. \n\n# Features \ud83c\udfaf\n- **Agents**: AI-powered agents that can be integrated into WebRTC applications, such as chatbots, voicebots, and video bots.\n\n- **LLM Models**: API for integrating Large Language Models (LLM) into WebRTC applications, such as Realtime API, Text-to-Speech, and Speech-to-Text.\n\n# Quick Start \ud83d\ude80\n\nTo install the core Agents library:\n```bash\npip install ai01\n```\n\nInternally the Library uses `huddle01-ai` package to interact with the dRTC Network and build WebRTC Connections.\n\n# Basic Usage \ud83d\udcdd\n\n```python\nimport asyncio\nfrom ai01.agent import Agent, AgentOptions\nfrom ai01.providers.openai import AudioTrack\nfrom ai01.rtc import RTCOptions, Role, HuddleClientOptions\n\n# Configure RTC options\nrtc_options = RTCOptions(\n api_key=\"your_huddle_api_key\",\n project_id=\"your_project_id\",\n room_id=\"your_room_id\",\n role=Role.HOST,\n metadata={\"displayName\": \"AI Agent\"}\n)\n\n# Initialize Agent\nagent = Agent(\n options=AgentOptions(\n rtc_options=rtc_options,\n audio_track=AudioTrack()\n )\n)\n```\n\n# Module Documentation \ud83d\udcd6\n\nThe core module can be broken down into the following submodules:\n\n- **`ai01.agent`**: Core module for creating AI agents that can be integrated into WebRTC applications, each agent can be considered as a separate entity that can be connected to a dRTC room, and can interact with other agents and users.\n\n- **`ai01.providers`**: Module for integrating AI providers into the core module, such as OpenAI, Google Cloud, and Microsoft Azure, and also exposes API to integrate custom AI providers.\n\n- **`ai01.rtc`**: Module for integrating Real-Time Communication (RTC) technologies into the core module, such as Huddle, Twilio, and Agora, and also exposes API to integrate custom RTC providers, Each Agent in itself is connected to an RTC room.\n\nAnything can be built with mixing and matching different agents with different models in any pattern which is suitable for the application.\n\n# Agent Module\nThe Agent module provides the core functionality for creating AI agents that can participate in real-time communication rooms.\n\nAn Agent is a high-level entity that can:\n\n- Connect to dRTC network\n- Interact with AI models\n- Process media streams\n- Handle room events\n\n## Agent Methods\n- **`join()`**\n\nJoin Method is used to join the dRTC Network and establish a websocket connection, upon successful connection the agent is assigned a `room` \nusing which agent can setup necessary event listeners before calling `connect` method.\n\n```python\nfrom ai01.agent import Agent, AgentOptions\nfrom ai01.rtc import RoomEvents\n\nroom = await agent.join()\n\n@room.on(RoomEvent.RoomJoined)\ndef on_room_joined():\n print(\"Room Joined\")\n\n```\n> Note: The `join` method is an async method and should be awaited before calling any other method.\n\n\n- **`connect(`)**\n\nConnect Method is used to establish a WebRTC Connection with the room, upon successful connection the agent can start sending and receiving media streams.\n\n```python\nawait agent.connect()\n```\n\n### Events\nThe Agent module exposes a set of events that can be used to handle room events\n\n```python\nfrom ai01.rtc import RoomEvents, RoomEventsData\n\n@room.on(RoomEvents.NewPeerJoined)\ndef on_room_joined(data: RoomEventsData.NewPeerJoined):\n print(\"New Peer Joined the Room\",data['remote_peer_id'])\n```\n\nThe following events are supported:\n\n- `RoomJoined`: Triggered when the agent successfully joins the room, which means the agent is successfully connected to the dRTC network.\n\n- `RoomJoinFailed`: Triggered when the agent fails to join the room, which means the agent is not connected to the dRTC network.\n\n- `RoomClosed`: Triggered when the room is closed, which means the room is no longer available.\n\n- `RoomConnecting`: Triggered when the agent successfully connects to the room, which means the agent is connected to the room.\n\n- `NewPeerJoined`: Triggered when a new peer joins the room, which means a new peer is connected to the room.\n\n- `PeerLeft`: Triggered when a peer leaves the room, which means a peer is disconnected from the room.\n\n- `RemoteProducerAdded`: Triggered when a remote producer is added to the room, which means a remote producer is added to the room.\n- `RemoteProducerRemoved`: Triggered when a remote producer is removed from the room, which means a remote producer is removed from the room.\n\n- `NewConsumerAdded`: Triggered when a new consumer is added to the room, which means a new consumer is added to the room.\n\n- `ConsumerClosed`: Triggered when a consumer is closed, which means a consumer is closed.\n\n- `ConsumerPaused`: Triggered when a consumer is paused, which means a consumer is paused.\n- `ConsumerResumed`: Triggered when a consumer is resumed, which means a consumer is resumed.\n\n\n# Providers Module\nThe Providers module provides a set of APIs for integrating AI providers into the core module.\n\nCurrently, the following providers are supported:\n- `OpenAI`: \n 1. Realtime API -> ALPHA\n 2. Speech-to-Text -> ALPHA\n 3. Text-to-Speech -> TODO\n- `Gemini`: TODO\n- `Anthropic`: TODO\n- `grok`: TODO\n\n## OpenAI Provider\nThe OpenAI provider provides an API for integrating OpenAI models into the core module.\n\n> Note: Currently, the OpenAI Realtime API is only available as provider.\n> Issues are open for adding more providers and models, feel free to open a PR or Issue for adding more providers and models.\n\n### Realtime API\nRealtime API is a most advanced model by Open_AI which provides direct voice-to-voice communication with the model, which makes it very powerful \nand the response time is the fastest among all the models.\n\n```python\nfrom ai01.providers.openai.realtime import RealTimeModel, RealTimeModelOptions\n\nopenai_api_key = os.getenv(\"OPENAI_API_KEY\")\n\n\nllm = RealTimeModel(\n agent=agent,\n options=RealTimeModelOptions(\n oai_api_key=openai_api_key,\n instructions=bot_prompt,\n ),\n)\n\nawait llm.connect()\n```\n\nThis model is still in Beta, but we actively working on it to make it more stable and reliable, and also adding more features to it.\n\n## Methods\n**`connect()`**: Connect method is used to establish a connection with the OpenAI Realtime API, upon successful connection the agent can start sending and receiving media streams.\n\n```python\nawait llm.connect()\n```\nUpon successful connection, the agent can start sending and receiving media streams, to and from the OpenAI Realtime API.\n\n## Events\n\nRight now the Model pushes all the streams to the `audio_track` of the agent, and the agent can process the audio stream as per the requirement.\n\n# RTC Module\n\nThe Core RTC Module is built on-top of the `huddle01` python package, for detailed documentation refer to the documentaion of the `huddle01` package\non [pypi](https://pypi.org/project/huddle01/)\n\n\n# Contributing \ud83e\udd1d\n> This Repository is under active development, and we are actively looking for contributors to help us build this project.\n\nIf you are interested in contributing to this project, please refer to the [Contributing Guidelines](https://github.com/Huddle01/huddle01-ai/blob/main/CONTRIBUTING.md).\n\nKeep checking for Latest Issues and PRs, and feel free to open an Issue or PR for any feature or bug.\n\nYou can also join the Huddle01 Discord Community for any queries or discussions.\n\nDiscord: [Huddle01 Discord](https://discord.gg/huddle01)\n\n# Setting Up the Project Locally \ud83d\udee0\ufe0f\n\nTo set up the project locally, follow the steps mentioned in the [Setup Guide](https://github.com/Huddle01/huddle01-ai/blob/main/setup.md)\n\n#\n\n\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "AI Agent SDK on Huddle01 dRTC Network",
"version": "0.2.16",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "98915aa88b43ed9d50c91ddc8207d1345ba6bc66e56ffb55498a489f46ebab3e",
"md5": "05f5785b4ac6bcaed0316fa15a8c5780",
"sha256": "af62ede6d1f760295125507cf88e060c693ab38b2a2aad1536294a8fc684f8db"
},
"downloads": -1,
"filename": "ai01-0.2.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "05f5785b4ac6bcaed0316fa15a8c5780",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 25976,
"upload_time": "2025-01-21T09:31:12",
"upload_time_iso_8601": "2025-01-21T09:31:12.563803Z",
"url": "https://files.pythonhosted.org/packages/98/91/5aa88b43ed9d50c91ddc8207d1345ba6bc66e56ffb55498a489f46ebab3e/ai01-0.2.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c90e31b8387a0be264c8d6564a708dd487908ce6c09719763068eae062566c7e",
"md5": "0bc103a9f83bffe7e64535f1380241f0",
"sha256": "e6dafdefc6380552a0023f9bf9f2eee9e553b4988603d5a55714fd952085056b"
},
"downloads": -1,
"filename": "ai01-0.2.16.tar.gz",
"has_sig": false,
"md5_digest": "0bc103a9f83bffe7e64535f1380241f0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 22412,
"upload_time": "2025-01-21T09:31:13",
"upload_time_iso_8601": "2025-01-21T09:31:13.880807Z",
"url": "https://files.pythonhosted.org/packages/c9/0e/31b8387a0be264c8d6564a708dd487908ce6c09719763068eae062566c7e/ai01-0.2.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-21 09:31:13",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ai01"
}
