<h1 style='text-align: center; margin-bottom: 1rem'> Gradio WebRTC ⚡️ </h1>
<div style="display: flex; flex-direction: row; justify-content: center">
<img style="display: block; padding-right: 5px; height: 20px;" alt="Static Badge" src="https://img.shields.io/pypi/v/gradio_webrtc">
<a href="https://github.com/freddyaboulton/gradio-webrtc" target="_blank"><img alt="Static Badge" style="display: block; padding-right: 5px; height: 20px;" src="https://img.shields.io/badge/github-white?logo=github&logoColor=black"></a>
<a href="https://freddyaboulton.github.io/gradio-webrtc/" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/Docs-ffcf40"></a>
</div>
<h3 style='text-align: center'>
Stream video and audio in real time with Gradio using WebRTC.
</h3>
## Installation
```bash
pip install gradio_webrtc
```
to use built-in pause detection (see [ReplyOnPause](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-pause)), install the `vad` extra:
```bash
pip install gradio_webrtc[vad]
```
For stop word detection (see [ReplyOnStopWords](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-stopwords)), install the `stopword` extra:
```bash
pip install gradio_webrtc[stopword]
```
## Docs
https://freddyaboulton.github.io/gradio-webrtc/
## Examples
<table>
<tr>
<td width="50%">
<h3>🗣️ Audio Input/Output with mini-omni2</h3>
<p>Build a GPT-4o like experience with mini-omni2, an audio-native LLM.</p>
<video width="100%" src="https://github.com/user-attachments/assets/58c06523-fc38-4f5f-a4ba-a02a28e7fa9e" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
<h3>🗣️ Talk to Claude</h3>
<p>Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.</p>
<video width="100%" src="https://github.com/user-attachments/assets/650bc492-798e-4995-8cef-159e1cfc2185" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-claude">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-claude/blob/main/app.py">Code</a>
</p>
</td>
</tr>
<tr>
<td width="50%">
<h3>🗣️ Kyutai Moshi</h3>
<p>Kyutai's moshi is a novel speech-to-speech model for modeling human conversations.</p>
<video width="100%" src="https://github.com/user-attachments/assets/becc7a13-9e89-4a19-9df2-5fb1467a0137" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
<h3>🗣️ Hello Llama: Stop Word Detection</h3>
<p>A code editor built with Llama 3.3 70b that is triggered by the phrase "Hello Llama". Build a Siri-like coding assistant in 100 lines of code!</p>
<video width="100%" src="https://github.com/user-attachments/assets/3e10cb15-ff1b-4b17-b141-ff0ad852e613" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py">Code</a>
</p>
</td>
</tr>
<tr>
<td width="50%">
<h3>🤖 Llama Code Editor</h3>
<p>Create and edit HTML pages with just your voice! Powered by SambaNova systems.</p>
<video width="100%" src="https://github.com/user-attachments/assets/a09647f1-33e1-4154-a5a3-ffefda8a736a" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/llama-code-editor">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/llama-code-editor/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
<h3>🗣️ Talk to Ultravox</h3>
<p>Talk to Fixie.AI's audio-native Ultravox LLM with the transformers library.</p>
<video width="100%" src="https://github.com/user-attachments/assets/e6e62482-518c-4021-9047-9da14cd82be1" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-ultravox">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-ultravox/blob/main/app.py">Code</a>
</p>
</td>
</tr>
<tr>
<td width="50%">
<h3>🗣️ Talk to Llama 3.2 3b</h3>
<p>Use the Lepton API to make Llama 3.2 talk back to you!</p>
<video width="100%" src="https://github.com/user-attachments/assets/3ee37a6b-0892-45f5-b801-73188fdfad9a" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/llama-3.2-3b-voice-webrtc">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/llama-3.2-3b-voice-webrtc/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
<h3>🤖 Talk to Qwen2-Audio</h3>
<p>Qwen2-Audio is a SOTA audio-to-text LLM developed by Alibaba.</p>
<video width="100%" src="https://github.com/user-attachments/assets/c821ad86-44cc-4d0c-8dc4-8c02ad1e5dc8" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-qwen-webrtc">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/talk-to-qwen-webrtc/blob/main/app.py">Code</a>
</p>
</td>
</tr>
<tr>
<td width="50%">
<h3>📷 Yolov10 Object Detection</h3>
<p>Run the Yolov10 model on a user webcam stream in real time!</p>
<video width="100%" src="https://github.com/user-attachments/assets/c90d8c9d-d2d5-462e-9e9b-af969f2ea73c" controls></video>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
<h3>📷 Video Object Detection with RT-DETR</h3>
<p>Upload a video and stream out frames with detected objects (powered by RT-DETR) model.</p>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc/blob/main/app.py">Code</a>
</p>
</td>
</tr>
<tr>
<td width="50%">
<h3>🔊 Text-to-Speech with Parler</h3>
<p>Stream out audio generated by Parler TTS!</p>
<p>
<a href="https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc">Demo</a> |
<a href="https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc/blob/main/app.py">Code</a>
</p>
</td>
<td width="50%">
</td>
</tr>
</table>
## Usage
This is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/).
To get started with WebRTC streams, all that's needed is to import the `WebRTC` component from this package and implement its `stream` event.
### Reply on Pause
Typically, you want to run an AI model that generates audio when the user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class
and passing it to the `stream` event of the `WebRTC` component.
```py
import gradio as gr
from gradio_webrtc import WebRTC, ReplyOnPause
def response(audio: tuple[int, np.ndarray]): # (1)
"""This function must yield audio frames"""
...
for numpy_array in generated_audio:
yield (sampling_rate, numpy_array, "mono") # (2)
with gr.Blocks() as demo:
gr.HTML(
"""
<h1 style='text-align: center'>
Chat (Powered by WebRTC ⚡️)
</h1>
"""
)
with gr.Column():
with gr.Group():
audio = WebRTC(
mode="send-receive", # (3)
modality="audio",
)
audio.stream(fn=ReplyOnPause(response),
inputs=[audio], outputs=[audio], # (4)
time_limit=60) # (5)
demo.launch()
```
1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components.
2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples).
3. The `mode` and `modality` arguments must be set to `"send-receive"` and `"audio"`.
4. The `WebRTC` component must be the first input and output component.
5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time.
### Reply On Stopwords
You can configure your AI model to run whenever a set of "stop words" are detected, like "Hey Siri" or "computer", with the `ReplyOnStopWords` class.
The API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter.
```py
import gradio as gr
from gradio_webrtc import WebRTC, ReplyOnPause
def response(audio: tuple[int, np.ndarray]):
"""This function must yield audio frames"""
...
for numpy_array in generated_audio:
yield (sampling_rate, numpy_array, "mono")
with gr.Blocks() as demo:
gr.HTML(
"""
<h1 style='text-align: center'>
Chat (Powered by WebRTC ⚡️)
</h1>
"""
)
with gr.Column():
with gr.Group():
audio = WebRTC(
mode="send",
modality="audio",
)
webrtc.stream(ReplyOnStopWords(generate,
input_sample_rate=16000,
stop_words=["computer"]), # (1)
inputs=[webrtc, history, code],
outputs=[webrtc], time_limit=90,
concurrency_limit=10)
demo.launch()
```
1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. "llama", "lamma". In my experience, it's best to use two very distinct words like "ok computer" or "hello iris".
### Audio Server-To-Clien
To stream only from the server to the client, implement a python generator and pass it to the component's `stream` event. The stream event must also specify a `trigger` corresponding to a UI interaction that starts the stream. In this case, it's a button click.
```py
import gradio as gr
from gradio_webrtc import WebRTC
from pydub import AudioSegment
def generation(num_steps):
for _ in range(num_steps):
segment = AudioSegment.from_file("audio_file.wav")
array = np.array(segment.get_array_of_samples()).reshape(1, -1)
yield (segment.frame_rate, array)
with gr.Blocks() as demo:
audio = WebRTC(label="Stream", mode="receive", # (1)
modality="audio")
num_steps = gr.Slider(label="Number of Steps", minimum=1,
maximum=10, step=1, value=5)
button = gr.Button("Generate")
audio.stream(
fn=generation, inputs=[num_steps], outputs=[audio],
trigger=button.click # (2)
)
```
1. Set `mode="receive"` to only receive audio from the server.
2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click.
### Video Input/Output Streaming
Set up a video Input/Output stream to continuosly receive webcam frames from the user and run an arbitrary python function to return a modified frame.
```py
import gradio as gr
from gradio_webrtc import WebRTC
def detection(image, conf_threshold=0.3): # (1)
... your detection code here ...
return modified_frame # (2)
with gr.Blocks() as demo:
image = WebRTC(label="Stream", mode="send-receive", modality="video") # (3)
conf_threshold = gr.Slider(
label="Confidence Threshold",
minimum=0.0,
maximum=1.0,
step=0.05,
value=0.30,
)
image.stream(
fn=detection,
inputs=[image, conf_threshold], # (4)
outputs=[image], time_limit=10
)
if __name__ == "__main__":
demo.launch()
```
1. The webcam frame will be represented as a numpy array of shape (height, width, RGB).
2. The function must return a numpy array. It can take arbitrary values from other components.
3. Set the `modality="video"` and `mode="send-receive"`
4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component.
### Server-to-Client Only
Set up a server-to-client stream to stream video from an arbitrary user interaction.
```py
import gradio as gr
from gradio_webrtc import WebRTC
import cv2
def generation():
url = "https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4"
cap = cv2.VideoCapture(url)
iterating = True
while iterating:
iterating, frame = cap.read()
yield frame # (1)
with gr.Blocks() as demo:
output_video = WebRTC(label="Video Stream", mode="receive", # (2)
modality="video")
button = gr.Button("Start", variant="primary")
output_video.stream(
fn=generation, inputs=None, outputs=[output_video],
trigger=button.click # (3)
)
demo.launch()
```
1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**.
2. Set `mode="receive"` to only receive audio from the server.
3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event.
### Additional Outputs
In order to modify other components from within the WebRTC stream, you must yield an instance of `AdditionalOutputs` and add an `on_additional_outputs` event to the `WebRTC` component.
This is common for displaying a multimodal text/audio conversation in a Chatbot UI.
``` py title="Additional Outputs"
from gradio_webrtc import AdditionalOutputs, WebRTC
def transcribe(audio: tuple[int, np.ndarray],
transformers_convo: list[dict],
gradio_convo: list[dict]):
response = model.generate(**inputs, max_length=256)
transformers_convo.append({"role": "assistant", "content": response})
gradio_convo.append({"role": "assistant", "content": response})
yield AdditionalOutputs(transformers_convo, gradio_convo) # (1)
with gr.Blocks() as demo:
gr.HTML(
"""
<h1 style='text-align: center'>
Talk to Qwen2Audio (Powered by WebRTC ⚡️)
</h1>
"""
)
transformers_convo = gr.State(value=[])
with gr.Row():
with gr.Column():
audio = WebRTC(
label="Stream",
mode="send", # (2)
modality="audio",
)
with gr.Column():
transcript = gr.Chatbot(label="transcript", type="messages")
audio.stream(ReplyOnPause(transcribe),
inputs=[audio, transformers_convo, transcript],
outputs=[audio], time_limit=90)
audio.on_additional_outputs(lambda s,a: (s,a), # (3)
outputs=[transformers_convo, transcript],
queue=False, show_progress="hidden")
demo.launch()
```
1. Pass your data to `AdditionalOutputs` and yield it.
2. In this case, no audio is being returned, so we set `mode="send"`. However, if we set `mode="send-receive"`, we could also yield generated audio and `AdditionalOutputs`.
3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update.
=== "Notes"
1. Pass your data to `AdditionalOutputs` and yield it.
2. In this case, no audio is being returned, so we set `mode="send"`. However, if we set `mode="send-receive"`, we could also yield generated audio and `AdditionalOutputs`.
3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update.
## Deployment
When deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic.
The easiest way to do this is to use a service like Twilio.
```python
from twilio.rest import Client
import os
account_sid = os.environ.get("TWILIO_ACCOUNT_SID")
auth_token = os.environ.get("TWILIO_AUTH_TOKEN")
client = Client(account_sid, auth_token)
token = client.tokens.create()
rtc_configuration = {
"iceServers": token.ice_servers,
"iceTransportPolicy": "relay",
}
with gr.Blocks() as demo:
...
rtc = WebRTC(rtc_configuration=rtc_configuration, ...)
...
```
Raw data
{
"_id": null,
"home_page": null,
"name": "gradio-webrtc",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "gradio-custom-component, gradio-template-Video, realtime, streaming, webrtc",
"author": null,
"author_email": "Freddy Boulton <YOUREMAIL@domain.com>",
"download_url": null,
"platform": null,
"description": "<h1 style='text-align: center; margin-bottom: 1rem'> Gradio WebRTC \u26a1\ufe0f </h1>\n\n<div style=\"display: flex; flex-direction: row; justify-content: center\">\n<img style=\"display: block; padding-right: 5px; height: 20px;\" alt=\"Static Badge\" src=\"https://img.shields.io/pypi/v/gradio_webrtc\"> \n<a href=\"https://github.com/freddyaboulton/gradio-webrtc\" target=\"_blank\"><img alt=\"Static Badge\" style=\"display: block; padding-right: 5px; height: 20px;\" src=\"https://img.shields.io/badge/github-white?logo=github&logoColor=black\"></a>\n<a href=\"https://freddyaboulton.github.io/gradio-webrtc/\" target=\"_blank\"><img alt=\"Static Badge\" src=\"https://img.shields.io/badge/Docs-ffcf40\"></a>\n</div>\n\n<h3 style='text-align: center'>\nStream video and audio in real time with Gradio using WebRTC. \n</h3>\n\n## Installation\n\n```bash\npip install gradio_webrtc\n```\n\nto use built-in pause detection (see [ReplyOnPause](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-pause)), install the `vad` extra:\n\n```bash\npip install gradio_webrtc[vad]\n```\n\nFor stop word detection (see [ReplyOnStopWords](https://freddyaboulton.github.io/gradio-webrtc//user-guide/#reply-on-stopwords)), install the `stopword` extra:\n\n```bash\npip install gradio_webrtc[stopword]\n```\n\n## Docs\n\nhttps://freddyaboulton.github.io/gradio-webrtc/\n\n## Examples\n\n<table>\n<tr>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Audio Input/Output with mini-omni2</h3>\n<p>Build a GPT-4o like experience with mini-omni2, an audio-native LLM.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/58c06523-fc38-4f5f-a4ba-a02a28e7fa9e\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/mini-omni2-webrtc/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Talk to Claude</h3>\n<p>Use the Anthropic and Play.Ht APIs to have an audio conversation with Claude.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/650bc492-798e-4995-8cef-159e1cfc2185\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-claude\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-claude/blob/main/app.py\">Code</a>\n</p>\n</td>\n</tr>\n\n<tr>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Kyutai Moshi</h3>\n<p>Kyutai's moshi is a novel speech-to-speech model for modeling human conversations.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/becc7a13-9e89-4a19-9df2-5fb1467a0137\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-moshi\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-moshi/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Hello Llama: Stop Word Detection</h3>\n<p>A code editor built with Llama 3.3 70b that is triggered by the phrase \"Hello Llama\". Build a Siri-like coding assistant in 100 lines of code!</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/3e10cb15-ff1b-4b17-b141-ff0ad852e613\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/hey-llama-code-editor/blob/main/app.py\">Code</a>\n</p>\n</td>\n</tr>\n\n<tr>\n<td width=\"50%\">\n<h3>\ud83e\udd16 Llama Code Editor</h3>\n<p>Create and edit HTML pages with just your voice! Powered by SambaNova systems.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/a09647f1-33e1-4154-a5a3-ffefda8a736a\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/llama-code-editor\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/llama-code-editor/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Talk to Ultravox</h3>\n<p>Talk to Fixie.AI's audio-native Ultravox LLM with the transformers library.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/e6e62482-518c-4021-9047-9da14cd82be1\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-ultravox\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-ultravox/blob/main/app.py\">Code</a>\n</p>\n</td>\n</tr>\n\n<tr>\n<td width=\"50%\">\n<h3>\ud83d\udde3\ufe0f Talk to Llama 3.2 3b</h3>\n<p>Use the Lepton API to make Llama 3.2 talk back to you!</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/3ee37a6b-0892-45f5-b801-73188fdfad9a\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/llama-3.2-3b-voice-webrtc\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/llama-3.2-3b-voice-webrtc/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n<h3>\ud83e\udd16 Talk to Qwen2-Audio</h3>\n<p>Qwen2-Audio is a SOTA audio-to-text LLM developed by Alibaba.</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/c821ad86-44cc-4d0c-8dc4-8c02ad1e5dc8\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-qwen-webrtc\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/talk-to-qwen-webrtc/blob/main/app.py\">Code</a>\n</p>\n</td>\n</tr>\n\n<tr>\n<td width=\"50%\">\n<h3>\ud83d\udcf7 Yolov10 Object Detection</h3>\n<p>Run the Yolov10 model on a user webcam stream in real time!</p>\n<video width=\"100%\" src=\"https://github.com/user-attachments/assets/c90d8c9d-d2d5-462e-9e9b-af969f2ea73c\" controls></video>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n<h3>\ud83d\udcf7 Video Object Detection with RT-DETR</h3>\n<p>Upload a video and stream out frames with detected objects (powered by RT-DETR) model.</p>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/rt-detr-object-detection-webrtc/blob/main/app.py\">Code</a>\n</p>\n</td>\n</tr>\n\n<tr>\n<td width=\"50%\">\n<h3>\ud83d\udd0a Text-to-Speech with Parler</h3>\n<p>Stream out audio generated by Parler TTS!</p>\n<p>\n<a href=\"https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc\">Demo</a> |\n<a href=\"https://huggingface.co/spaces/freddyaboulton/parler-tts-streaming-webrtc/blob/main/app.py\">Code</a>\n</p>\n</td>\n<td width=\"50%\">\n</td>\n</tr>\n</table>\n\n## Usage\n\nThis is an shortened version of the official [usage guide](https://freddyaboulton.github.io/gradio-webrtc/user-guide/). \n\nTo get started with WebRTC streams, all that's needed is to import the `WebRTC` component from this package and implement its `stream` event. \n\n### Reply on Pause\n\nTypically, you want to run an AI model that generates audio when the user has stopped speaking. This can be done by wrapping a python generator with the `ReplyOnPause` class\nand passing it to the `stream` event of the `WebRTC` component.\n\n```py \nimport gradio as gr\nfrom gradio_webrtc import WebRTC, ReplyOnPause\n\ndef response(audio: tuple[int, np.ndarray]): # (1)\n \"\"\"This function must yield audio frames\"\"\"\n ...\n for numpy_array in generated_audio:\n yield (sampling_rate, numpy_array, \"mono\") # (2)\n\n\nwith gr.Blocks() as demo:\n gr.HTML(\n \"\"\"\n <h1 style='text-align: center'>\n Chat (Powered by WebRTC \u26a1\ufe0f)\n </h1>\n \"\"\"\n )\n with gr.Column():\n with gr.Group():\n audio = WebRTC(\n mode=\"send-receive\", # (3)\n modality=\"audio\",\n )\n audio.stream(fn=ReplyOnPause(response),\n inputs=[audio], outputs=[audio], # (4)\n time_limit=60) # (5)\n\ndemo.launch()\n```\n\n1. The python generator will receive the **entire** audio up until the user stopped. It will be a tuple of the form (sampling_rate, numpy array of audio). The array will have a shape of (1, num_samples). You can also pass in additional input components.\n\n2. The generator must yield audio chunks as a tuple of (sampling_rate, numpy audio array). Each numpy audio array must have a shape of (1, num_samples).\n\n3. The `mode` and `modality` arguments must be set to `\"send-receive\"` and `\"audio\"`.\n\n4. The `WebRTC` component must be the first input and output component. \n\n5. Set a `time_limit` to control how long a conversation will last. If the `concurrency_count` is 1 (default), only one conversation will be handled at a time.\n\n\n### Reply On Stopwords\n\nYou can configure your AI model to run whenever a set of \"stop words\" are detected, like \"Hey Siri\" or \"computer\", with the `ReplyOnStopWords` class. \n\nThe API is similar to `ReplyOnPause` with the addition of a `stop_words` parameter.\n\n\n```py \nimport gradio as gr\nfrom gradio_webrtc import WebRTC, ReplyOnPause\n\ndef response(audio: tuple[int, np.ndarray]):\n \"\"\"This function must yield audio frames\"\"\"\n ...\n for numpy_array in generated_audio:\n yield (sampling_rate, numpy_array, \"mono\")\n\n\nwith gr.Blocks() as demo:\n gr.HTML(\n \"\"\"\n <h1 style='text-align: center'>\n Chat (Powered by WebRTC \u26a1\ufe0f)\n </h1>\n \"\"\"\n )\n with gr.Column():\n with gr.Group():\n audio = WebRTC(\n mode=\"send\",\n modality=\"audio\",\n )\n webrtc.stream(ReplyOnStopWords(generate,\n input_sample_rate=16000,\n stop_words=[\"computer\"]), # (1)\n inputs=[webrtc, history, code],\n outputs=[webrtc], time_limit=90,\n concurrency_limit=10)\n\ndemo.launch()\n```\n\n1. The `stop_words` can be single words or pairs of words. Be sure to include common misspellings of your word for more robust detection, e.g. \"llama\", \"lamma\". In my experience, it's best to use two very distinct words like \"ok computer\" or \"hello iris\". \n \n\n### Audio Server-To-Clien\n\nTo stream only from the server to the client, implement a python generator and pass it to the component's `stream` event. The stream event must also specify a `trigger` corresponding to a UI interaction that starts the stream. In this case, it's a button click.\n\n\n\n```py\nimport gradio as gr\nfrom gradio_webrtc import WebRTC\nfrom pydub import AudioSegment\n\ndef generation(num_steps):\n for _ in range(num_steps):\n segment = AudioSegment.from_file(\"audio_file.wav\")\n array = np.array(segment.get_array_of_samples()).reshape(1, -1)\n yield (segment.frame_rate, array)\n\nwith gr.Blocks() as demo:\n audio = WebRTC(label=\"Stream\", mode=\"receive\", # (1)\n modality=\"audio\")\n num_steps = gr.Slider(label=\"Number of Steps\", minimum=1,\n maximum=10, step=1, value=5)\n button = gr.Button(\"Generate\")\n\n audio.stream(\n fn=generation, inputs=[num_steps], outputs=[audio],\n trigger=button.click # (2)\n )\n```\n\n1. Set `mode=\"receive\"` to only receive audio from the server.\n2. The `stream` event must take a `trigger` that corresponds to the gradio event that starts the stream. In this case, it's the button click.\n\n\n### Video Input/Output Streaming\nSet up a video Input/Output stream to continuosly receive webcam frames from the user and run an arbitrary python function to return a modified frame.\n \n```py\nimport gradio as gr\nfrom gradio_webrtc import WebRTC\n\n\ndef detection(image, conf_threshold=0.3): # (1)\n ... your detection code here ...\n return modified_frame # (2)\n\n\nwith gr.Blocks() as demo:\n image = WebRTC(label=\"Stream\", mode=\"send-receive\", modality=\"video\") # (3)\n conf_threshold = gr.Slider(\n label=\"Confidence Threshold\",\n minimum=0.0,\n maximum=1.0,\n step=0.05,\n value=0.30,\n )\n image.stream(\n fn=detection,\n inputs=[image, conf_threshold], # (4)\n outputs=[image], time_limit=10\n )\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n\n1. The webcam frame will be represented as a numpy array of shape (height, width, RGB).\n2. The function must return a numpy array. It can take arbitrary values from other components.\n3. Set the `modality=\"video\"` and `mode=\"send-receive\"`\n4. The `inputs` parameter should be a list where the first element is the WebRTC component. The only output allowed is the WebRTC component.\n\n### Server-to-Client Only\n\nSet up a server-to-client stream to stream video from an arbitrary user interaction.\n\n```py \nimport gradio as gr\nfrom gradio_webrtc import WebRTC\nimport cv2\n\ndef generation():\n url = \"https://download.tsi.telecom-paristech.fr/gpac/dataset/dash/uhd/mux_sources/hevcds_720p30_2M.mp4\"\n cap = cv2.VideoCapture(url)\n iterating = True\n while iterating:\n iterating, frame = cap.read()\n yield frame # (1)\n\nwith gr.Blocks() as demo:\n output_video = WebRTC(label=\"Video Stream\", mode=\"receive\", # (2)\n modality=\"video\")\n button = gr.Button(\"Start\", variant=\"primary\")\n output_video.stream(\n fn=generation, inputs=None, outputs=[output_video],\n trigger=button.click # (3)\n )\n demo.launch()\n```\n\n1. The `stream` event's `fn` parameter is a generator function that yields the next frame from the video as a **numpy array**.\n2. Set `mode=\"receive\"` to only receive audio from the server.\n3. The `trigger` parameter the gradio event that will trigger the stream. In this case, the button click event.\n\n\n### Additional Outputs\n\nIn order to modify other components from within the WebRTC stream, you must yield an instance of `AdditionalOutputs` and add an `on_additional_outputs` event to the `WebRTC` component.\n\nThis is common for displaying a multimodal text/audio conversation in a Chatbot UI.\n\n\n\n``` py title=\"Additional Outputs\"\nfrom gradio_webrtc import AdditionalOutputs, WebRTC\n\ndef transcribe(audio: tuple[int, np.ndarray],\n transformers_convo: list[dict],\n gradio_convo: list[dict]):\n response = model.generate(**inputs, max_length=256)\n transformers_convo.append({\"role\": \"assistant\", \"content\": response})\n gradio_convo.append({\"role\": \"assistant\", \"content\": response})\n yield AdditionalOutputs(transformers_convo, gradio_convo) # (1)\n\n\nwith gr.Blocks() as demo:\n gr.HTML(\n \"\"\"\n <h1 style='text-align: center'>\n Talk to Qwen2Audio (Powered by WebRTC \u26a1\ufe0f)\n </h1>\n \"\"\"\n )\n transformers_convo = gr.State(value=[])\n with gr.Row():\n with gr.Column():\n audio = WebRTC(\n label=\"Stream\",\n mode=\"send\", # (2)\n modality=\"audio\",\n )\n with gr.Column():\n transcript = gr.Chatbot(label=\"transcript\", type=\"messages\")\n\n audio.stream(ReplyOnPause(transcribe),\n inputs=[audio, transformers_convo, transcript],\n outputs=[audio], time_limit=90)\n audio.on_additional_outputs(lambda s,a: (s,a), # (3)\n outputs=[transformers_convo, transcript],\n queue=False, show_progress=\"hidden\")\n demo.launch()\n```\n \n 1. Pass your data to `AdditionalOutputs` and yield it.\n 2. In this case, no audio is being returned, so we set `mode=\"send\"`. However, if we set `mode=\"send-receive\"`, we could also yield generated audio and `AdditionalOutputs`.\n 3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update.\n=== \"Notes\"\n 1. Pass your data to `AdditionalOutputs` and yield it.\n 2. In this case, no audio is being returned, so we set `mode=\"send\"`. However, if we set `mode=\"send-receive\"`, we could also yield generated audio and `AdditionalOutputs`.\n 3. The `on_additional_outputs` event does not take `inputs`. It's common practice to not run this event on the queue since it is just a quick UI update.\n\n\n## Deployment\n\nWhen deploying in a cloud environment (like Hugging Face Spaces, EC2, etc), you need to set up a TURN server to relay the WebRTC traffic.\nThe easiest way to do this is to use a service like Twilio.\n\n```python\nfrom twilio.rest import Client\nimport os\n\naccount_sid = os.environ.get(\"TWILIO_ACCOUNT_SID\")\nauth_token = os.environ.get(\"TWILIO_AUTH_TOKEN\")\n\nclient = Client(account_sid, auth_token)\n\ntoken = client.tokens.create()\n\nrtc_configuration = {\n \"iceServers\": token.ice_servers,\n \"iceTransportPolicy\": \"relay\",\n}\n\nwith gr.Blocks() as demo:\n ...\n rtc = WebRTC(rtc_configuration=rtc_configuration, ...)\n ...\n```",
"bugtrack_url": null,
"license": null,
"summary": "Stream images in realtime with webrtc",
"version": "0.0.25",
"project_urls": null,
"split_keywords": [
"gradio-custom-component",
" gradio-template-video",
" realtime",
" streaming",
" webrtc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "119f2bd3a43f75da0e1c3802c7f56a69dc3248739325e5287cde57d31fca24fb",
"md5": "0072239abcf24d34092bfdbd4d8ee90e",
"sha256": "3ac2daada09d8e996fbe072339e81393c766a8d8fdd6715f2e9cda1ea84181b5"
},
"downloads": -1,
"filename": "gradio_webrtc-0.0.25-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0072239abcf24d34092bfdbd4d8ee90e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 1169355,
"upload_time": "2024-12-20T20:11:51",
"upload_time_iso_8601": "2024-12-20T20:11:51.393797Z",
"url": "https://files.pythonhosted.org/packages/11/9f/2bd3a43f75da0e1c3802c7f56a69dc3248739325e5287cde57d31fca24fb/gradio_webrtc-0.0.25-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 20:11:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "gradio-webrtc"
}
JSON