# GStreamer Python ML
This project provides Python base classes and GStreamer elements supporting a broad range
of ML features.
Supported functionality includes:
1. object detection
1. tracking
1. video captioning
1. translation
1. transcription
1. speech to text
1. text to speech
1. text to image
1. LLMs
1. serializing model metadata to Kafka server
ML toolkits are supported via the `MLEngine` abstraction - we have nominal support for
TensorFlow, LiteRT and OpenVINO, but all testing thus far has been done with PyTorch.
These elements will work with your distribution's GStreamer packages. They have been tested on Ubuntu 24 with GStreamer 1.24.
## Python Version
All elements have been tested with Python 3.12, the installed version of Python on Ubuntu 24
## Install
There are two installation options described below: installing on your host machine,
or installing with a Docker container:
### Host Install (Ubuntu 24)
#### Install packages
```
sudo apt update && sudo apt -y upgrade
sudo apt install -y python3-pip python3-venv \
gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps \
gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
gir1.2-gst-plugins-bad-1.0 python3-gst-1.0 gstreamer1.0-python3-plugin-loader \
libcairo2 libcairo2-dev git
```
#### Install venv
`python3 -m venv --system-site-packages ~/venv`
#### Clone repo (host)
`git clone https://github.com/collabora/gst-python-ml.git`
#### Update .bashrc
```
export VIRTUAL_ENV=$HOME/venv
export PATH=$VIRTUAL_ENV/bin:$PATH
export GST_PLUGIN_PATH=$HOME/src/gst-python-ml/plugins
```
and then
`source ~/.bashrc`
#### Activate venv and install basic pip packages
```
source $VIRTUAL_ENV/bin/activate
pip install --upgrade pip
```
#### Install pip requirements
```
cd ~/src/gst-python-ml
pip install -r requirements.txt
```
### Docker Install
#### Build Docker Container
Important Note:
This Dockerfile maps a local `gst-python-ml` repository to the container,
and expects this repository to be located in `~/src` i.e. `~/src/gst-python-ml`.
#### Enable Docker GPU Support on Host
To use the host GPU in a docker container, you will need to install the nvidia container toolkit. If running on CPU, these steps can be skipped.
Add nvidia repository (Ubuntu)
```
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```
Then
```
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo systemctl restart docker
```
#### Build Ubuntu 24.04 Container
`docker build -f ./Dockerfile -t ubuntu24:latest .`
#### Run Docker Container
a) If running on CPU, just remove `--gpus all` from command below
b) This command assumes you have set up a Kafka network as described below
`docker run -v ~/src/gst-python-ml/:/root/gst-python-ml -it --rm --gpus all --name ubuntu24 ubuntu24:latest /bin/bash`
In the container shell, run
`pip install -r requirements.txt`
to install base requirements, and then
`cd gst-python-ml` to run the pipelines below. After installing requirements,
it is recommended to open another terminal on host and run
`docker ps` to get the container id, and then run
`docker commit $CONTAINER_ID` to commit the changes, where `$CONTAINER_ID`
is the id for your docker instance.
#### Docker Cleanup
If you want to purge existing docker containers and images:
```
docker container prune -f
docker image prune -a -f
```
## IMPORTANT NOTE
To use the language elements included in this project, the `nvidia-cuda-toolkit`
ubuntu package must be installed, and additional pip requirements must be installed from
`requirements/language_requrements.txt`
## Post Install
Run `gst-inspect-1.0 python` to see all of the pyml elements listed.
# Building PyPI Package
1. `pip install setuptools wheel twine`
2. `python setup.py sdist bdist_wheel`
3. ls dist/
## Using GStreamer Python ML Elements
## Pipelines
Below are some sample pipelines for the various elements in this project.
### kafkasink
#### Setting up kafka network
`docker network create kafka-network`
and list networks
`docker network ls`
#### docker launch
To launch a docker instance with the kafka network, add ` --network kafka-network `
to the docker launch command above.
#### Set up kafka and zookeeper
Note: setup below assumes you are running your pipeline in a docker container.
If running pipeline from host, then the port changes from `9092` to `29092`,
and the broker changes from `kafka` to `localhost`.
```
docker stop kafka zookeeper
docker rm kafka zookeeper
docker run -d --name zookeeper --network kafka-network -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest
docker run -d --name kafka --network kafka-network \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=INSIDE://kafka:9092,OUTSIDE://localhost:29092 \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT \
-e KAFKA_LISTENERS=INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092 \
-e KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE \
-e KAFKA_BROKER_ID=1 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-p 9092:9092 \
-p 29092:29092 \
confluentinc/cp-kafka:latest
```
#### Create test topic
```
docker exec kafka kafka-topics --create --topic test-kafkasink-topic --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1
```
#### list topics
`docker exec -it kafka kafka-topics --list --bootstrap-server kafka:9092`
#### delete topic
`docker exec -it kafka kafka-topics --delete --topic test-topic --bootstrap-server kafka:9092`
#### consume topic
`docker exec -it kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test-kafkasink-topic --from-beginning`
### non ML
`GST_DEBUG=4 gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720 ! pyml_overlay meta-path=data/sample_metadata.json tracking=true ! videoconvert ! autovideosink`
Note: make sure to set the following in `.bashrc` file :
`export GST_PLUGIN_PATH=/home/$USER/src/gst-python-ml/plugins:$GST_PLUGIN_PATH`
### Bird's Eye View
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videoconvert ! pyml_birdseye ! videoconvert ! autovideosink`
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videorate ! video/x-raw,framerate=30/1 ! videoconvert ! pyml_birdseye ! videoconvert ! openh264enc ! h264parse ! matroskamux ! filesink location=output.mkv`
### Object Detection
Possible model names:
`fasterrcnn_resnet50_fpn`
`retinanet_resnet50_fpn`
#### fasterrcnn/kafka
`GST_DEBUG=4 gst-launch-1.0 multifilesrc location=data/000015.jpg ! jpegdec ! videoconvert ! videoscale ! pyml_objectdetector model-name=fasterrcnn_resnet50_fpn device=cuda batch-size=4 ! pyml_kafkasink schema-file=data/pyml_object_detector.json broker=kafka:9092 topic=test-kafkasink-topic 2>&1 | grep pyml_kafkasink`
#### maskrcnn
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! pyml_maskrcnn device=cuda batch-size=4 model-name=maskrcnn_resnet50_fpn ! videoconvert ! objectdetectionoverlay labels-color=0xFFFF0000 object-detection-outline-color=0xFFFF0000 ! autovideosink`
#### yolo with tracking
`gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! videoconvert ! pyml_overlay labels-color=0xFFFF0000 object-detection-outline-color=0xFFFF0000 ! autovideosink`
#### yolo with overlay
`gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! pyml_overlay ! videoconvert ! autovideosink`
### streammux pipeline
`GST_DEBUG=4 gst-launch-1.0 pyml_streammux name=mux ! videoconvert ! fakesink videotestsrc ! mux. videotestsrc pattern=ball ! mux. videotestsrc pattern=snow ! mux.`
### Transcription
#### transcription with initial prompt set
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko initial_prompt = "Air Traffic Control은, radar systems를, weather conditions에, flight paths를, communication은, unexpected weather conditions가, continuous training을, dedication과, professionalism" ! fakesink`
#### translation to English
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! fakesink`
#### coquitts
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_coquitts device=cuda ! audioconvert ! wavenc ! filesink location=output_audio.wav`
#### whisperspeechtts
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_whisperspeechtts device=cuda ! audioconvert ! wavenc ! filesink location=output_audio.wav`
#### mariantranslate
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_mariantranslate device=cuda src=en target=fr ! fakesink`
Supported src/target languages:
https://huggingface.co/models?sort=trending&search=Helsinki
#### whisperlive
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whisperlive device=cuda language=ko translate=yes llm-model-name="microsoft/phi-2" ! audioconvert ! wavenc ! filesink location=output_audio.wav`
### LLM
1. generate HuggingFace token
2. `huggingface-cli login`
and pass in token
3. LLM pipeline (in this case, we use phi-2)
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/prompt_for_llm.txt ! pyml_llm device=cuda model-name="microsoft/phi-2" ! fakesink`
### stablediffusion
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/prompt_for_stable_diffusion.txt ! pyml_stablediffusion device=cuda ! pngenc ! filesink location=output_image.png`
#### Caption
#### caption + yolo
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! pyml_caption device=cuda:0 ! textoverlay ! pyml_overlay ! videoconvert ! autovideosink`
#### caption
`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! pyml_caption device=cuda:0 downsampled_width=320 downsampled_height=240 prompt="What is the name of the game being played?" ! textoverlay ! autovideosink`
Raw data
{
"_id": null,
"home_page": "https://github.com/collabora/gst-python-ml",
"name": "gst-python-ml",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Aaron Boxer",
"author_email": "aaron.boxer@collabora.com",
"download_url": "https://files.pythonhosted.org/packages/c3/12/118a4974b5c74ce4963ceae993b7266fec53c052005549bda3dd514c9e35/gst_python_ml-0.1.0.tar.gz",
"platform": null,
"description": "# GStreamer Python ML\n\nThis project provides Python base classes and GStreamer elements supporting a broad range\nof ML features. \n\nSupported functionality includes:\n\n1. object detection\n1. tracking\n1. video captioning\n1. translation\n1. transcription\n1. speech to text\n1. text to speech\n1. text to image\n1. LLMs\n1. serializing model metadata to Kafka server\n\nML toolkits are supported via the `MLEngine` abstraction - we have nominal support for\nTensorFlow, LiteRT and OpenVINO, but all testing thus far has been done with PyTorch.\n\nThese elements will work with your distribution's GStreamer packages. They have been tested on Ubuntu 24 with GStreamer 1.24.\n\n## Python Version\n\nAll elements have been tested with Python 3.12, the installed version of Python on Ubuntu 24\n\n## Install\n\nThere are two installation options described below: installing on your host machine,\nor installing with a Docker container:\n\n### Host Install (Ubuntu 24)\n\n#### Install packages\n\n```\nsudo apt update && sudo apt -y upgrade\nsudo apt install -y python3-pip python3-venv \\\n gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps \\\n gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \\\n gir1.2-gst-plugins-bad-1.0 python3-gst-1.0 gstreamer1.0-python3-plugin-loader \\\n libcairo2 libcairo2-dev git\n```\n\n#### Install venv\n\n`python3 -m venv --system-site-packages ~/venv`\n\n\n#### Clone repo (host)\n\n`git clone https://github.com/collabora/gst-python-ml.git`\n\n#### Update .bashrc\n\n```\nexport VIRTUAL_ENV=$HOME/venv\nexport PATH=$VIRTUAL_ENV/bin:$PATH\nexport GST_PLUGIN_PATH=$HOME/src/gst-python-ml/plugins\n```\n\nand then\n\n`source ~/.bashrc`\n\n#### Activate venv and install basic pip packages\n\n```\nsource $VIRTUAL_ENV/bin/activate\npip install --upgrade pip\n```\n\n#### Install pip requirements\n\n```\ncd ~/src/gst-python-ml\npip install -r requirements.txt\n```\n\n### Docker Install\n\n#### Build Docker Container\n\nImportant Note:\n\nThis Dockerfile maps a local `gst-python-ml` repository to the container,\nand expects this repository to be located in `~/src` i.e. `~/src/gst-python-ml`.\n\n\n#### Enable Docker GPU Support on Host\n\nTo use the host GPU in a docker container, you will need to install the nvidia container toolkit. If running on CPU, these steps can be skipped.\n\n\nAdd nvidia repository (Ubuntu)\n\n```\ncurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \\\n && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \\\n sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \\\n sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list\n```\n\nThen\n\n```\nsudo apt update\nsudo apt install -y nvidia-container-toolkit\nsudo systemctl restart docker\n```\n\n#### Build Ubuntu 24.04 Container\n`docker build -f ./Dockerfile -t ubuntu24:latest .`\n\n#### Run Docker Container\n\na) If running on CPU, just remove `--gpus all` from command below\nb) This command assumes you have set up a Kafka network as described below\n\n`docker run -v ~/src/gst-python-ml/:/root/gst-python-ml -it --rm --gpus all --name ubuntu24 ubuntu24:latest /bin/bash`\n\nIn the container shell, run\n\n`pip install -r requirements.txt`\n\nto install base requirements, and then\n\n`cd gst-python-ml` to run the pipelines below. After installing requirements,\nit is recommended to open another terminal on host and run\n\n`docker ps` to get the container id, and then run\n\n`docker commit $CONTAINER_ID` to commit the changes, where `$CONTAINER_ID`\nis the id for your docker instance.\n\n#### Docker Cleanup\n\nIf you want to purge existing docker containers and images:\n\n```\ndocker container prune -f\ndocker image prune -a -f\n```\n\n## IMPORTANT NOTE\n\nTo use the language elements included in this project, the `nvidia-cuda-toolkit`\nubuntu package must be installed, and additional pip requirements must be installed from\n`requirements/language_requrements.txt`\n\n## Post Install\n\nRun `gst-inspect-1.0 python` to see all of the pyml elements listed.\n\n# Building PyPI Package\n\n1. `pip install setuptools wheel twine`\n2. `python setup.py sdist bdist_wheel`\n3. ls dist/\n\n\n## Using GStreamer Python ML Elements\n\n## Pipelines\n\nBelow are some sample pipelines for the various elements in this project.\n\n### kafkasink\n\n#### Setting up kafka network\n\n`docker network create kafka-network`\n\nand list networks\n\n`docker network ls`\n\n#### docker launch\n\nTo launch a docker instance with the kafka network, add ` --network kafka-network `\nto the docker launch command above.\n\n#### Set up kafka and zookeeper\n\nNote: setup below assumes you are running your pipeline in a docker container. \nIf running pipeline from host, then the port changes from `9092` to `29092`,\nand the broker changes from `kafka` to `localhost`.\n\n```\ndocker stop kafka zookeeper\ndocker rm kafka zookeeper\ndocker run -d --name zookeeper --network kafka-network -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest\ndocker run -d --name kafka --network kafka-network \\\n -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \\\n -e KAFKA_ADVERTISED_LISTENERS=INSIDE://kafka:9092,OUTSIDE://localhost:29092 \\\n -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT \\\n -e KAFKA_LISTENERS=INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092 \\\n -e KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE \\\n -e KAFKA_BROKER_ID=1 \\\n -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \\\n -p 9092:9092 \\\n -p 29092:29092 \\\n confluentinc/cp-kafka:latest\n```\n\n#### Create test topic\n```\ndocker exec kafka kafka-topics --create --topic test-kafkasink-topic --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1\n```\n\n#### list topics\n\n`docker exec -it kafka kafka-topics --list --bootstrap-server kafka:9092`\n\n\n#### delete topic\n\n`docker exec -it kafka kafka-topics --delete --topic test-topic --bootstrap-server kafka:9092`\n\n\n#### consume topic\n\n`docker exec -it kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test-kafkasink-topic --from-beginning`\n\n\n### non ML\n\n`GST_DEBUG=4 gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720 ! pyml_overlay meta-path=data/sample_metadata.json tracking=true ! videoconvert ! autovideosink`\n\nNote: make sure to set the following in `.bashrc` file :\n\n`export GST_PLUGIN_PATH=/home/$USER/src/gst-python-ml/plugins:$GST_PLUGIN_PATH`\n\n\n### Bird's Eye View\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videoconvert ! pyml_birdseye ! videoconvert ! autovideosink`\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videorate ! video/x-raw,framerate=30/1 ! videoconvert ! pyml_birdseye ! videoconvert ! openh264enc ! h264parse ! matroskamux ! filesink location=output.mkv`\n\n### Object Detection\n\nPossible model names:\n`fasterrcnn_resnet50_fpn`\n`retinanet_resnet50_fpn`\n\n#### fasterrcnn/kafka\n\n`GST_DEBUG=4 gst-launch-1.0 multifilesrc location=data/000015.jpg ! jpegdec ! videoconvert ! videoscale ! pyml_objectdetector model-name=fasterrcnn_resnet50_fpn device=cuda batch-size=4 ! pyml_kafkasink schema-file=data/pyml_object_detector.json broker=kafka:9092 topic=test-kafkasink-topic 2>&1 | grep pyml_kafkasink`\n\n#### maskrcnn\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! pyml_maskrcnn device=cuda batch-size=4 model-name=maskrcnn_resnet50_fpn ! videoconvert ! objectdetectionoverlay labels-color=0xFFFF0000 object-detection-outline-color=0xFFFF0000 ! autovideosink`\n\n\n#### yolo with tracking\n\n`gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! videoconvert ! pyml_overlay labels-color=0xFFFF0000 object-detection-outline-color=0xFFFF0000 ! autovideosink`\n\n#### yolo with overlay\n\n `gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! pyml_overlay ! videoconvert ! autovideosink`\n\n\n### streammux pipeline\n\n`GST_DEBUG=4 gst-launch-1.0 pyml_streammux name=mux ! videoconvert ! fakesink videotestsrc ! mux. videotestsrc pattern=ball ! mux. videotestsrc pattern=snow ! mux.`\n\n\n### Transcription\n\n#### transcription with initial prompt set\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko initial_prompt = \"Air Traffic Control\uc740, radar systems\ub97c, weather conditions\uc5d0, flight paths\ub97c, communication\uc740, unexpected weather conditions\uac00, continuous training\uc744, dedication\uacfc, professionalism\" ! fakesink`\n\n#### translation to English\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! fakesink`\n\n#### coquitts\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_coquitts device=cuda ! audioconvert ! wavenc ! filesink location=output_audio.wav`\n\n\n#### whisperspeechtts\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_whisperspeechtts device=cuda ! audioconvert ! wavenc ! filesink location=output_audio.wav`\n\n\n#### mariantranslate\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! pyml_mariantranslate device=cuda src=en target=fr ! fakesink`\n\nSupported src/target languages:\n\nhttps://huggingface.co/models?sort=trending&search=Helsinki\n\n\n#### whisperlive\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whisperlive device=cuda language=ko translate=yes llm-model-name=\"microsoft/phi-2\" ! audioconvert ! wavenc ! filesink location=output_audio.wav`\n\n### LLM\n\n1. generate HuggingFace token\n\n2. `huggingface-cli login`\n and pass in token\n\n3. LLM pipeline (in this case, we use phi-2)\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/prompt_for_llm.txt ! pyml_llm device=cuda model-name=\"microsoft/phi-2\" ! fakesink`\n\n### stablediffusion\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/prompt_for_stable_diffusion.txt ! pyml_stablediffusion device=cuda ! pngenc ! filesink location=output_image.png`\n\n#### Caption\n\n#### caption + yolo\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! pyml_caption device=cuda:0 ! textoverlay ! pyml_overlay ! videoconvert ! autovideosink`\n\n\n#### caption\n\n`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! pyml_caption device=cuda:0 downsampled_width=320 downsampled_height=240 prompt=\"What is the name of the game being played?\" ! textoverlay ! autovideosink`\n",
"bugtrack_url": null,
"license": "LGPL-3.0",
"summary": "An ML package for GStreamer",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/collabora/gst-python-ml"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "85f98cad23c998902a63783c6caf95d6cd79c88766aa604b67f06143bcd14b43",
"md5": "2f76d5668fea753faac3f53b204331f5",
"sha256": "ad6763c0ecaaa12e20657a79a5e62f2837ddd8118490de2fee7818956ada3ddd"
},
"downloads": -1,
"filename": "gst_python_ml-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2f76d5668fea753faac3f53b204331f5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 22223,
"upload_time": "2025-01-31T16:35:18",
"upload_time_iso_8601": "2025-01-31T16:35:18.120406Z",
"url": "https://files.pythonhosted.org/packages/85/f9/8cad23c998902a63783c6caf95d6cd79c88766aa604b67f06143bcd14b43/gst_python_ml-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c312118a4974b5c74ce4963ceae993b7266fec53c052005549bda3dd514c9e35",
"md5": "2bfc32277b49fe8a74e956a6f11dc53d",
"sha256": "6a084722b7826459a74bec77de353ad6958a9c8ec689350cb84c40dffc567b0b"
},
"downloads": -1,
"filename": "gst_python_ml-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "2bfc32277b49fe8a74e956a6f11dc53d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 52749,
"upload_time": "2025-01-31T16:35:20",
"upload_time_iso_8601": "2025-01-31T16:35:20.470817Z",
"url": "https://files.pythonhosted.org/packages/c3/12/118a4974b5c74ce4963ceae993b7266fec53c052005549bda3dd514c9e35/gst_python_ml-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-31 16:35:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "collabora",
"github_project": "gst-python-ml",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "pygobject",
"specs": []
},
{
"name": "torch",
"specs": []
},
{
"name": "torchvision",
"specs": []
},
{
"name": "torchaudio",
"specs": []
},
{
"name": "transformers",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "huggingface-hub",
"specs": []
},
{
"name": "lap",
"specs": []
},
{
"name": "ultralytics",
"specs": []
},
{
"name": "pycairo",
"specs": []
}
],
"lcname": "gst-python-ml"
}
JSON
