# gTech Ads Ariel for AI Video Ad Dubbing
### Ariel is an open-source Python library that facilitates efficient and cost-effective dubbing of video ads into multiple languages.
[](https://www.python.org)
[](https://pypi.org/project/gtech-ariel/)
[](https://github.com/google-marketing-solutions/ariel/commits)
[](https://google.github.io/styleguide/pyguide.html)
[](https://colab.research.google.com/github/google-marketing-solutions/ariel/blob/main/examples/dubbing_workflow.ipynb)
<div align="center">
<img src="ariel_logo.png" alt="Ariel Logo" width="300">
</div>
##### _This is not an official Google product._
[Overview](#overview) •
[Features](#features) •
[Benefits](#benefits) •
[Language Compatibility](#language-compatibility) •
[Before You Begin](#before-you-begin) •
[Getting Started](#getting-started) •
[Building Blocks](#building-blocks) •
[References](#references)
## Overview
Ariel is a cutting-edge solution designed to enhance the global reach of digital advertising. It enables advertisers to automate the translation and dubbing of their video ads into a wide range of languages.
## Features
* **Automated Dubbing:** Streamline the generation of high-quality dubbed versions of video ads in various target languages.
* **Scalability:** Handle large volumes of videos and diverse languages efficiently.
* **User-Friendly:** Offers a straightforward API and/or user interface for simplified operation.
* **Cost-Effective:** Significantly reduce dubbing costs compared to traditional methods. The primary expenses are limited to Gemini API and Text-To-Speech API calls.
## Benefits
* **Enhanced Ad Performance:** Improve viewer engagement and potentially increase conversion rates with localized ads.
* **Streamlined Production:** Minimize the time and cost associated with manual translation and voiceover work.
* **Rapid Turnaround:** Quickly generate dubbed versions of ads to accelerate multilingual campaign deployment.
* **Expanded Global Reach:** Reach broader audiences worldwide with localized advertising content.
## Language Compatibility
You can dub video ads from and to the following languages:
* Arabic (ar-SA), (ar-EG)
* Bengali (bn-BD), (bn-IN)
* Bulgarian (bg-BG)
* Chinese (Simplified) (zh-CN)
* Chinese (Traditional) (zh-TW)
* Croatian (hr-HR)
* Czech (cs-CZ)
* Danish (da-DK)
* Dutch (nl-NL)
* English (en-US), (en-GB), (en-CA), (en-AU)
* Estonian (et-EE)
* Finnish (fi-FI)
* French (fr-FR), (fr-CA)
* German (de-DE)
* Greek (el-GR)
* Gujarati (gu-IN)
* Hebrew (he-IL) (Note: Not supported with ElevenLabs API)
* Hindi (hi-IN)
* Hungarian (hu-HU)
* Indonesian (id-ID)
* Italian (it-IT)
* Japanese (ja-JP)
* Kannada (kn-IN)
* Korean (ko-KR)
* Latvian (lv-LV)
* Lithuanian (lt-LT)
* Malayalam (ml-IN)
* Marathi (mr-IN)
* Norwegian (nb-NO), (nn-NO)
* Polish (pl-PL)
* Portuguese (pt-PT), (pt-BR)
* Romanian (ro-RO)
* Russian (ru-RU)
* Serbian (sr-RS)
* Slovak (sk-SK)
* Slovenian (sl-SI)
* Spanish (es-ES), (es-MX)
* Swahili (sw-KE)
* Swedish (sv-SE)
* Tamil (ta-IN), (ta-LK)
* Telugu (te-IN)
* Thai (th-TH)
* Turkish (tr-TR)
* Ukrainian (uk-UA)
* Vietnamese (vi-VN)
## Before You Begin
* **System Requirements:**
* **Installed FFmpeg:** For video and audio processing. You don't need to install this if you're running from Google Colab.
* **GPU (Recommended):** For optimal performance, especially with larger videos. It's available for free in Google Colab.
* **Accounts and Tokens:**
* **Google Cloud Platform (GCP) Project:** Set up a GCP project. See [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects) for instructions.
* **Enabled Vertex AI API:** Enable the Vertex API in your GCP project. See [here](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal) for instructions.
* **Enabled Cloud Storage API:** Enable the Cloud Storage API in your GCP project. See [here](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal) for instructions.
* You need access to create and delete Google Cloud Storage (GCS) buckets.
* **Enabled Text-To-Speech API:** Enable the Text-To-Speech API in your GCP project if you choose it for the Text-To-Speech part of the process. See [here](https://cloud.google.com/text-to-speech/docs/before-you-begin) for instructions.
* **Google Drive API:** Enable the Google Drive API if you use the demo notebook called 'dubbing_workflow.ipynb in Google Colab.
* **Google Sheets API:** Enable the Google Sheets API if you use the demo notebook called 'dubbing_workflow.ipynb in Google Colab and want to pass script voice metadata from Google Sheets.
* **Hugging Face Token:** To access the PyAnnote speaker diarization model. See [here](https://huggingface.co/docs/hub/en/security-tokens) on how to get the token.
* **Hugging Face Model License:** You must accept the user conditions for the PyAnnote speaker diarization [here](https://huggingface.co/pyannote/speaker-diarization-3.1) and segmentation models [here](https://huggingface.co/pyannote/segmentation-3.0).
* **[OPTIONAL] ElevenLabs API:** To access the ElevenLabs API. See [here](https://help.elevenlabs.io/hc/en-us/articles/14599447207697-How-to-authorize-yourself-using-your-xi-api-key).
* **Commercial Use:** You are responsible for selecting the right ElevenLabs license if you decide to use the outputs from Ariel in a commercial setting. See the pricing [here](https://elevenlabs.io/pricing). Else, ElevenLabs is free to use.
* **Data handling:**
* **Input files:** Ariel can work with both video and audio ads and they need to be in the MP4 or MP3 file formats respectively. See [here](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash) for the limitations on the duration of the ads enforeced by Gemini models.
* **Storage:** Ariel reads, processes and saves all the files in your environment, e.g. Colab. Only on one instance it creates a temporary Google Cloud Storage (GCS) bucket to upload the input file there for the Gemini model to indentify unique speakers. The bucket with all its contents is removed immediately afterwards. You can modify the `gcp_region` argument to choose the best location to perform this operation.
* **Voice cloning:** You can clone voices from the input file to make the dubbing sound close to the original if you decide to use ElevenLabs. It's is your responsibility to ensure that you are legally allowed to do so.
## Getting started
To start using Ariel, just click on the this button: [](https://colab.research.google.com/github/google-marketing-solutions/ariel/blob/main/examples/dubbing_workflow.ipynb)
## Building Blocks
Ariel leverages a powerful combination of state-of-the-art AI and audio processing techniques to deliver accurate and efficient dubbing results:
1. **Video Processing:** Extracts the audio track from the input video file.
2. **Audio Processing:**
* **DEMUCS:** Employed for advanced audio source separation.
* **pyannote:** Performs speaker diarization to identify and separate individual speakers.
3. **Speech-To-Text (STT):**
* **faster-whisper:** A high-performance speech-to-text model.
* **Gemini 1.5 Flash:** A powerful multimodal language model that contributes to enhanced transcription.
4. **Translation:**
* **Gemini 1.5 Flash:** Leverages its language understanding for accurate and contextually relevant translation.
5. **Text-to-Speech (TTS):**
* **GCP's Text-To-Speech:** Generates natural-sounding speech in the target language.
* **[OPTIONAL] ElevenLabs:** An alternative API to generate speech. It's recommened for the best results. **WARNING:** ElevenLabs is a paid solution and will generate extra costs. See the pricing [here](https://elevenlabs.io/pricing).
## Cloud-based Ariel
For a more user-friendly experience, we're also providing an Ariel version that
can be deployed onto Google Cloud Platform with a GUI.
### Requirements
In order to use Ariel UI, you need the following:
1. **Google Cloud Platform Project** to host the backend. The following components
are created during deployment:
* **Cloud Storage Bucket** to store input & output videos, video dubbing artifacts and
all other metadata files. This bucket is also used as an interaction point with GUI
(files created/removed there are triggering the backend processing).
* **Cloud Run** instance that processes all the steps. This part is implemented as a
Python Docker container (which is also built during installation).
* **Pub/Sub Infrastructure** based on EventArc, it notifies the container of new files.
2. **AppsScript Project** to host the frontend, an Angular web-app. For this,
you need *Google Workspace* access.
### Deployment
Please make sure you have fulfilled all prerequisites mentioned under [Requirements](#Requirements) first.
1. Make sure your system has an up-to-date installation of [Node.js and npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
1. Install [clasp](https://github.com/google/clasp) by running `npm install @google/clasp -g`, then login via `clasp login`.
1. Navigate to the [Apps Script Settings page](https://script.google.com/home/usersettings) and `enable` the Apps Script API.
1. Make sure your system has an up-to-date installation of the [gcloud CLI](https://cloud.google.com/sdk/docs/install), then login via `gcloud auth login`.
1. Make sure your system has an up-to-date installation of `git` and use it to clone this repository:
`git clone https://github.com/google-marketing-solutions/ariel`.
1. Navigate to the directory where the source code lives: `cd ariel`.
1. Run `npm start`. This will prompt you for configuration values and suggest reasonable defaults.
## References
* **DEMUCS:** [https://github.com/facebookresearch/demucs](https://github.com/facebookresearch/demucs)
* **pyannote:** [https://github.com/pyannote/pyannote-audio](https://github.com/pyannote/pyannote-audio)
* **faster-whisper:** [https://github.com/SYSTRAN/faster-whisper](https://github.com/SYSTRAN/faster-whisper)
* **ElevenLabs:** [https://elevenlabs.io/docs/introduction](https://elevenlabs.io/docs/introduction)
Raw data
{
"_id": null,
"home_page": "https://github.com/google-marketing-solutions/ariel",
"name": "gtech-ariel",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "python ai genai speech-to-text translation text-to-speech video dubbing youtube gcp",
"author": "Google EMEA gTech Ads Data Science Team",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/cf/e0/b8f45ad66e028b52b72291d93cc38a37514aa866e58d4b83002b466b7b24/gtech_ariel-0.0.25.tar.gz",
"platform": null,
"description": "# gTech Ads Ariel for AI Video Ad Dubbing\n\n### Ariel is an open-source Python library that facilitates efficient and cost-effective dubbing of video ads into multiple languages.\n\n[](https://www.python.org)\n[](https://pypi.org/project/gtech-ariel/)\n[](https://github.com/google-marketing-solutions/ariel/commits)\n[](https://google.github.io/styleguide/pyguide.html)\n[](https://colab.research.google.com/github/google-marketing-solutions/ariel/blob/main/examples/dubbing_workflow.ipynb)\n\n<div align=\"center\">\n <img src=\"ariel_logo.png\" alt=\"Ariel Logo\" width=\"300\">\n</div>\n\n##### _This is not an official Google product._\n\n[Overview](#overview) \u2022\n[Features](#features) \u2022\n[Benefits](#benefits) \u2022\n[Language Compatibility](#language-compatibility) \u2022\n[Before You Begin](#before-you-begin) \u2022\n[Getting Started](#getting-started) \u2022\n[Building Blocks](#building-blocks) \u2022\n[References](#references)\n\n## Overview\n\nAriel is a cutting-edge solution designed to enhance the global reach of digital advertising. It enables advertisers to automate the translation and dubbing of their video ads into a wide range of languages.\n\n## Features\n\n* **Automated Dubbing:** Streamline the generation of high-quality dubbed versions of video ads in various target languages.\n* **Scalability:** Handle large volumes of videos and diverse languages efficiently.\n* **User-Friendly:** Offers a straightforward API and/or user interface for simplified operation.\n* **Cost-Effective:** Significantly reduce dubbing costs compared to traditional methods. The primary expenses are limited to Gemini API and Text-To-Speech API calls.\n\n## Benefits\n\n* **Enhanced Ad Performance:** Improve viewer engagement and potentially increase conversion rates with localized ads.\n* **Streamlined Production:** Minimize the time and cost associated with manual translation and voiceover work.\n* **Rapid Turnaround:** Quickly generate dubbed versions of ads to accelerate multilingual campaign deployment.\n* **Expanded Global Reach:** Reach broader audiences worldwide with localized advertising content.\n\n## Language Compatibility\n\nYou can dub video ads from and to the following languages:\n\n* Arabic (ar-SA), (ar-EG)\n* Bengali (bn-BD), (bn-IN)\n* Bulgarian (bg-BG)\n* Chinese (Simplified) (zh-CN)\n* Chinese (Traditional) (zh-TW)\n* Croatian (hr-HR)\n* Czech (cs-CZ)\n* Danish (da-DK)\n* Dutch (nl-NL)\n* English (en-US), (en-GB), (en-CA), (en-AU)\n* Estonian (et-EE)\n* Finnish (fi-FI)\n* French (fr-FR), (fr-CA)\n* German (de-DE)\n* Greek (el-GR)\n* Gujarati (gu-IN)\n* Hebrew (he-IL) (Note: Not supported with ElevenLabs API)\n* Hindi (hi-IN)\n* Hungarian (hu-HU)\n* Indonesian (id-ID)\n* Italian (it-IT)\n* Japanese (ja-JP)\n* Kannada (kn-IN)\n* Korean (ko-KR)\n* Latvian (lv-LV)\n* Lithuanian (lt-LT)\n* Malayalam (ml-IN)\n* Marathi (mr-IN)\n* Norwegian (nb-NO), (nn-NO)\n* Polish (pl-PL)\n* Portuguese (pt-PT), (pt-BR)\n* Romanian (ro-RO)\n* Russian (ru-RU)\n* Serbian (sr-RS)\n* Slovak (sk-SK)\n* Slovenian (sl-SI)\n* Spanish (es-ES), (es-MX)\n* Swahili (sw-KE)\n* Swedish (sv-SE)\n* Tamil (ta-IN), (ta-LK)\n* Telugu (te-IN)\n* Thai (th-TH)\n* Turkish (tr-TR)\n* Ukrainian (uk-UA)\n* Vietnamese (vi-VN)\n\n## Before You Begin\n\n* **System Requirements:**\n * **Installed FFmpeg:** For video and audio processing. You don't need to install this if you're running from Google Colab.\n * **GPU (Recommended):** For optimal performance, especially with larger videos. It's available for free in Google Colab.\n* **Accounts and Tokens:**\n * **Google Cloud Platform (GCP) Project:** Set up a GCP project. See [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects) for instructions.\n * **Enabled Vertex AI API:** Enable the Vertex API in your GCP project. See [here](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal) for instructions.\n * **Enabled Cloud Storage API:** Enable the Cloud Storage API in your GCP project. See [here](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal) for instructions.\n * You need access to create and delete Google Cloud Storage (GCS) buckets.\n * **Enabled Text-To-Speech API:** Enable the Text-To-Speech API in your GCP project if you choose it for the Text-To-Speech part of the process. See [here](https://cloud.google.com/text-to-speech/docs/before-you-begin) for instructions.\n * **Google Drive API:** Enable the Google Drive API if you use the demo notebook called 'dubbing_workflow.ipynb in Google Colab.\n * **Google Sheets API:** Enable the Google Sheets API if you use the demo notebook called 'dubbing_workflow.ipynb in Google Colab and want to pass script voice metadata from Google Sheets.\n * **Hugging Face Token:** To access the PyAnnote speaker diarization model. See [here](https://huggingface.co/docs/hub/en/security-tokens) on how to get the token.\n * **Hugging Face Model License:** You must accept the user conditions for the PyAnnote speaker diarization [here](https://huggingface.co/pyannote/speaker-diarization-3.1) and segmentation models [here](https://huggingface.co/pyannote/segmentation-3.0).\n * **[OPTIONAL] ElevenLabs API:** To access the ElevenLabs API. See [here](https://help.elevenlabs.io/hc/en-us/articles/14599447207697-How-to-authorize-yourself-using-your-xi-api-key).\n * **Commercial Use:** You are responsible for selecting the right ElevenLabs license if you decide to use the outputs from Ariel in a commercial setting. See the pricing [here](https://elevenlabs.io/pricing). Else, ElevenLabs is free to use.\n* **Data handling:**\n * **Input files:** Ariel can work with both video and audio ads and they need to be in the MP4 or MP3 file formats respectively. See [here](https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-flash) for the limitations on the duration of the ads enforeced by Gemini models.\n * **Storage:** Ariel reads, processes and saves all the files in your environment, e.g. Colab. Only on one instance it creates a temporary Google Cloud Storage (GCS) bucket to upload the input file there for the Gemini model to indentify unique speakers. The bucket with all its contents is removed immediately afterwards. You can modify the `gcp_region` argument to choose the best location to perform this operation.\n * **Voice cloning:** You can clone voices from the input file to make the dubbing sound close to the original if you decide to use ElevenLabs. It's is your responsibility to ensure that you are legally allowed to do so.\n\n## Getting started\n\nTo start using Ariel, just click on the this button: [](https://colab.research.google.com/github/google-marketing-solutions/ariel/blob/main/examples/dubbing_workflow.ipynb)\n\n## Building Blocks\n\nAriel leverages a powerful combination of state-of-the-art AI and audio processing techniques to deliver accurate and efficient dubbing results:\n\n1. **Video Processing:** Extracts the audio track from the input video file.\n2. **Audio Processing:**\n * **DEMUCS:** Employed for advanced audio source separation.\n * **pyannote:** Performs speaker diarization to identify and separate individual speakers.\n3. **Speech-To-Text (STT):**\n * **faster-whisper:** A high-performance speech-to-text model.\n * **Gemini 1.5 Flash:** A powerful multimodal language model that contributes to enhanced transcription.\n4. **Translation:**\n * **Gemini 1.5 Flash:** Leverages its language understanding for accurate and contextually relevant translation.\n5. **Text-to-Speech (TTS):**\n * **GCP's Text-To-Speech:** Generates natural-sounding speech in the target language.\n * **[OPTIONAL] ElevenLabs:** An alternative API to generate speech. It's recommened for the best results. **WARNING:** ElevenLabs is a paid solution and will generate extra costs. See the pricing [here](https://elevenlabs.io/pricing).\n\n## Cloud-based Ariel\nFor a more user-friendly experience, we're also providing an Ariel version that\ncan be deployed onto Google Cloud Platform with a GUI.\n### Requirements\nIn order to use Ariel UI, you need the following:\n1. **Google Cloud Platform Project** to host the backend. The following components\n are created during deployment:\n * **Cloud Storage Bucket** to store input & output videos, video dubbing artifacts and\n all other metadata files. This bucket is also used as an interaction point with GUI\n (files created/removed there are triggering the backend processing).\n * **Cloud Run** instance that processes all the steps. This part is implemented as a\n Python Docker container (which is also built during installation).\n * **Pub/Sub Infrastructure** based on EventArc, it notifies the container of new files.\n2. **AppsScript Project** to host the frontend, an Angular web-app. For this,\n you need *Google Workspace* access.\n\n### Deployment\nPlease make sure you have fulfilled all prerequisites mentioned under [Requirements](#Requirements) first.\n\n1. Make sure your system has an up-to-date installation of [Node.js and npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).\n1. Install [clasp](https://github.com/google/clasp) by running `npm install @google/clasp -g`, then login via `clasp login`.\n1. Navigate to the [Apps Script Settings page](https://script.google.com/home/usersettings) and `enable` the Apps Script API.\n1. Make sure your system has an up-to-date installation of the [gcloud CLI](https://cloud.google.com/sdk/docs/install), then login via `gcloud auth login`.\n1. Make sure your system has an up-to-date installation of `git` and use it to clone this repository:\n `git clone https://github.com/google-marketing-solutions/ariel`.\n1. Navigate to the directory where the source code lives: `cd ariel`.\n1. Run `npm start`. This will prompt you for configuration values and suggest reasonable defaults.\n\n## References\n\n* **DEMUCS:** [https://github.com/facebookresearch/demucs](https://github.com/facebookresearch/demucs)\n* **pyannote:** [https://github.com/pyannote/pyannote-audio](https://github.com/pyannote/pyannote-audio)\n* **faster-whisper:** [https://github.com/SYSTRAN/faster-whisper](https://github.com/SYSTRAN/faster-whisper)\n* **ElevenLabs:** [https://elevenlabs.io/docs/introduction](https://elevenlabs.io/docs/introduction)\n",
"bugtrack_url": null,
"license": "Apache Software License 2.0",
"summary": "Google EMEA gTech Ads Data Science Team's solution to automatically translate and dub video ads into multiple languages using AI.",
"version": "0.0.25",
"project_urls": {
"Homepage": "https://github.com/google-marketing-solutions/ariel"
},
"split_keywords": [
"python",
"ai",
"genai",
"speech-to-text",
"translation",
"text-to-speech",
"video",
"dubbing",
"youtube",
"gcp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "899e9a9f17368a78ae26bcae17ee6e9573629f1b8191cdcd8b7831daa3196dee",
"md5": "77c8e93555e73022eeec0fca8aed0895",
"sha256": "040168e83e1fc42919e2022109b285067494e62a69915b356519b448274d928b"
},
"downloads": -1,
"filename": "gtech_ariel-0.0.25-py3-none-any.whl",
"has_sig": false,
"md5_digest": "77c8e93555e73022eeec0fca8aed0895",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 70594,
"upload_time": "2024-11-21T13:57:40",
"upload_time_iso_8601": "2024-11-21T13:57:40.505064Z",
"url": "https://files.pythonhosted.org/packages/89/9e/9a9f17368a78ae26bcae17ee6e9573629f1b8191cdcd8b7831daa3196dee/gtech_ariel-0.0.25-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cfe0b8f45ad66e028b52b72291d93cc38a37514aa866e58d4b83002b466b7b24",
"md5": "778938ed1e8c759c4c9b1b966ca57a26",
"sha256": "42b1b261b48fc2fc4997693bc0cadcce272748c91cc4c2d5d541150dee8dfb51"
},
"downloads": -1,
"filename": "gtech_ariel-0.0.25.tar.gz",
"has_sig": false,
"md5_digest": "778938ed1e8c759c4c9b1b966ca57a26",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 67691,
"upload_time": "2024-11-21T13:57:41",
"upload_time_iso_8601": "2024-11-21T13:57:41.601013Z",
"url": "https://files.pythonhosted.org/packages/cf/e0/b8f45ad66e028b52b72291d93cc38a37514aa866e58d4b83002b466b7b24/gtech_ariel-0.0.25.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-21 13:57:41",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "google-marketing-solutions",
"github_project": "ariel",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "gtech-ariel"
}
JSON
