# ytdl-server
ytdl-server is a program that allows you to download videos onto a remote server
using youtube-dl.
Using youtube-dl to download videos onto a network file share is inefficient
since the video file has to pass through your workstation. Additionally, many of
the post-processing options that youtube-dl provides will create a temporary
copy of the file, which means that the amount of data uploaded over the network
can be 3-4x the size of the actual file.
ytdl-server solves this by allowing you to download videos directly to the
remote server via a REST API.
ytdl-server also supports high-availability, which means that you can perform
maintenance without downtime.
[TOC]
## How it works
ytdl-server has two main components: the Flask REST API and the Celery worker.
The REST API communicates with the user in order to create jobs. The jobs are
then run by the worker in order to download the desired videos.
The complete structure of ytdl-server is shown in the below diagram:
*Included in ytdl-server* means that the component is part of ytdl-server.
*Included in the Docker images* means that the component is built into the Docker images. You'll need to install it separately if you're not using them.
*Not included* means that the component is not part of ytdl-server and needs to
be installed separately.
- Reverse proxy / Load balancer (*not included*)
Used to forward user-requests to the WSGI server(s).
Some available options are [NGINX](https://www.nginx.com/) and
[Caddy](https://caddyserver.com/).
Load balancing is only needed if you're running multiple WSGI server
instances.
- WSGI server (*included in the Docker images*)
Used to run the Flask REST API(s).
Some available options are [Gunicorn][gunicorn] and
[gevent](https://www.gevent.org/).
[gunicorn]: https://gunicorn.org/
If you're using the Docker images, Gunicorn is included.
If you're running multiple REST API instances, each one needs its own WSGI
server instance.
- Flask REST API (*included in ytdl-server*)
Used to create and manage jobs.
You can run multiple instances if you want high-availability. Otherwise, a
single instance is fine.
- Celery broker (*not included*)
Used to dispatch jobs created by the REST API to a Celery worker.
You can use any broker [supported by Celery][celery_brokers], including
[RabbitMQ](https://www.rabbitmq.com/) and [Redis][redis].
[celery_brokers]: https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html#broker-overview
[redis]: https://redis.io/
- Celery worker (*included in ytdl-server*)
Used to run jobs (download videos).
You can run multiple instances if you want high-availability. Otherwise, a
single instance is fine.
- PostgreSQL database (*not included*)
Used to store information about jobs, and also functions as the Celery result
backend.
## Installation
### Docker (recommended)
Prebuilt Docker images are provided. See [Registry](#registry) for a list of
supported tags.
An example Docker Compose file is provided [here](docker-compose.yml).
To use it, copy the Compose file to a directory, and run the following command
to start it:
```bash
docker-compose up
```
The REST API will be accessible at port 8000. See [Usage](#usage) for how to use
it.
If you want to use a custom config file, overwrite the existing config file at
`/config/ytdl-server.yml`. See [Configuration](#configuration) for available
options.
### Manual
ytdl-server requires [Python](https://www.python.org/) 3.9+.
Install from [PyPI](https://pypi.org/project/ytdl-server/):
```bash
pip3 install 'ytdl-server[dbapi]'
```
Install from source:
```bash
git clone 'https://gitlab.com/adralioh/ytdl-server.git'
pip3 install './ytdl-server[dbapi]'
```
> The ``[dbapi]`` extra installs [Psycopg2](https://www.psycopg.org/), which is
used to connect to the PostgreSQL database. You must install a DBAPI for
ytdl-server to work. You can alternatively install ``psycopg2-binary`` if you
don't want to compile it from source. See
[Installation - Psycopg 2](https://www.psycopg.org/docs/install.html) for
details.
If you're using [Redis][redis] or [Amazon SQS](https://aws.amazon.com/sqs/) as
the broker, you must also install the following extra dependencies:
```bash
# Redis
pip3 install 'celery[redis]'
# Amazon SQS
pip3 install 'celery[sqs]'
```
Run the REST API using a WSGI server such as [Gunicorn][gunicorn]:
```bash
gunicorn -w 4 'ytdl_server.flask:with_logging()'
```
Run the Celery worker using Celery:
```bash
celery -A 'ytdl_server.run_celery' worker
```
You also need to set up the reverse proxy, Celery broker, and PostgreSQL
database. See [How it works](#how-it-works) for information, and see
[Configuration](#configuration) for how to configure ytdl-server to use the
database and broker.
## Updating
When updating ytdl-server, refer to the
[versioning scheme](https://adralioh.gitlab.io/ytdl-server/version.html) and the
[changelog](CHANGELOG.md) for information about what has changed.
## Configuration
See [Configuration](https://adralioh.gitlab.io/ytdl-server/config.html).
## Usage
Videos are downloaded via jobs. You create a new job whenever you want to
download one or more videos.
> The following examples access the ytdl-server directly using curl. See
[Frontends](#frontends) for more user-friendly ways of accessing it.
Create a job:
```bash
curl ytdl.example.com/jobs/ \
-H 'Content-Type: application/json' \
-d '{
"urls": [ "https://www.youtube.com/watch?v=AAAAAAAAAAA",
"https://www.youtube.com/watch?v=BBBBBBBBBBB" ],
"ytdl_opts": { "format": "best",
"outtmpl": "%(id)s.%(ext)s" }
}'
```
Jobs are created asynchronously, which means that the videos will be downloaded
in the background.
You can check the status of a job as follows:
```bash
curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942
```
*ffedda2d-7fa4-4839-a705-88c893e3f942* is the job ID. This is obtained from the
output of the previous command where you created the job.
You can also cancel a running job:
```bash
curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942 \
-X PATCH \
-H 'Content-Type: application/json' \
-d '{
"status": "cancelled"
}'
```
> For complete documentation about the REST API, see
[REST API](https://adralioh.gitlab.io/ytdl-server/api.html).
## Registry
Prebuit Docker images are provided at `registry.gitlab.com/adralioh/ytdl-server`
for the REST API and the Celery worker.
The supported tags are rebuilt monthly in order to provide the latest security
updates.
### REST API
Images for the REST API are available at
`registry.gitlab.com/adralioh/ytdl-server/api`.
The images include [Gunicorn][gunicorn], which is used as the WSGI server.
Supported tags:
- `latest`, `1`, `1.1`, `1.1.7`
### Celery worker
Images for the Celery worker are available at
`registry.gitlab.com/adralioh/ytdl-server/worker`.
The *yt_dlp* tags use [yt-dlp](https://github.com/yt-dlp/yt-dlp) instead of
youtube-dl. You also need to set the *YTDL_MODULE* env-var to `yt_dlp` on the
REST API when using these.
The *ffmpeg* tags include [FFmpeg](https://www.ffmpeg.org/), which is required
for many of youtube-dl's post-processing options.
Supported tags:
- `latest`, `1`, `1.1`, `1.1.7`
- `yt_dlp`, `1-yt_dlp`, `1.1-yt_dlp`, `1.1.7-yt_dlp`
- `ffmpeg`, `1-ffmpeg`, `1.1-ffmpeg`, `1.1.7-ffmpeg`
- `yt_dlp-ffmpeg`, `1-yt_dlp-ffmpeg`, `1.1-yt_dlp-ffmpeg`, `1.1.7-yt_dlp-ffmpeg`
## Testing and development
See [tests](tests)
## Frontends
- [ytcl](https://gitlab.com/adralioh/ytcl) - Command-line interface with syntax
similar to youtube-dl
## Frequently asked questions
See
[Frequently asked questions](https://adralioh.gitlab.io/ytdl-server/faq.html).
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/adralioh/ytdl-server",
"name": "ytdl-server",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "",
"keywords": "youtube-dl,yt-dlp,youtube,download,rest,api",
"author": "Adralioh",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/76/7e/77dbda09e96e4766465de1951af39f519330be3fa0b57a32fd355fc34dff/ytdl-server-1.1.7.tar.gz",
"platform": null,
"description": "# ytdl-server\nytdl-server is a program that allows you to download videos onto a remote server\nusing youtube-dl.\n\nUsing youtube-dl to download videos onto a network file share is inefficient\nsince the video file has to pass through your workstation. Additionally, many of\nthe post-processing options that youtube-dl provides will create a temporary\ncopy of the file, which means that the amount of data uploaded over the network\ncan be 3-4x the size of the actual file.\n\nytdl-server solves this by allowing you to download videos directly to the\nremote server via a REST API.\n\nytdl-server also supports high-availability, which means that you can perform\nmaintenance without downtime.\n\n[TOC]\n\n## How it works\nytdl-server has two main components: the Flask REST API and the Celery worker.\n\nThe REST API communicates with the user in order to create jobs. The jobs are\nthen run by the worker in order to download the desired videos.\n\nThe complete structure of ytdl-server is shown in the below diagram:\n\n\n\n*Included in ytdl-server* means that the component is part of ytdl-server.\n\n*Included in the Docker images* means that the component is built into the Docker images. You'll need to install it separately if you're not using them.\n\n*Not included* means that the component is not part of ytdl-server and needs to\nbe installed separately.\n\n- Reverse proxy / Load balancer (*not included*)\n\n Used to forward user-requests to the WSGI server(s).\n\n Some available options are [NGINX](https://www.nginx.com/) and\n [Caddy](https://caddyserver.com/).\n\n Load balancing is only needed if you're running multiple WSGI server\n instances.\n\n- WSGI server (*included in the Docker images*)\n\n Used to run the Flask REST API(s).\n\n Some available options are [Gunicorn][gunicorn] and\n [gevent](https://www.gevent.org/).\n\n [gunicorn]: https://gunicorn.org/\n\n If you're using the Docker images, Gunicorn is included.\n\n If you're running multiple REST API instances, each one needs its own WSGI\n server instance.\n\n- Flask REST API (*included in ytdl-server*)\n\n Used to create and manage jobs.\n\n You can run multiple instances if you want high-availability. Otherwise, a\n single instance is fine.\n\n- Celery broker (*not included*)\n\n Used to dispatch jobs created by the REST API to a Celery worker.\n\n You can use any broker [supported by Celery][celery_brokers], including\n [RabbitMQ](https://www.rabbitmq.com/) and [Redis][redis].\n\n [celery_brokers]: https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html#broker-overview\n [redis]: https://redis.io/\n\n- Celery worker (*included in ytdl-server*)\n\n Used to run jobs (download videos).\n\n You can run multiple instances if you want high-availability. Otherwise, a\n single instance is fine.\n\n- PostgreSQL database (*not included*)\n\n Used to store information about jobs, and also functions as the Celery result\n backend.\n\n## Installation\n### Docker (recommended)\nPrebuilt Docker images are provided. See [Registry](#registry) for a list of\nsupported tags.\n\nAn example Docker Compose file is provided [here](docker-compose.yml).\nTo use it, copy the Compose file to a directory, and run the following command\nto start it:\n```bash\ndocker-compose up\n```\n\nThe REST API will be accessible at port 8000. See [Usage](#usage) for how to use\nit.\n\nIf you want to use a custom config file, overwrite the existing config file at\n`/config/ytdl-server.yml`. See [Configuration](#configuration) for available\noptions.\n\n### Manual\nytdl-server requires [Python](https://www.python.org/) 3.9+.\n\nInstall from [PyPI](https://pypi.org/project/ytdl-server/):\n```bash\npip3 install 'ytdl-server[dbapi]'\n```\n\nInstall from source:\n```bash\ngit clone 'https://gitlab.com/adralioh/ytdl-server.git'\npip3 install './ytdl-server[dbapi]'\n```\n\n> The ``[dbapi]`` extra installs [Psycopg2](https://www.psycopg.org/), which is\n used to connect to the PostgreSQL database. You must install a DBAPI for\n ytdl-server to work. You can alternatively install ``psycopg2-binary`` if you\n don't want to compile it from source. See\n [Installation - Psycopg 2](https://www.psycopg.org/docs/install.html) for\n details.\n\nIf you're using [Redis][redis] or [Amazon SQS](https://aws.amazon.com/sqs/) as\nthe broker, you must also install the following extra dependencies:\n```bash\n# Redis\npip3 install 'celery[redis]'\n# Amazon SQS\npip3 install 'celery[sqs]'\n```\n\nRun the REST API using a WSGI server such as [Gunicorn][gunicorn]:\n```bash\ngunicorn -w 4 'ytdl_server.flask:with_logging()'\n```\n\nRun the Celery worker using Celery:\n```bash\ncelery -A 'ytdl_server.run_celery' worker\n```\n\nYou also need to set up the reverse proxy, Celery broker, and PostgreSQL\ndatabase. See [How it works](#how-it-works) for information, and see\n[Configuration](#configuration) for how to configure ytdl-server to use the\ndatabase and broker.\n\n## Updating\nWhen updating ytdl-server, refer to the\n[versioning scheme](https://adralioh.gitlab.io/ytdl-server/version.html) and the\n[changelog](CHANGELOG.md) for information about what has changed.\n\n## Configuration\nSee [Configuration](https://adralioh.gitlab.io/ytdl-server/config.html).\n\n## Usage\nVideos are downloaded via jobs. You create a new job whenever you want to\ndownload one or more videos.\n\n> The following examples access the ytdl-server directly using curl. See\n [Frontends](#frontends) for more user-friendly ways of accessing it.\n\nCreate a job:\n```bash\ncurl ytdl.example.com/jobs/ \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"urls\": [ \"https://www.youtube.com/watch?v=AAAAAAAAAAA\",\n \"https://www.youtube.com/watch?v=BBBBBBBBBBB\" ],\n \"ytdl_opts\": { \"format\": \"best\",\n \"outtmpl\": \"%(id)s.%(ext)s\" }\n }'\n```\n\nJobs are created asynchronously, which means that the videos will be downloaded\nin the background.\n\nYou can check the status of a job as follows:\n```bash\ncurl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942\n```\n\n*ffedda2d-7fa4-4839-a705-88c893e3f942* is the job ID. This is obtained from the\noutput of the previous command where you created the job.\n\nYou can also cancel a running job:\n```bash\ncurl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942 \\\n -X PATCH \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"status\": \"cancelled\"\n }'\n```\n\n> For complete documentation about the REST API, see\n [REST API](https://adralioh.gitlab.io/ytdl-server/api.html).\n\n## Registry\nPrebuit Docker images are provided at `registry.gitlab.com/adralioh/ytdl-server`\nfor the REST API and the Celery worker.\n\nThe supported tags are rebuilt monthly in order to provide the latest security\nupdates.\n\n### REST API\nImages for the REST API are available at\n`registry.gitlab.com/adralioh/ytdl-server/api`.\n\nThe images include [Gunicorn][gunicorn], which is used as the WSGI server.\n\nSupported tags:\n- `latest`, `1`, `1.1`, `1.1.7`\n\n### Celery worker\nImages for the Celery worker are available at\n`registry.gitlab.com/adralioh/ytdl-server/worker`.\n\nThe *yt_dlp* tags use [yt-dlp](https://github.com/yt-dlp/yt-dlp) instead of\nyoutube-dl. You also need to set the *YTDL_MODULE* env-var to `yt_dlp` on the\nREST API when using these.\n\nThe *ffmpeg* tags include [FFmpeg](https://www.ffmpeg.org/), which is required\nfor many of youtube-dl's post-processing options.\n\nSupported tags:\n- `latest`, `1`, `1.1`, `1.1.7`\n- `yt_dlp`, `1-yt_dlp`, `1.1-yt_dlp`, `1.1.7-yt_dlp`\n- `ffmpeg`, `1-ffmpeg`, `1.1-ffmpeg`, `1.1.7-ffmpeg`\n- `yt_dlp-ffmpeg`, `1-yt_dlp-ffmpeg`, `1.1-yt_dlp-ffmpeg`, `1.1.7-yt_dlp-ffmpeg`\n\n## Testing and development\nSee [tests](tests)\n\n## Frontends\n- [ytcl](https://gitlab.com/adralioh/ytcl) - Command-line interface with syntax\n similar to youtube-dl\n\n## Frequently asked questions\nSee\n[Frequently asked questions](https://adralioh.gitlab.io/ytdl-server/faq.html).\n",
"bugtrack_url": null,
"license": "",
"summary": "Download videos to a remote server using youtube-dl via a REST API",
"version": "1.1.7",
"project_urls": {
"Documentation": "https://adralioh.gitlab.io/ytdl-server",
"Homepage": "https://gitlab.com/adralioh/ytdl-server"
},
"split_keywords": [
"youtube-dl",
"yt-dlp",
"youtube",
"download",
"rest",
"api"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "89d86e6ad4fd9ca050e547f11bcdc40b332f2822abf1019b472726f2595028f1",
"md5": "253e124a89d4b8807f85b66a8387203a",
"sha256": "2fb345e4dd19e3fce03ded3441cbd61b7174622a91e4037b48a3977fc57dbe72"
},
"downloads": -1,
"filename": "ytdl_server-1.1.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "253e124a89d4b8807f85b66a8387203a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 59485,
"upload_time": "2023-05-30T06:49:22",
"upload_time_iso_8601": "2023-05-30T06:49:22.882040Z",
"url": "https://files.pythonhosted.org/packages/89/d8/6e6ad4fd9ca050e547f11bcdc40b332f2822abf1019b472726f2595028f1/ytdl_server-1.1.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "767e77dbda09e96e4766465de1951af39f519330be3fa0b57a32fd355fc34dff",
"md5": "14bdab2b3a29772fb1ff5bb519c4bfe8",
"sha256": "cc320cc370d25a07410866035547ff5dd6ee07d886b7e74f588bf774d450e393"
},
"downloads": -1,
"filename": "ytdl-server-1.1.7.tar.gz",
"has_sig": false,
"md5_digest": "14bdab2b3a29772fb1ff5bb519c4bfe8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 53564,
"upload_time": "2023-05-30T06:49:24",
"upload_time_iso_8601": "2023-05-30T06:49:24.356095Z",
"url": "https://files.pythonhosted.org/packages/76/7e/77dbda09e96e4766465de1951af39f519330be3fa0b57a32fd355fc34dff/ytdl-server-1.1.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-30 06:49:24",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "adralioh",
"gitlab_project": "ytdl-server",
"lcname": "ytdl-server"
}
JSON