[](https://discord.gg/qUtxnK2NMf)
# Gen1
My Implementation of " Structure and Content-Guided Video Synthesis with Diffusion Models" by RunwayML
The flow:
```
image => midas => clip => spacetime unet => diffusion
```
# Install
`pip3 install gen1`
# Usage
```python
import torch
from gen1.model import Gen1
model = Gen1()
images = torch.randn(1, 3, 128, 128)
video = torch.randn(1, 3, 16, 128, 128)
run_out = model.forward(images, video)
```
## Datasets
Here is a summary table of the datasets used in the Structure and Content-Guided Video Synthesis with Diffusion Models paper:
| Dataset | Type | Size | Domain | Description | Source |
|-|-|-|-|-|-|
| Internal dataset | Images | 240M | General | Uncaptioned images | Private |
| Custom video dataset | Videos | 6.4M clips | General | Uncaptioned short video clips | Private |
| DAVIS | Videos | - | General | Video object segmentation | [Link](https://davischallenge.org/) |
| Stock footage | Videos | - | General | Diverse video clips | - |
## Citation
```
@misc{2302.03011,
Author = {Patrick Esser and Johnathan Chiu and Parmida Atighehchian and Jonathan Granskog and Anastasis Germanidis},
Title = {Structure and Content-Guided Video Synthesis with Diffusion Models},
Year = {2023},
Eprint = {arXiv:2302.03011},
```
Raw data
{
"_id": null,
"home_page": "https://github.com/kyegomez/gen1",
"name": "gen1",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "Transformers,zeta scale",
"author": "Gen1",
"author_email": "kye@apac.ai",
"download_url": "https://files.pythonhosted.org/packages/36/4b/aa3fea2124b92cdcf39c5cc6cc8bdb4391e8b0100c0dd4fd395163912e40/gen1-0.0.4.tar.gz",
"platform": null,
"description": "[](https://discord.gg/qUtxnK2NMf)\n\n\n# Gen1\nMy Implementation of \" Structure and Content-Guided Video Synthesis with Diffusion Models\" by RunwayML\n\n\nThe flow:\n\n```\nimage => midas => clip => spacetime unet => diffusion\n```\n\n\n# Install\n`pip3 install gen1`\n\n# Usage\n```python\nimport torch\nfrom gen1.model import Gen1\n\nmodel = Gen1()\n\nimages = torch.randn(1, 3, 128, 128)\nvideo = torch.randn(1, 3, 16, 128, 128)\n\nrun_out = model.forward(images, video)\n\n```\n\n## Datasets\nHere is a summary table of the datasets used in the Structure and Content-Guided Video Synthesis with Diffusion Models paper:\n\n| Dataset | Type | Size | Domain | Description | Source |\n|-|-|-|-|-|-|\n| Internal dataset | Images | 240M | General | Uncaptioned images | Private | \n| Custom video dataset | Videos | 6.4M clips | General | Uncaptioned short video clips | Private |\n| DAVIS | Videos | - | General | Video object segmentation | [Link](https://davischallenge.org/) |\n| Stock footage | Videos | - | General | Diverse video clips | - |\n\n\n\n## Citation\n```\n@misc{2302.03011,\nAuthor = {Patrick Esser and Johnathan Chiu and Parmida Atighehchian and Jonathan Granskog and Anastasis Germanidis},\nTitle = {Structure and Content-Guided Video Synthesis with Diffusion Models},\nYear = {2023},\nEprint = {arXiv:2302.03011},\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Text to Video synthesis",
"version": "0.0.4",
"project_urls": {
"Homepage": "https://github.com/kyegomez/gen1"
},
"split_keywords": [
"transformers",
"zeta scale"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b133b2df800bc8b40f239d4cb305832e227e257abffd186e23648b455b69212c",
"md5": "8f8a0e09380375c2889e626227090b97",
"sha256": "06e6d25367928702f9dca663424331276a21a503963167a514b2e8efad66a6a9"
},
"downloads": -1,
"filename": "gen1-0.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8f8a0e09380375c2889e626227090b97",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 21419,
"upload_time": "2023-09-28T19:20:25",
"upload_time_iso_8601": "2023-09-28T19:20:25.541916Z",
"url": "https://files.pythonhosted.org/packages/b1/33/b2df800bc8b40f239d4cb305832e227e257abffd186e23648b455b69212c/gen1-0.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "364baa3fea2124b92cdcf39c5cc6cc8bdb4391e8b0100c0dd4fd395163912e40",
"md5": "371b19d5d46bfa6a941ccc6e962a618e",
"sha256": "f888ccc4cb30269c73d0a6b35d915798428daa8b33db468f0632b36ec685e747"
},
"downloads": -1,
"filename": "gen1-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "371b19d5d46bfa6a941ccc6e962a618e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 20539,
"upload_time": "2023-09-28T19:20:27",
"upload_time_iso_8601": "2023-09-28T19:20:27.319674Z",
"url": "https://files.pythonhosted.org/packages/36/4b/aa3fea2124b92cdcf39c5cc6cc8bdb4391e8b0100c0dd4fd395163912e40/gen1-0.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-28 19:20:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kyegomez",
"github_project": "gen1",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "gen1"
}
JSON
