| Name | experimental-yt-fts JSON |
| Version |
0.1.49
JSON |
| download |
| home_page | None |
| Summary | Search all of a YouTube channel from the command line |
| upload_time | 2024-04-05 23:37:06 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | This is free and unencumbered software released into the public domain. Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For more information, please refer to <https://unlicense.org> |
| keywords |
youtube
subtitles
search
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Will this deoploy if I make a push to a branch?
what about a pr?
maybe I need to push the active branch with the tag?
1. You make changes in a branch
2. make a pr
3. merge the pr
4. add a tag to the last commit in the main branch
5. push the tag to main
```
git tag -l
git tag v0.1.48
git push origin v0.1.48
```
# yt-fts - Youtube Full Text Search
`yt-fts` is a command line program that uses [yt-dlp](https://github.com/yt-dlp/yt-dlp) to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to
the video containing the keyword.
It also supports semantic search via the [OpenAI embeddings API](https://beta.openai.com/docs/api-reference/) using [chromadb](https://github.com/chroma-core/chroma).
- [Blog Post](https://notjoemartinez.com/blog/youtube_full_text_search/)
- [Semantic Search](#Semantic-Search-via-OpenAI-embeddings-API)
- [CHANGELOG](CHANGELOG.md)
https://github.com/NotJoeMartinez/yt-fts/assets/39905973/6ffd8962-d060-490f-9e73-9ab179402f14
## Installation
```bash
pip install yt-fts
```
**yt-dlp dependency:**
This project requires [yt-dlp](https://github.com/yt-dlp/yt-dlp) installed globally. Platform specific installation instructions are available on the [yt-dlp wiki](https://github.com/yt-dlp/yt-dlp/wiki/Installation).
```bash
# MacOS/Homebrew
brew install yt-dlp
# Windows/winget
winget install yt-dlp
# pip
python3 -m pip install -U yt-dlp
```
## `download`
Download subtitles for a channel.
Takes a channel url or id as an argument. Specify the number of jobs to parallelize the download with the `--number-of-jobs` option.
```bash
yt-fts download --number-of-jobs 5 "https://www.youtube.com/@3blue1brown"
```
## `list`
List saved channels.
The (ss) next to the channel name indicates that the channel has semantic search enabled.
```bash
yt-fts list
```
```
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ID ┃ Name ┃ Count ┃ Channel ID ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1 │ ChessPage1 (ss) │ 19 │ UCO2QPmnJFjdvJ6ch-pe27dQ │
│ 2 │ 3Blue1Brown │ 127 │ UCYO_jab_esuFRV4b17AJtAw │
│ 3 │ george hotz archive │ 410 │ UCwgKmJM4ZJQRJ-U5NjvR2dg │
│ 4 │ The Tim Dillon Show │ 288 │ UC4woSp8ITBoYDmjkukhEhxg │
│ 5 │ Academy of Ideas (ss) │ 190 │ UCiRiQGCHGjDLT9FQXFW0I3A │
└────┴───────────────────────┴───────┴──────────────────────────┘
```
## `search` (Full Text Search)
Full text search for a string in saved channels.
- The search string does not have to be a word for word and match
- Search strings are limited to 40 characters.
```bash
# search in all channels
yt-fts search "[search query]"
# search in channel
yt-fts search "[search query]" --channel "[channel name or id]"
# search in specific video
yt-fts search "[search query]" --video "[video id]"
# limit results
yt-fts search "[search query]" --limit "[number of results]" --channel "[channel name or id]"
# export results to csv
yt-fts search "[search query]" --export --channel "[channel name or id]"
```
Advanced Search Syntax:
The search string supports sqlite [Enhanced Query Syntax](https://www.sqlite.org/fts3.html#full_text_index_queries).
which includes things like [prefix queries](https://www.sqlite.org/fts3.html#termprefix) which you can use to match parts of a word.
```bash
# AND search
yt-fts search "knife AND Malibu" --channel "The Tim Dillon Show"
# OR SEARCH
yt-fts search "knife OR Malibu" --channel "The Tim Dillon Show"
# wild cards
yt-fts search "rea* kni* Mali*" --channel "The Tim Dillon Show"
```
# Semantic Search
You can enable semantic search for a channel by using the `get-embeddings` command.
This requires an OpenAI API key set in the environment variable `OPENAI_API_KEY`, or
you can pass the key with the `--openai-api-key` flag.
## `get-embedings`
Fetches OpenAI embeddings for specified channel
```bash
# make sure openAI key is set
# export OPENAI_API_KEY="[yourOpenAIKey]"
yt-fts get-embeddings --channel "3Blue1Brown"
```
After the embeddings are saved you will see a `(ss)` next to the channel name when you
list channels and you will be able to use the `vsearch` command for that channel.
## `vsearch` (Semantic Search)
`vsearch` is for "Vector search". This requires that you enable semantic
search for a channel with `get-embeddings`. It has the same options as
`search` but output will be sorted by similarity to the search string and
the default return limit is 10.
```bash
# search by channel name
yt-fts vsearch "[search query]" --channel "[channel name or id]"
# search in specific video
yt-fts vsearch "[search query]" --video "[video id]"
# limit results
yt-fts vsearch "[search query]" --limit "[number of results]" --channel "[channel name or id]"
# export results to csv
yt-fts vsearch "[search query]" --export --channel "[channel name or id]"
```
## How To
**Export search results:**
For both the `search` and `vsearch` commands you can export the results to a csv file with
the `--export` flag. and it will save the results to a csv file in the current directory.
```bash
yt-fts search "life in the big city" --export
yt-fts vsearch "existing in large metropolaten center" --export
```
**Delete a channel:**
You can delete a channel with the `delete` command.
```bash
yt-fts delete --channel "3Blue1Brown"
```
**Update a channel:**
The update command currently only works for full text search and will not update the
semantic search embeddings.
```bash
yt-fts update --channel "3Blue1Brown"
```
**Export all of a channel's transcript:**
This command will create a directory in current working directory with the youtube
channel id of the specified channel.
```bash
# Export to vtt
yt-fts export --channel "[id/name]" --format "[vtt/txt]"
```
Raw data
{
"_id": null,
"home_page": null,
"name": "experimental-yt-fts",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "youtube, subtitles, search",
"author": null,
"author_email": "NotJoeMartinez <notjoemartinez@protonmail.com>",
"download_url": "https://files.pythonhosted.org/packages/22/27/2ea88a17fd5701a6e07218b994c8982f8651bbe38718f229f9a11305a1f3/experimental-yt-fts-0.1.49.tar.gz",
"platform": null,
"description": "Will this deoploy if I make a push to a branch?\nwhat about a pr?\nmaybe I need to push the active branch with the tag?\n\n1. You make changes in a branch\n2. make a pr\n3. merge the pr\n4. add a tag to the last commit in the main branch\n5. push the tag to main\n\n```\ngit tag -l\ngit tag v0.1.48\ngit push origin v0.1.48\n```\n\n# yt-fts - Youtube Full Text Search \n`yt-fts` is a command line program that uses [yt-dlp](https://github.com/yt-dlp/yt-dlp) to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to\nthe video containing the keyword. \n\nIt also supports semantic search via the [OpenAI embeddings API](https://beta.openai.com/docs/api-reference/) using [chromadb](https://github.com/chroma-core/chroma).\n\n- [Blog Post](https://notjoemartinez.com/blog/youtube_full_text_search/)\n- [Semantic Search](#Semantic-Search-via-OpenAI-embeddings-API) \n- [CHANGELOG](CHANGELOG.md)\n\nhttps://github.com/NotJoeMartinez/yt-fts/assets/39905973/6ffd8962-d060-490f-9e73-9ab179402f14\n\n## Installation \n\n```bash\npip install yt-fts\n```\n\n**yt-dlp dependency:**\n\nThis project requires [yt-dlp](https://github.com/yt-dlp/yt-dlp) installed globally. Platform specific installation instructions are available on the [yt-dlp wiki](https://github.com/yt-dlp/yt-dlp/wiki/Installation). \n\n```bash\n# MacOS/Homebrew\nbrew install yt-dlp\n# Windows/winget\nwinget install yt-dlp\n# pip\npython3 -m pip install -U yt-dlp\n```\n\n## `download`\nDownload subtitles for a channel. \n\nTakes a channel url or id as an argument. Specify the number of jobs to parallelize the download with the `--number-of-jobs` option. \n\n```bash\nyt-fts download --number-of-jobs 5 \"https://www.youtube.com/@3blue1brown\"\n```\n\n## `list`\nList saved channels.\n\nThe (ss) next to the channel name indicates that the channel has semantic search enabled. \n\n```bash\nyt-fts list\n```\n\n```\n\u250f\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 ID \u2503 Name \u2503 Count \u2503 Channel ID \u2503\n\u2521\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 1 \u2502 ChessPage1 (ss) \u2502 19 \u2502 UCO2QPmnJFjdvJ6ch-pe27dQ \u2502\n\u2502 2 \u2502 3Blue1Brown \u2502 127 \u2502 UCYO_jab_esuFRV4b17AJtAw \u2502\n\u2502 3 \u2502 george hotz archive \u2502 410 \u2502 UCwgKmJM4ZJQRJ-U5NjvR2dg \u2502\n\u2502 4 \u2502 The Tim Dillon Show \u2502 288 \u2502 UC4woSp8ITBoYDmjkukhEhxg \u2502\n\u2502 5 \u2502 Academy of Ideas (ss) \u2502 190 \u2502 UCiRiQGCHGjDLT9FQXFW0I3A \u2502\n\u2514\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n```\n\n## `search` (Full Text Search)\nFull text search for a string in saved channels.\n\n- The search string does not have to be a word for word and match \n- Search strings are limited to 40 characters. \n\n```bash\n# search in all channels\nyt-fts search \"[search query]\" \n\n# search in channel \nyt-fts search \"[search query]\" --channel \"[channel name or id]\" \n\n# search in specific video\nyt-fts search \"[search query]\" --video \"[video id]\"\n\n# limit results \nyt-fts search \"[search query]\" --limit \"[number of results]\" --channel \"[channel name or id]\"\n\n# export results to csv\nyt-fts search \"[search query]\" --export --channel \"[channel name or id]\" \n```\n\nAdvanced Search Syntax:\n\nThe search string supports sqlite [Enhanced Query Syntax](https://www.sqlite.org/fts3.html#full_text_index_queries).\nwhich includes things like [prefix queries](https://www.sqlite.org/fts3.html#termprefix) which you can use to match parts of a word. \n\n```bash\n# AND search\nyt-fts search \"knife AND Malibu\" --channel \"The Tim Dillon Show\" \n\n# OR SEARCH \nyt-fts search \"knife OR Malibu\" --channel \"The Tim Dillon Show\" \n\n# wild cards\nyt-fts search \"rea* kni* Mali*\" --channel \"The Tim Dillon Show\" \n```\n\n\n# Semantic Search \nYou can enable semantic search for a channel by using the `get-embeddings` command.\nThis requires an OpenAI API key set in the environment variable `OPENAI_API_KEY`, or \nyou can pass the key with the `--openai-api-key` flag. \n\n\n## `get-embedings`\nFetches OpenAI embeddings for specified channel\n```bash\n\n# make sure openAI key is set\n# export OPENAI_API_KEY=\"[yourOpenAIKey]\"\n\nyt-fts get-embeddings --channel \"3Blue1Brown\"\n```\n\nAfter the embeddings are saved you will see a `(ss)` next to the channel name when you \nlist channels and you will be able to use the `vsearch` command for that channel. \n\n## `vsearch` (Semantic Search)\n`vsearch` is for \"Vector search\". This requires that you enable semantic \nsearch for a channel with `get-embeddings`. It has the same options as \n`search` but output will be sorted by similarity to the search string and \nthe default return limit is 10. \n\n```bash\n# search by channel name\nyt-fts vsearch \"[search query]\" --channel \"[channel name or id]\"\n\n# search in specific video\nyt-fts vsearch \"[search query]\" --video \"[video id]\"\n\n# limit results \nyt-fts vsearch \"[search query]\" --limit \"[number of results]\" --channel \"[channel name or id]\"\n\n# export results to csv\nyt-fts vsearch \"[search query]\" --export --channel \"[channel name or id]\" \n\n```\n\n\n\n## How To\n\n**Export search results:**\nFor both the `search` and `vsearch` commands you can export the results to a csv file with \nthe `--export` flag. and it will save the results to a csv file in the current directory. \n```bash\nyt-fts search \"life in the big city\" --export\nyt-fts vsearch \"existing in large metropolaten center\" --export\n```\n\n**Delete a channel:**\nYou can delete a channel with the `delete` command. \n\n```bash\nyt-fts delete --channel \"3Blue1Brown\"\n```\n\n\n**Update a channel:**\nThe update command currently only works for full text search and will not update the \nsemantic search embeddings. \n\n```bash\nyt-fts update --channel \"3Blue1Brown\"\n```\n\n\n**Export all of a channel's transcript:**\nThis command will create a directory in current working directory with the youtube \nchannel id of the specified channel.\n```bash\n# Export to vtt\nyt-fts export --channel \"[id/name]\" --format \"[vtt/txt]\"\n```\n",
"bugtrack_url": null,
"license": "This is free and unencumbered software released into the public domain. Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means. In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. For more information, please refer to <https://unlicense.org> ",
"summary": "Search all of a YouTube channel from the command line",
"version": "0.1.49",
"project_urls": null,
"split_keywords": [
"youtube",
" subtitles",
" search"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "42d3681437c86e307a6f663a0dcb2715f30f93fcd848a42022163447c98ee346",
"md5": "630d9005b905917065fcf1202af55068",
"sha256": "fef4a9827ca3a47eec9199ff84f5e41b3fae3172efb672224243beafee7d9ff2"
},
"downloads": -1,
"filename": "experimental_yt_fts-0.1.49-py3-none-any.whl",
"has_sig": false,
"md5_digest": "630d9005b905917065fcf1202af55068",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 23082,
"upload_time": "2024-04-05T23:37:04",
"upload_time_iso_8601": "2024-04-05T23:37:04.225737Z",
"url": "https://files.pythonhosted.org/packages/42/d3/681437c86e307a6f663a0dcb2715f30f93fcd848a42022163447c98ee346/experimental_yt_fts-0.1.49-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "22272ea88a17fd5701a6e07218b994c8982f8651bbe38718f229f9a11305a1f3",
"md5": "2480e4e2cd2d526077c2fcafa7c30f21",
"sha256": "ab611ef935f49a8e2610e3a4a1467f2124ef4a73660f900f0c733a6543c62a85"
},
"downloads": -1,
"filename": "experimental-yt-fts-0.1.49.tar.gz",
"has_sig": false,
"md5_digest": "2480e4e2cd2d526077c2fcafa7c30f21",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 18692,
"upload_time": "2024-04-05T23:37:06",
"upload_time_iso_8601": "2024-04-05T23:37:06.117039Z",
"url": "https://files.pythonhosted.org/packages/22/27/2ea88a17fd5701a6e07218b994c8982f8651bbe38718f229f9a11305a1f3/experimental-yt-fts-0.1.49.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-05 23:37:06",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "experimental-yt-fts"
}
