| Name | linovelib2epub JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | |
| Summary | Craw light novel from [哔哩轻小说(linovelib)](https://w.linovelib.com/) and convert to epub. |
| upload_time | 2023-07-22 10:38:13 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | |
| keywords |
ebook
epub
library
light novel
哔哩轻小说
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# linovelib2epub
Crawl light novel from [哔哩轻小说(linovelib)](https://w.linovelib.com/) and convert to epub.
[](https://github.com/pypa/hatch)
[](https://github.com/PyCQA/flake8)
[](https://github.com/lightnovel-center/linovelib2epub/actions/workflows/build-and-publish.yml)
[](https://hitsofcode.com/github/lightnovel-center/linovelib2epub/view?branch=main)
## preview
> A picture is worth a thousand words. Talk is cheap, show me the real effect.
> This demo use [this screen recorder tool](https://github.com/faressoft/terminalizer) to record.
## Features
- [x] flexible `has_illustration` and `divide_volume` option for epub output
- [x] support download a certain volume of a novel
- [x] built-in http request retry mechanism to improve network fault tolerance
- [x] built-in random browser user_agent through fake_useragent library
- [x] built-in strict integrity check about image download
- [x] built-in mechanism for saving temporary book data by pickle library
- [x] use multi-process to download images
- [x] support add custom css style to epub
## Supported Websites (plan)
| 序号 | 网站名称 | 语言 | 爬虫难度 | 支持进度 | 备注 |
| ---- | -------------------- | -------- | -------------------- | ---- | ---- |
| 1 | [哔哩轻小说（Mobile）](https://w.linovelib.com/) | 简/繁 | 中😰 | :ok: | 默认选项。 |
| 2 | [哔哩轻小说（Web）](https://www.linovelib.com/) | 简/繁 | 中😰 | 🚫 | 资源同Mobile，没必要。 |
| 3 | [轻之国度](https://www.lightnovel.us/) | 简/繁 | 高🤣👿 | 🚫 | 需要登录，轻币门槛，导航分类混乱。 |
| 4 | [无限轻小说](https://www.8novel.com/) | 繁 | 中😰 | ？ | 不用登录。一章多页。 |
| 5 | [轻小说文库](https://www.wenku8.net/) | 简/繁 | 中😰 | ？ | 需要登录。一章一页。 |
| 6 | [轻小说百科](https://lnovel.org/ ) | 简/繁 | 低😆 | ？ | 不用登录，一章一页。遗憾的是插图清晰度低。 |
| 7 | [真白萌](https://masiro.me/admin/novels ) | 简/繁 | 中😰 | ？ | 需要登录，一章一页。 |
> 爬虫友好度有两个重要指标：
> - 1.访问门槛。是否需要登陆以及积分。
> - 2.页面结构。一个章节多页渲染的视为中等难度。
如果你发现其他的很好轻小说目标源，资源丰富，更新及时，插图清晰，并且爬虫门槛合理的，可以在issue发起补充。
代码实现中对其他轻小说源的支持，关键是继承并重写这个 `BaseNovelWebsiteSpider` 类。
- 其他参考：https://github.com/ilusrdbb/lightnovel-pydownloader
## Usage
### install from source
1. clone this repo
```bash
git clone https://github.com/lightnovel-center/linovelib2epub.git
```
2. set up a clean local python venv
> See also: [creating-virtual-environments](https://docs.python.org/3/library/venv.html#creating-virtual-environments)
replace `py` with your real python command if needed. e.g. `python` or `python3`.
```bash
# new a venv
py -m venv venv
# activate venv
.\venv\Scripts\activate
# install dependencies
py -m pip install -r requirements.txt
# install this package in local
# under project root folder: linovelib2epub/
python -m pip install -e .
```
3. Now you can use this package as a pypi remote package.
```python
from linovelib2epub.linovel import Linovelib2Epub
# warning!: must run within __main__ module guard due to process spawn issue.
if __name__ == '__main__':
linovelib_epub = Linovelib2Epub(book_id=3279)
linovelib_epub.run()
```
### install from pypi
1. Install this package from pypi:
```
pip install linovelib2epub
```
Or update to the latest version:
```
pip install linovelib2epub --upgrade
```
2. create a python file and edit the content as follows:
```python
from linovelib2epub.linovel import Linovelib2Epub
# warning!: must run within __main__ module guard due to process spawn issue.
if __name__ == '__main__':
linovelib_epub = Linovelib2Epub(book_id=3279)
linovelib_epub.run()
```
If it finished without errors, you can see the epub file is under the folder where your python file is located.
## Options
| Parameters | type | required | default | description |
| --------------------- | ------- | -------- | ------------------------------- | ----------------------------------------------------------- |
| book_id | number | YES | None | 书籍ID。 |
| base_url | string | NO | 'https://w.linovelib.com/novel' | 哔哩轻小说主页URL |
| divide_volume | boolean | NO | False | 是否分卷 |
| select_volume_mode | boolean | NO | False | 选择卷模式，它为True时 divide_volume 强制为True。 |
| has_illustration | boolean | NO | True | 是否下载插图 |
| image_download_folder | string | NO | "images" | 图片下载临时文件夹. 不允许以相对路径../开头。 |
| pickle_temp_folder | string | NO | "pickle" | pickle临时数据保存的文件夹。 |
| http_timeout | number | NO | 10 | 一个HTTP请求的超时等待时间(秒)。代表connect和read timeout。 |
| http_retries | number | NO | 5 | 当一个HTTP请求失败后，重试的最大次数。 |
| http_cookie | string | NO | '' | 自定义HTTP cookie。 |
| custom_style_cover | string | NO | '' | 自定义cover.xhtml的样式 |
| custom_style_nav | string | NO | '' | 自定义nav.xhtml的样式 |
| custom_style_chapter | string | NO | '' | 自定义每章(?.xhtml)的样式 |
|disable_proxy |boolean|NO| True| 是否禁用所在的代理环境，默认禁用|
## Todo
- [ ] quality: setup pytest and codecov
- [ ] quality: setup more formatter and linter for maintainability
## Contributors
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
[](#contributors-)
<!-- ALL-CONTRIBUTORS-BADGE:END -->
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
<tbody>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/GOUKOU007"><img src="https://avatars.githubusercontent.com/u/40916324?v=4?s=60" width="60px;" alt="GokouRuri"/><br /><sub><b>GokouRuri</b></sub></a><br /><a href="https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3AGOUKOU007" title="Bug reports">🐛</a> <a href="https://github.com/lightnovel-center/linovelib2epub/commits?author=GOUKOU007" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/xxxfhy"><img src="https://avatars.githubusercontent.com/u/40598925?v=4?s=60" width="60px;" alt="xxxfhy"/><br /><sub><b>xxxfhy</b></sub></a><br /><a href="https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3Axxxfhy" title="Bug reports">🐛</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://foxlesbiao.github.io/"><img src="https://avatars.githubusercontent.com/u/41581909?v=4?s=60" width="60px;" alt="lesfox"/><br /><sub><b>lesfox</b></sub></a><br /><a href="https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3Afoxlesbiao" title="Bug reports">🐛</a></td>
<td align="center" valign="top" width="14.28%"><a href="http://dongliteahouse.wordpress.com"><img src="https://avatars.githubusercontent.com/u/56831381?v=4?s=60" width="60px;" alt="Holence"/><br /><sub><b>Holence</b></sub></a><br /><a href="https://github.com/lightnovel-center/linovelib2epub/commits?author=Holence" title="Code">💻</a></td>
</tr>
</tbody>
</table>
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
## Acknowledgements
- [biliNovel2Epub](https://github.com/fangxx3863/biliNovel2Epub)
Raw data
{
"_id": null,
"home_page": "",
"name": "linovelib2epub",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "ebook,epub,library,light novel,\u54d4\u54e9\u8f7b\u5c0f\u8bf4",
"author": "",
"author_email": "wdpm <1137299673@qq.com>",
"download_url": "",
"platform": null,
"description": "# linovelib2epub\n\nCrawl light novel from [\u54d4\u54e9\u8f7b\u5c0f\u8bf4(linovelib)](https://w.linovelib.com/) and convert to epub.\n\n[](https://github.com/pypa/hatch)\n[](https://github.com/PyCQA/flake8)\n[](https://github.com/lightnovel-center/linovelib2epub/actions/workflows/build-and-publish.yml)\n\n\n\n[](https://hitsofcode.com/github/lightnovel-center/linovelib2epub/view?branch=main)\n\n\n## preview\n> A picture is worth a thousand words. Talk is cheap, show me the real effect.\n\n\n\n> This demo use [this screen recorder tool](https://github.com/faressoft/terminalizer) to record.\n\n\n## Features\n\n- [x] flexible `has_illustration` and `divide_volume` option for epub output\n- [x] support download a certain volume of a novel\n- [x] built-in http request retry mechanism to improve network fault tolerance\n- [x] built-in random browser user_agent through fake_useragent library\n- [x] built-in strict integrity check about image download\n- [x] built-in mechanism for saving temporary book data by pickle library\n- [x] use multi-process to download images\n- [x] support add custom css style to epub\n\n## Supported Websites (plan)\n\n| \u5e8f\u53f7 | \u7f51\u7ad9\u540d\u79f0 | \u8bed\u8a00 | \u722c\u866b\u96be\u5ea6 | \u652f\u6301\u8fdb\u5ea6 | \u5907\u6ce8 |\n| ---- | -------------------- | -------- | -------------------- | ---- | ---- |\n| 1 | [\u54d4\u54e9\u8f7b\u5c0f\u8bf4\uff08Mobile\uff09](https://w.linovelib.com/) | \u7b80/\u7e41 | \u4e2d\ud83d\ude30 | :ok: | \u9ed8\u8ba4\u9009\u9879\u3002 |\n| 2 | [\u54d4\u54e9\u8f7b\u5c0f\u8bf4\uff08Web\uff09](https://www.linovelib.com/) | \u7b80/\u7e41 | \u4e2d\ud83d\ude30 | \ud83d\udeab | \u8d44\u6e90\u540cMobile\uff0c\u6ca1\u5fc5\u8981\u3002 |\n| 3 | [\u8f7b\u4e4b\u56fd\u5ea6](https://www.lightnovel.us/) | \u7b80/\u7e41 | \u9ad8\ud83e\udd23\ud83d\udc7f | \ud83d\udeab | \u9700\u8981\u767b\u5f55\uff0c\u8f7b\u5e01\u95e8\u69db\uff0c\u5bfc\u822a\u5206\u7c7b\u6df7\u4e71\u3002 |\n| 4 | [\u65e0\u9650\u8f7b\u5c0f\u8bf4](https://www.8novel.com/) | \u7e41 | \u4e2d\ud83d\ude30 | \uff1f | \u4e0d\u7528\u767b\u5f55\u3002\u4e00\u7ae0\u591a\u9875\u3002 |\n| 5 | [\u8f7b\u5c0f\u8bf4\u6587\u5e93](https://www.wenku8.net/) | \u7b80/\u7e41 | \u4e2d\ud83d\ude30 | \uff1f | \u9700\u8981\u767b\u5f55\u3002\u4e00\u7ae0\u4e00\u9875\u3002 |\n| 6 | [\u8f7b\u5c0f\u8bf4\u767e\u79d1](https://lnovel.org/ ) | \u7b80/\u7e41 | \u4f4e\ud83d\ude06 | \uff1f | \u4e0d\u7528\u767b\u5f55\uff0c\u4e00\u7ae0\u4e00\u9875\u3002\u9057\u61be\u7684\u662f\u63d2\u56fe\u6e05\u6670\u5ea6\u4f4e\u3002 |\n| 7 | [\u771f\u767d\u840c](https://masiro.me/admin/novels ) | \u7b80/\u7e41 | \u4e2d\ud83d\ude30 | \uff1f | \u9700\u8981\u767b\u5f55\uff0c\u4e00\u7ae0\u4e00\u9875\u3002 |\n\n> \u722c\u866b\u53cb\u597d\u5ea6\u6709\u4e24\u4e2a\u91cd\u8981\u6307\u6807\uff1a\n> - 1.\u8bbf\u95ee\u95e8\u69db\u3002\u662f\u5426\u9700\u8981\u767b\u9646\u4ee5\u53ca\u79ef\u5206\u3002\n> - 2.\u9875\u9762\u7ed3\u6784\u3002\u4e00\u4e2a\u7ae0\u8282\u591a\u9875\u6e32\u67d3\u7684\u89c6\u4e3a\u4e2d\u7b49\u96be\u5ea6\u3002\n\n\u5982\u679c\u4f60\u53d1\u73b0\u5176\u4ed6\u7684\u5f88\u597d\u8f7b\u5c0f\u8bf4\u76ee\u6807\u6e90\uff0c\u8d44\u6e90\u4e30\u5bcc\uff0c\u66f4\u65b0\u53ca\u65f6\uff0c\u63d2\u56fe\u6e05\u6670\uff0c\u5e76\u4e14\u722c\u866b\u95e8\u69db\u5408\u7406\u7684\uff0c\u53ef\u4ee5\u5728issue\u53d1\u8d77\u8865\u5145\u3002\n\n\u4ee3\u7801\u5b9e\u73b0\u4e2d\u5bf9\u5176\u4ed6\u8f7b\u5c0f\u8bf4\u6e90\u7684\u652f\u6301\uff0c\u5173\u952e\u662f\u7ee7\u627f\u5e76\u91cd\u5199\u8fd9\u4e2a `BaseNovelWebsiteSpider` \u7c7b\u3002\n- \u5176\u4ed6\u53c2\u8003\uff1ahttps://github.com/ilusrdbb/lightnovel-pydownloader\n\n## Usage\n\n### install from source\n1. clone this repo\n```bash\ngit clone https://github.com/lightnovel-center/linovelib2epub.git\n```\n2. set up a clean local python venv\n> See also: [creating-virtual-environments](https://docs.python.org/3/library/venv.html#creating-virtual-environments)\n\nreplace `py` with your real python command if needed. e.g. `python` or `python3`.\n\n```bash\n# new a venv\npy -m venv venv\n\n# activate venv\n.\\venv\\Scripts\\activate\n\n# install dependencies\npy -m pip install -r requirements.txt\n\n# install this package in local\n# under project root folder: linovelib2epub/\npython -m pip install -e .\n```\n\n3. Now you can use this package as a pypi remote package.\n```python\nfrom linovelib2epub.linovel import Linovelib2Epub\n\n# warning!: must run within __main__ module guard due to process spawn issue.\nif __name__ == '__main__':\n linovelib_epub = Linovelib2Epub(book_id=3279)\n linovelib_epub.run()\n```\n\n### install from pypi\n1. Install this package from pypi:\n```\npip install linovelib2epub\n```\nOr update to the latest version:\n```\npip install linovelib2epub --upgrade\n```\n2. create a python file and edit the content as follows:\n```python\nfrom linovelib2epub.linovel import Linovelib2Epub\n\n# warning!: must run within __main__ module guard due to process spawn issue.\nif __name__ == '__main__':\n linovelib_epub = Linovelib2Epub(book_id=3279)\n linovelib_epub.run()\n```\nIf it finished without errors, you can see the epub file is under the folder where your python file is located.\n\n\n\n## Options\n\n| Parameters | type | required | default | description |\n| --------------------- | ------- | -------- | ------------------------------- | ----------------------------------------------------------- |\n| book_id | number | YES | None | \u4e66\u7c4dID\u3002 |\n| base_url | string | NO | 'https://w.linovelib.com/novel' | \u54d4\u54e9\u8f7b\u5c0f\u8bf4\u4e3b\u9875URL |\n| divide_volume | boolean | NO | False | \u662f\u5426\u5206\u5377 |\n| select_volume_mode | boolean | NO | False | \u9009\u62e9\u5377\u6a21\u5f0f\uff0c\u5b83\u4e3aTrue\u65f6 divide_volume \u5f3a\u5236\u4e3aTrue\u3002 |\n| has_illustration | boolean | NO | True | \u662f\u5426\u4e0b\u8f7d\u63d2\u56fe |\n| image_download_folder | string | NO | \"images\" | \u56fe\u7247\u4e0b\u8f7d\u4e34\u65f6\u6587\u4ef6\u5939. \u4e0d\u5141\u8bb8\u4ee5\u76f8\u5bf9\u8def\u5f84../\u5f00\u5934\u3002 |\n| pickle_temp_folder | string | NO | \"pickle\" | pickle\u4e34\u65f6\u6570\u636e\u4fdd\u5b58\u7684\u6587\u4ef6\u5939\u3002 |\n| http_timeout | number | NO | 10 | \u4e00\u4e2aHTTP\u8bf7\u6c42\u7684\u8d85\u65f6\u7b49\u5f85\u65f6\u95f4(\u79d2)\u3002\u4ee3\u8868connect\u548cread timeout\u3002 |\n| http_retries | number | NO | 5 | \u5f53\u4e00\u4e2aHTTP\u8bf7\u6c42\u5931\u8d25\u540e\uff0c\u91cd\u8bd5\u7684\u6700\u5927\u6b21\u6570\u3002 |\n| http_cookie | string | NO | '' | \u81ea\u5b9a\u4e49HTTP cookie\u3002 |\n| custom_style_cover | string | NO | '' | \u81ea\u5b9a\u4e49cover.xhtml\u7684\u6837\u5f0f |\n| custom_style_nav | string | NO | '' | \u81ea\u5b9a\u4e49nav.xhtml\u7684\u6837\u5f0f |\n| custom_style_chapter | string | NO | '' | \u81ea\u5b9a\u4e49\u6bcf\u7ae0(?.xhtml)\u7684\u6837\u5f0f |\n|disable_proxy |boolean|NO| True| \u662f\u5426\u7981\u7528\u6240\u5728\u7684\u4ee3\u7406\u73af\u5883\uff0c\u9ed8\u8ba4\u7981\u7528|\n\n\n## Todo\n\n- [ ] quality: setup pytest and codecov\n- [ ] quality: setup more formatter and linter for maintainability\n\n## Contributors\n<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->\n[](#contributors-)\n<!-- ALL-CONTRIBUTORS-BADGE:END -->\n\n<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->\n<!-- prettier-ignore-start -->\n<!-- markdownlint-disable -->\n<table>\n <tbody>\n <tr>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/GOUKOU007\"><img src=\"https://avatars.githubusercontent.com/u/40916324?v=4?s=60\" width=\"60px;\" alt=\"GokouRuri\"/><br /><sub><b>GokouRuri</b></sub></a><br /><a href=\"https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3AGOUKOU007\" title=\"Bug reports\">\ud83d\udc1b</a> <a href=\"https://github.com/lightnovel-center/linovelib2epub/commits?author=GOUKOU007\" title=\"Code\">\ud83d\udcbb</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/xxxfhy\"><img src=\"https://avatars.githubusercontent.com/u/40598925?v=4?s=60\" width=\"60px;\" alt=\"xxxfhy\"/><br /><sub><b>xxxfhy</b></sub></a><br /><a href=\"https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3Axxxfhy\" title=\"Bug reports\">\ud83d\udc1b</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://foxlesbiao.github.io/\"><img src=\"https://avatars.githubusercontent.com/u/41581909?v=4?s=60\" width=\"60px;\" alt=\"lesfox\"/><br /><sub><b>lesfox</b></sub></a><br /><a href=\"https://github.com/lightnovel-center/linovelib2epub/issues?q=author%3Afoxlesbiao\" title=\"Bug reports\">\ud83d\udc1b</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"http://dongliteahouse.wordpress.com\"><img src=\"https://avatars.githubusercontent.com/u/56831381?v=4?s=60\" width=\"60px;\" alt=\"Holence\"/><br /><sub><b>Holence</b></sub></a><br /><a href=\"https://github.com/lightnovel-center/linovelib2epub/commits?author=Holence\" title=\"Code\">\ud83d\udcbb</a></td>\n </tr>\n </tbody>\n</table>\n\n<!-- markdownlint-restore -->\n<!-- prettier-ignore-end -->\n\n<!-- ALL-CONTRIBUTORS-LIST:END -->\n<!-- prettier-ignore-start -->\n<!-- markdownlint-disable -->\n\n<!-- markdownlint-restore -->\n<!-- prettier-ignore-end -->\n\n<!-- ALL-CONTRIBUTORS-LIST:END -->\n\n## Acknowledgements\n\n- [biliNovel2Epub](https://github.com/fangxx3863/biliNovel2Epub)\n",
"bugtrack_url": null,
"license": "",
"summary": "Craw light novel from [\u54d4\u54e9\u8f7b\u5c0f\u8bf4(linovelib)](https://w.linovelib.com/) and convert to epub.",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/wdpm/linovelib2epub",
"Source": "https://github.com/wdpm/linovelib2epub",
"Tracker": "https://github.com/wdpm/linovelib2epub/issues"
},
"split_keywords": [
"ebook",
"epub",
"library",
"light novel",
"\u54d4\u54e9\u8f7b\u5c0f\u8bf4"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "aa483b943df0ec22b1280a1c7b39aa893c4ab5abf0683407b59f77682b0fc5e1",
"md5": "ed80060ddc5a31d0993f3bd214be5cf1",
"sha256": "a269d006f986e2b9836ad0f0eae4610ca7c5b02c8f6eed902419368f98560645"
},
"downloads": -1,
"filename": "linovelib2epub-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ed80060ddc5a31d0993f3bd214be5cf1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 37728,
"upload_time": "2023-07-22T10:38:13",
"upload_time_iso_8601": "2023-07-22T10:38:13.445062Z",
"url": "https://files.pythonhosted.org/packages/aa/48/3b943df0ec22b1280a1c7b39aa893c4ab5abf0683407b59f77682b0fc5e1/linovelib2epub-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-22 10:38:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "wdpm",
"github_project": "linovelib2epub",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "linovelib2epub"
}
