# Devdocs scraper
This scraper downloads [devdocs.io](https://devdocs.io/) documentation databases and puts them in ZIM files,
a clean and user friendly format for storing content for offline usage.
[](https://www.codefactor.io/repository/github/openzim/devdocs)
[](https://www.gnu.org/licenses/gpl-3.0)
[](https://codecov.io/gh/openzim/devdocs)
[](https://pypi.org/project/devdocs2zim/)
[](https://pypi.org/project/devdocs2zim)
## Installation
There are three main ways to install and use `devdocs2zim` from most recommended to least:
<details>
<summary>Install using a pre-built container</summary>
1. Download the image using `docker`:
```sh
docker pull ghcr.io/openzim/devdocs
```
</details>
<details>
<summary>Build your own container</summary>
1. Clone the repository locally:
```sh
git clone https://github.com/openzim/devdocs.git && cd devdocs
```
1. Build the image:
```sh
docker build -t ghcr.io/openzim/devdocs .
```
</details>
<details>
<summary>Run the software locally using Hatch</summary>
1. Clone the repository locally:
```sh
git clone https://github.com/openzim/devdocs.git && cd devdocs
```
1. Install [Hatch](https://hatch.pypa.io/):
```sh
pip3 install hatch
```
1. Start a hatch shell to install software and dependencies in an isolated virtual environment.
```sh
hatch shell
```
1. Run the `devdocs2zim` command:
```sh
devdocs2zim --help
```
</details>
## Usage
> [!WARNING]
> This project is still a work in progress and isn't ready for use yet, the commands below are examples only.
```sh
# Usage
docker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim [--all|--slug=SLUG|--first=N]
# Fetch all documents
docker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all
# Fetch all documents except Ansible
docker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all --skip-slug-regex "^ansible.*"
# Fetch Vue related documents
docker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --slug vue~3 --slug vue_router~4
# Fetch the docs for the two most recent versions of each software
docker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --first=2
```
**One of the following flags is required:**
* `--all`: Fetch all Devdocs resources, and produce one ZIM per resource.
* `--slug SLUG`: Fetch the provided Devdocs resource. Slugs are the first path entry in the Devdocs URL.
For example, the slug for: `https://devdocs.io/gcc~12/` is `gcc~12`. Use --slug several times to add multiple.
* `--first N`: Fetch the first number of items per slug as shown in the DevDocs UI.
**Optional Flags:**
* `--skip-slug-regex REGEX`: Skips slugs matching the given regular expression.
* `--output OUTPUT_FOLDER`: Output folder for ZIMs. Default: /output
* `--creator CREATOR`: Name of content creator. Default: 'DevDocs'
* `--publisher PUBLISHER`: Custom publisher name. Default: 'openZIM'
* `--name-format FORMAT`: Custom name format for individual ZIMs.
Default: 'devdocs_{slug_without_version}_{version}'
* `--title-format FORMAT`: Custom title format for individual ZIMs.
Value will be truncated to 30 chars. Default: '{full_name} Documentation'
* `--description-format FORMAT`: Custom description format for individual ZIMs.
Value will be truncated to 80 chars. Default: '{full_name} Documentation'
* `--long-description-format FORMAT`: Custom long description format for your ZIM.
Value will be truncated to 4000 chars.Default: '{full_name} documentation by DevDocs'
* `--tag TAG`: Add tag to the ZIM. Use --tag several times to add multiple.
Formatting is supported. Default: ['devdocs', '{slug_without_version}']
**Formatting Placeholders**
The following formatting placeholders are supported:
* `{name}`: Human readable name of the resource e.g. `Python`.
* `{full_name}`: Name with optional version for the resource e.g. `Python 3.12`.
* `{slug}`: Devdocs slug for the resource e.g. `python~3.12`.
* `{clean_slug}`: Slug with non alphanumeric/period characters replaced with `-` e.g. `python-3.12`.
* `{slug_without_version}`: Devdocs slug for the resource without the version e.g. `python`.
* `{version}`: Shortened version displayed in devdocs, if any e.g. `3.12`.
* `{release}`: Specific release of the software the documentation is for, if any e.g. `3.12.1`.
* `{attribution}`: License and attribution information about the resource.
* `{home_link}`: Link to the project's home page, if any: e.g. `https://python.org`.
* `{code_link}`: Link to the project's source, if any: e.g. `https://github.com/python/cpython`.
* `{period}`: The current date in `YYYY-MM` format e.g. `2024-02`.
## Developing
Use the commands below to set up the project once:
```sh
# Install hatch if it isn't installed already.
❯ pip install hatch
# Local install (in default env) / re-sync packages
❯ hatch run pip list
# Set-up pre-commit
❯ pre-commit install
```
The following commands can be used to build and test the scraper:
```sh
# Show scripts
❯ hatch env show
# linting, testing, coverage, checking
❯ hatch run lint:all
❯ hatch run lint:fixall
# run tests on all matrixed' envs
❯ hatch run test:run
# run tests in a single matrixed' env
❯ hatch env run -e test -i py=3.12 coverage
# run static type checks
❯ hatch env run check:all
# building packages
❯ hatch build
```
### Contributing
This project adheres to openZIM's [Contribution Guidelines](https://github.com/openzim/overview/wiki/Contributing).
This project has implemented openZIM's [Python bootstrap, conventions and policies](https://github.com/openzim/_python-bootstrap/docs/Policy.md) **v1.0.3**.
Raw data
{
"_id": null,
"home_page": null,
"name": "devdocs2zim",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.12",
"maintainer_email": null,
"keywords": "devdocs, offline, openzim, zim",
"author": null,
"author_email": "openZIM <dev@openzim.org>",
"download_url": "https://files.pythonhosted.org/packages/6c/58/b519663b69962c7fe7d65c44fd85b3b3923a249806689fa979689434b9ec/devdocs2zim-0.1.0.tar.gz",
"platform": null,
"description": "# Devdocs scraper\n\nThis scraper downloads [devdocs.io](https://devdocs.io/) documentation databases and puts them in ZIM files,\na clean and user friendly format for storing content for offline usage.\n\n[](https://www.codefactor.io/repository/github/openzim/devdocs)\n[](https://www.gnu.org/licenses/gpl-3.0)\n[](https://codecov.io/gh/openzim/devdocs)\n[](https://pypi.org/project/devdocs2zim/)\n[](https://pypi.org/project/devdocs2zim)\n\n\n## Installation\n\nThere are three main ways to install and use `devdocs2zim` from most recommended to least:\n\n<details>\n<summary>Install using a pre-built container</summary>\n\n\n1. Download the image using `docker`:\n\n ```sh\n docker pull ghcr.io/openzim/devdocs\n ```\n\n</details>\n<details>\n<summary>Build your own container</summary>\n\n1. Clone the repository locally:\n\n ```sh\n git clone https://github.com/openzim/devdocs.git && cd devdocs\n ```\n\n1. Build the image:\n\n ```sh\n docker build -t ghcr.io/openzim/devdocs .\n ```\n\n</details>\n<details>\n<summary>Run the software locally using Hatch</summary>\n\n1. Clone the repository locally:\n\n ```sh\n git clone https://github.com/openzim/devdocs.git && cd devdocs\n ```\n\n1. Install [Hatch](https://hatch.pypa.io/):\n\n ```sh\n pip3 install hatch\n ```\n\n1. Start a hatch shell to install software and dependencies in an isolated virtual environment.\n\n ```sh\n hatch shell\n ```\n\n1. Run the `devdocs2zim` command:\n\n ```sh\n devdocs2zim --help\n ```\n\n</details>\n\n## Usage\n\n> [!WARNING]\n> This project is still a work in progress and isn't ready for use yet, the commands below are examples only.\n\n\n```sh\n# Usage\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim [--all|--slug=SLUG|--first=N]\n\n# Fetch all documents\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all\n\n# Fetch all documents except Ansible\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --all --skip-slug-regex \"^ansible.*\"\n\n# Fetch Vue related documents\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --slug vue~3 --slug vue_router~4\n\n# Fetch the docs for the two most recent versions of each software\ndocker run -v my_dir:/output ghcr.io/openzim/devdocs devdocs2zim --first=2\n```\n\n\n**One of the following flags is required:**\n\n* `--all`: Fetch all Devdocs resources, and produce one ZIM per resource.\n* `--slug SLUG`: Fetch the provided Devdocs resource. Slugs are the first path entry in the Devdocs URL.\n For example, the slug for: `https://devdocs.io/gcc~12/` is `gcc~12`. Use --slug several times to add multiple.\n* `--first N`: Fetch the first number of items per slug as shown in the DevDocs UI.\n\n\n**Optional Flags:**\n\n* `--skip-slug-regex REGEX`: Skips slugs matching the given regular expression.\n* `--output OUTPUT_FOLDER`: Output folder for ZIMs. Default: /output\n* `--creator CREATOR`: Name of content creator. Default: 'DevDocs'\n* `--publisher PUBLISHER`: Custom publisher name. Default: 'openZIM'\n* `--name-format FORMAT`: Custom name format for individual ZIMs.\n Default: 'devdocs_{slug_without_version}_{version}'\n* `--title-format FORMAT`: Custom title format for individual ZIMs.\n Value will be truncated to 30 chars. Default: '{full_name} Documentation'\n* `--description-format FORMAT`: Custom description format for individual ZIMs.\n Value will be truncated to 80 chars. Default: '{full_name} Documentation'\n* `--long-description-format FORMAT`: Custom long description format for your ZIM.\n Value will be truncated to 4000 chars.Default: '{full_name} documentation by DevDocs'\n* `--tag TAG`: Add tag to the ZIM. Use --tag several times to add multiple.\n Formatting is supported. Default: ['devdocs', '{slug_without_version}']\n\n**Formatting Placeholders**\n\nThe following formatting placeholders are supported:\n\n* `{name}`: Human readable name of the resource e.g. `Python`.\n* `{full_name}`: Name with optional version for the resource e.g. `Python 3.12`.\n* `{slug}`: Devdocs slug for the resource e.g. `python~3.12`.\n* `{clean_slug}`: Slug with non alphanumeric/period characters replaced with `-` e.g. `python-3.12`.\n* `{slug_without_version}`: Devdocs slug for the resource without the version e.g. `python`.\n* `{version}`: Shortened version displayed in devdocs, if any e.g. `3.12`.\n* `{release}`: Specific release of the software the documentation is for, if any e.g. `3.12.1`.\n* `{attribution}`: License and attribution information about the resource.\n* `{home_link}`: Link to the project's home page, if any: e.g. `https://python.org`.\n* `{code_link}`: Link to the project's source, if any: e.g. `https://github.com/python/cpython`.\n* `{period}`: The current date in `YYYY-MM` format e.g. `2024-02`.\n\n## Developing\n\nUse the commands below to set up the project once:\n\n```sh\n# Install hatch if it isn't installed already.\n\u276f pip install hatch\n\n# Local install (in default env) / re-sync packages\n\u276f hatch run pip list\n\n# Set-up pre-commit\n\u276f pre-commit install\n```\n\nThe following commands can be used to build and test the scraper:\n\n```sh\n# Show scripts\n\u276f hatch env show\n\n# linting, testing, coverage, checking\n\u276f hatch run lint:all\n\u276f hatch run lint:fixall\n\n# run tests on all matrixed' envs\n\u276f hatch run test:run\n\n# run tests in a single matrixed' env\n\u276f hatch env run -e test -i py=3.12 coverage\n\n# run static type checks\n\u276f hatch env run check:all\n\n# building packages\n\u276f hatch build\n```\n\n\n### Contributing\n\nThis project adheres to openZIM's [Contribution Guidelines](https://github.com/openzim/overview/wiki/Contributing).\n\nThis project has implemented openZIM's [Python bootstrap, conventions and policies](https://github.com/openzim/_python-bootstrap/docs/Policy.md) **v1.0.3**.\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Make ZIM files from DevDocs.io",
"version": "0.1.0",
"project_urls": {
"Donate": "https://www.kiwix.org/en/support-us/",
"Homepage": "https://github.com/openzim/devdocs"
},
"split_keywords": [
"devdocs",
" offline",
" openzim",
" zim"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7cf4580372397c7147bf944524065a3a1a5d8f6e42984828f6828617e4d800fa",
"md5": "1a14d9c4f54c9ab559cc9fc6082b6ebd",
"sha256": "581acd057874a27ae8ff548beb88686af1c8268868f1a990353c654dfd079e6b"
},
"downloads": -1,
"filename": "devdocs2zim-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1a14d9c4f54c9ab559cc9fc6082b6ebd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.12",
"size": 36560,
"upload_time": "2024-09-27T09:37:58",
"upload_time_iso_8601": "2024-09-27T09:37:58.002571Z",
"url": "https://files.pythonhosted.org/packages/7c/f4/580372397c7147bf944524065a3a1a5d8f6e42984828f6828617e4d800fa/devdocs2zim-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6c58b519663b69962c7fe7d65c44fd85b3b3923a249806689fa979689434b9ec",
"md5": "61615b39a25d8fe8fa5fdfce51bf9cbd",
"sha256": "c758e86f77bccfa89ec4f4377983a9b76fa8ba233313e35494229f85123997e1"
},
"downloads": -1,
"filename": "devdocs2zim-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "61615b39a25d8fe8fa5fdfce51bf9cbd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.12",
"size": 42078,
"upload_time": "2024-09-27T09:37:59",
"upload_time_iso_8601": "2024-09-27T09:37:59.506748Z",
"url": "https://files.pythonhosted.org/packages/6c/58/b519663b69962c7fe7d65c44fd85b3b3923a249806689fa979689434b9ec/devdocs2zim-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-27 09:37:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "openzim",
"github_project": "devdocs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "devdocs2zim"
}
JSON
