# MkDocs Translate
A translate script is provided to facilitate working with pandoc and deepl translation services.
The user manual is available: https://jodygarnett.github.io/translate/
The user manual is written in sphinx reStructuredText and translated to mkdocs as a regression test:
* https://github.com/jodygarnett/translate/tree/main/example
## Quickstart
1. This script requires ***pandoc*** be installed:
Ubuntu:
```bash
apt-get install pandoc
```
macOS:
``` bash
brew install pandoc
```
References:
* https://pandoc.org/installing.html
2. A writable python environment is required.
If you use homebrew (popular on macOS). This installs into user space so it is a witable environment.
```bash
brew install python
```
You may also use the system python provided by:
* Linux distribution
* Microsoft App Store
* https://www.python.org/ (windows and macOS)
The system python is not used directly, it includes `virtualenv` used to setup a writable Python enviornment:
```bash
virtualenv venv
source venv/bin/activate
```
3. Install mkdocs_translate into your writable Python environment.
To install from pypi (not yet available):
```
pip install -i https://test.pypi.org/simple/ mkdocs-translate==0.4.0
```
To install development version use (to preview and provide feedback):
```
pip install git+https://github.com/jodygarnett/translate.git
```
4. To check it is installed correctly:
```
mkdocs_translate --help
```
5. The script is intended to run from the location of your mkdocs project (with `docs` and `mkdocs.yml` files):
```
cd core-genetwork/docs/manual
```
6. The script makes use of existing ``build`` (or ``target``) folder for scratch files:
```
mkdir build
```
7. Optional: Create a [translate.yml](translate.yml) filling in the conversion parameters for your project.
This file is used to indicate the `build` or `target` directory to use for temporary files.
Additional configuration options are required for advanced sphinx-build `config.py` options like substitutions and external links.
## Setting up mkdocs project
A working [example](example) is provided to be adapted for your project:
1. Create [requirements.txt](example/requirements.txt) with mkdocs plugins required.
2. Create [mkdocs.yml](example/mkdocs.yml).
3. Optional: If your content uses `download` directive to include external content, there is a `mkdocs` hook for processing of `download.txt` files.
Create [download.py](example/download.py).
Register hook with `mkdocs.yml`:
```yaml
# Customizations
hooks:
- download.py
```
4. Use `.gitignore` to ignore the following:
```
build
target
```
5. The resulting directory structure is:
```
doc/
source/
.gitignore
requirements.txt
mkdocs.yml
download.py
```
## Format conversion from sphinx-build rst files
GeoServer is used as an example here, which is a maven project with a convention of `target` for temporary files.
1. Copy everything over (so all the images and so on are present)
```
cd geoserver/doc/en/user
copy -r source doc
```
2. Clean sphinx-build `conf.py` and ``rst`` files from `docs/` folder.
```
find doc -type f -regex ".*\.rst" -delete
rm doc/conf.py
```
4. To scan rst files before conversion:
* `all`: (default)
* `index`: scan anchors and headings into `target/convert/anchors.txt` for `doc` and `ref` directives.
* `download`: scan `download` directives for external content, into `docs` folder, producing `download/download.txt` folders.
* `toc`: scan `toc` directives, producing nav structure for use with `mkdocs.yml` file
```
mkdocs_translate scan
```
5. To bulk convert all content from ``rst`` to ``md``:
```
mkdocs_translate rst source/**/*.rst
```
6. Review this content you may find individual files to fix.
Some formatting is easier to fix in the `rst` files before conversion:
* Indention of nested lists in ``rst`` is often incorrect, resulting in restarted numbering or block quotes.
* Random ``{.title-ref}`` snippets is a general indication to simplify the rst and re-translate.
* Anchors or headings with trailing whitespace throwing off the heading scan, resulting in broken references
6. Convert a single file:
```
mkdocs_translate rst source/introduction/license.rst
```
7. Bulk convert files in a folder:
```
mkdocs_translate rst source/introduction/**/*.rst
```
8. To generate out navigation tree:
```
mkdocs_translate scan toc
```
The output is printed to standard out and may be appended to `mkdocs.yml` file.
### Known limitations
Some things are not supported by pandoc, which will produce ``WARNING:`` messages:
* substitutions used for inline images
* Underlines: replace with bold or italic
```
WARNING: broken reference 'getting_involved' link:getting_involved-broken.rst
```
## Language Translation
Translations are listed alongside english markdown:
* `example.md`
* `example.fr.md`
Using ***pandoc*** to convert to `html`, and then using the [Deepl REST API](http://deepl.com).
4. Provide environmental variable with Deepl authentication key:
```
export DEEPL_AUTH="xxxxxxxx-xxx-...-xxxxx:fx"
```
5. Translate a document to french using pandoc and deepl:
```
mkdocs_translate french docs/help/index.md
```
6. To translate several documents in a folder:
```
mkdocs_translate french docs/overview/*.md
```
Deepl charges by the character so bulk translation not advisable.
See ``mkdocs_translate french --help`` for more options.
You are welcome to use google translate, ChatGPT, or Deepl directly - keeping in mind markdown formatting may be lost.
Please see the writing guide for what mkdocs functionality is supported.
## Local Development
To build and test locally:
1. Clone:
```
git clone https://github.com/jodygarnett/translate.git translate
```
2. Install requirements:
```
cd translate
pip3 install -r mkdocs_translate/requirements.txt
```
2. Install locally:
```
pip3 install -e .
```
Distribution:
1. Update version number in ``mkdocs_translate/__init__.py`` version:
```python
__version__ = 0.4.0
```
2. Build wheel:
```bash
python3 -m build
```
3. Upload wheel:
```bash
python3 -m twine upload --repository pypi dist/*
```
Debugging:
1. Recommend troubleshooting a single file at a time:
```rst
mkdocs_translate rst docs/index.rst
```
2. Compare the temporary files staged for pandoc conversion:
```
bbedit docs/index.rst docs/index.md target/convert/index.tmp.html target/convert/index/tmp.md
```
3. To turn on logging during conversion:
```bash
mkdocs_translate --log=DEBUG translate.yml rst
```
Pandoc:
1. The pandoc plugin settings are in two constants:
```python
md_extensions_to =
'markdown+definition_lists+fenced_divs+backtick_code_blocks+fenced_code_attributes-simple_tables+pipe_tables'
md_extensions_from =
'markdown+definition_lists+fenced_divs+backtick_code_blocks+fenced_code_attributes+pipe_tables'
```
2. The pandoc extensions are chosen to align with mkdocs use of markdown extensions, or with post-processing:
| markdown extension | pandoc extension | post processing |
|----------------------|------------------------|-----------------|
| tables | pipe_tables | |
| pymdownx.keys | | post processing |
| pymdownx.superfences | backtick_code_blocks | post processing |
| admonition | fenced_divs | post processing |
3. To troubleshoot just the markdown to html conversion:
```bash
mkdocs_translate internal_html manual/docs/contributing/style-guide.md
mkdocs_translate internal_markdown target/contributing/style-guide.html
diff manual/docs/contributing/style-guide.md target/contributing/style-guide.md
```
## Configuration
For geoserver or core-geonetwork (or other projects following maven conventions) no configuration is required.
To override configuration on command line add `-concfig <file.yml>` before the command:
```bash
mkdocs_translate --config translate.yml rst
```
The file `mkdocs_translate/config.yml` file contains some settings (defaults are shown below):
* `deepl_base_url`: "https://api-free.deepl.com"
Customize if you are paying customer.
* `project_folder`: "."
Default assumes you are running from the current directory.
* `rst_folder`: "source"
* `docs_folder`: "docs"
* `build_folder`: "target"
The use of "target" follows maven convention, python projects may wish to use "build"
* `docs_folder`: "docs"
mkdocs convention.
* `anchor_file`: 'anchors.txt'
* `upload_folder`: "translate"
Combined with ``build_folder`` to stage html files for translation (example: `build/translate`)
* `convert_folder`: "convert"
Combined with ``build_folder`` for rst conversion temporary files (example: `build/convert`).
Temporary files are required for use by pandoc.
* `download_folder`: "translate"
Combined with ``build_folder`` to retrieve translation results (example: `build/translate`)
Temporary files are required for use by pandoc.
* `substitutions`: dictionary of `|substitutions|` to use when converting config.py rst_epilog common substitutions.
```
project: GeoServer
author: Open Source Geospatial Foundation
copyright: 2023, Open Source Geospatial Foundation
project_copyright: 2023, Open Source Geospatial Foundation
```
* The built-in substitutions for `|version|` and `|release|` are changed to `{{ version }}` and `{{ release }}``
variables for use with `mkdocs-macros-plugin` variable substitution:
Use `mkdocs.yml` to define:
```
extra:
homepage: https://geoserver.org/
version: '2.24'
release: '2.24.2'
```
* `extlinks`: dictionary of config.py extlinks substitutions.
To convert sphinx-build config.py:
```
extlinks = {
'wiki': ('https://github.com/geoserver/geoserver/wiki/%s', None),
'user': ('https://docs.geoserver.org/'+branch+'/en/user/%s', None),
'geos': ('https://osgeo-org.atlassian.net/browse/GEOS-%s','GEOS-%s')
}
```
Use config.yml (note use of mkdocs-macros-plugin for variable substitution:
```
extlinks:
wiki: https://github.com/geoserver/geoserver/wiki/%s
user: https://docs.geoserver.org/{{ branch }}/en/user/%s
geos: https://osgeo-org.atlassian.net/browse/GEOS-%s|GEOS-%s
download_release: https://sourceforge.net/projects/geoserver/files/GeoServer/{{ release }}/geoserver-{{ release }}-%s.zip|geoserver-{{ release }}-%s.zip
```
Raw data
{
"_id": null,
"home_page": null,
"name": "mkdocs-translate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "mkdocs, markdown, pandoc, sphinx-build, sphinx, rst, reStructuredText, translate",
"author": null,
"author_email": "Jody Garnett <jody.garnett@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/04/01/3802d6232742b125b17f57f1e67d60b19fbef68de4f1bda2f6a8164b2f23/mkdocs-translate-0.4.0.tar.gz",
"platform": null,
"description": "# MkDocs Translate\n\nA translate script is provided to facilitate working with pandoc and deepl translation services.\n\nThe user manual is available: https://jodygarnett.github.io/translate/\n\nThe user manual is written in sphinx reStructuredText and translated to mkdocs as a regression test:\n\n* https://github.com/jodygarnett/translate/tree/main/example \n\n## Quickstart\n\n1. This script requires ***pandoc*** be installed:\n\n Ubuntu:\n ```bash\n apt-get install pandoc\n ```\n\n macOS:\n ``` bash\n brew install pandoc\n ```\n\n References:\n\n * https://pandoc.org/installing.html\n\n\n2. A writable python environment is required.\n \n If you use homebrew (popular on macOS). This installs into user space so it is a witable environment.\n \n ```bash\n brew install python\n ```\n \n You may also use the system python provided by:\n \n * Linux distribution\n * Microsoft App Store\n * https://www.python.org/ (windows and macOS)\n \n The system python is not used directly, it includes `virtualenv` used to setup a writable Python enviornment:\n \n ```bash\n virtualenv venv\n source venv/bin/activate\n ```\n\n3. Install mkdocs_translate into your writable Python environment.\n\n To install from pypi (not yet available):\n \n ```\n pip install -i https://test.pypi.org/simple/ mkdocs-translate==0.4.0\n ```\n \n To install development version use (to preview and provide feedback):\n\n ```\n pip install git+https://github.com/jodygarnett/translate.git\n ```\n \n \n4. To check it is installed correctly:\n\n ```\n mkdocs_translate --help\n ```\n\n5. The script is intended to run from the location of your mkdocs project (with `docs` and `mkdocs.yml` files):\n\n ```\n cd core-genetwork/docs/manual\n ```\n\n6. The script makes use of existing ``build`` (or ``target``) folder for scratch files:\n\n ```\n mkdir build\n ```\n\n7. Optional: Create a [translate.yml](translate.yml) filling in the conversion parameters for your project.\n\n This file is used to indicate the `build` or `target` directory to use for temporary files.\n \n Additional configuration options are required for advanced sphinx-build `config.py` options like substitutions and external links.\n\n## Setting up mkdocs project\n\nA working [example](example) is provided to be adapted for your project:\n\n1. Create [requirements.txt](example/requirements.txt) with mkdocs plugins required.\n\n2. Create [mkdocs.yml](example/mkdocs.yml). \n\n3. Optional: If your content uses `download` directive to include external content, there is a `mkdocs` hook for processing of `download.txt` files. \n \n Create [download.py](example/download.py).\n\n Register hook with `mkdocs.yml`:\n \n ```yaml\n # Customizations\n hooks:\n - download.py\n ```\n\n4. Use `.gitignore` to ignore the following:\n \n ```\n build\n target\n ```\n\n5. The resulting directory structure is:\n \n ```\n doc/\n source/\n .gitignore \n requirements.txt\n mkdocs.yml\n download.py\n ```\n\n## Format conversion from sphinx-build rst files\n\nGeoServer is used as an example here, which is a maven project with a convention of `target` for temporary files.\n\n1. Copy everything over (so all the images and so on are present)\n \n ```\n cd geoserver/doc/en/user\n copy -r source doc\n ```\n \n2. Clean sphinx-build `conf.py` and ``rst`` files from `docs/` folder.\n\n ```\n find doc -type f -regex \".*\\.rst\" -delete\n rm doc/conf.py\n ```\n\n4. To scan rst files before conversion:\n\n * `all`: (default)\n * `index`: scan anchors and headings into `target/convert/anchors.txt` for `doc` and `ref` directives.\n * `download`: scan `download` directives for external content, into `docs` folder, producing `download/download.txt` folders.\n * `toc`: scan `toc` directives, producing nav structure for use with `mkdocs.yml` file\n \n ```\n mkdocs_translate scan\n ```\n\n5. To bulk convert all content from ``rst`` to ``md``:\n \n ```\n mkdocs_translate rst source/**/*.rst\n ```\n \n6. Review this content you may find individual files to fix.\n\n Some formatting is easier to fix in the `rst` files before conversion:\n \n * Indention of nested lists in ``rst`` is often incorrect, resulting in restarted numbering or block quotes.\n \n * Random ``{.title-ref}`` snippets is a general indication to simplify the rst and re-translate.\n\n * Anchors or headings with trailing whitespace throwing off the heading scan, resulting in broken references\n\n6. Convert a single file:\n \n ```\n mkdocs_translate rst source/introduction/license.rst\n ```\n\n7. Bulk convert files in a folder:\n \n ```\n mkdocs_translate rst source/introduction/**/*.rst\n ```\n\n8. To generate out navigation tree:\n \n ```\n mkdocs_translate scan toc\n ```\n \n The output is printed to standard out and may be appended to `mkdocs.yml` file.\n\n### Known limitations\n\nSome things are not supported by pandoc, which will produce ``WARNING:`` messages:\n\n* substitutions used for inline images\n\n* Underlines: replace with bold or italic\n \n ```\n WARNING: broken reference 'getting_involved' link:getting_involved-broken.rst\n ```\n \n\n## Language Translation\n\nTranslations are listed alongside english markdown:\n\n* `example.md`\n* `example.fr.md`\n\nUsing ***pandoc*** to convert to `html`, and then using the [Deepl REST API](http://deepl.com).\n\n4. Provide environmental variable with Deepl authentication key:\n\n ```\n export DEEPL_AUTH=\"xxxxxxxx-xxx-...-xxxxx:fx\"\n ```\n\n5. Translate a document to french using pandoc and deepl:\n\n ```\n mkdocs_translate french docs/help/index.md\n ```\n \n6. To translate several documents in a folder:\n\n ```\n mkdocs_translate french docs/overview/*.md\n ```\n \n Deepl charges by the character so bulk translation not advisable.\n\nSee ``mkdocs_translate french --help`` for more options.\n\nYou are welcome to use google translate, ChatGPT, or Deepl directly - keeping in mind markdown formatting may be lost.\n\nPlease see the writing guide for what mkdocs functionality is supported.\n\n## Local Development\n\nTo build and test locally:\n\n1. Clone:\n\n ```\n git clone https://github.com/jodygarnett/translate.git translate\n ```\n\n2. Install requirements:\n ```\n cd translate\n pip3 install -r mkdocs_translate/requirements.txt\n ```\n\n2. Install locally:\n ```\n pip3 install -e .\n ```\n\nDistribution:\n\n1. Update version number in ``mkdocs_translate/__init__.py`` version:\n \n ```python\n __version__ = 0.4.0\n ```\n \n2. Build wheel:\n \n ```bash\n python3 -m build \n ```\n\n3. Upload wheel:\n \n ```bash\n python3 -m twine upload --repository pypi dist/* \n ```\n\nDebugging:\n\n1. Recommend troubleshooting a single file at a time:\n\n ```rst\n mkdocs_translate rst docs/index.rst\n ```\n \n2. Compare the temporary files staged for pandoc conversion:\n\n ```\n bbedit docs/index.rst docs/index.md target/convert/index.tmp.html target/convert/index/tmp.md\n ```\n \n3. To turn on logging during conversion:\n\n ```bash\n mkdocs_translate --log=DEBUG translate.yml rst\n ```\n\nPandoc:\n\n1. The pandoc plugin settings are in two constants:\n\n ```python \n md_extensions_to =\n 'markdown+definition_lists+fenced_divs+backtick_code_blocks+fenced_code_attributes-simple_tables+pipe_tables'\n md_extensions_from =\n 'markdown+definition_lists+fenced_divs+backtick_code_blocks+fenced_code_attributes+pipe_tables'\n ```\n\n2. The pandoc extensions are chosen to align with mkdocs use of markdown extensions, or with post-processing:\n\n | markdown extension | pandoc extension | post processing |\n |----------------------|------------------------|-----------------|\n | tables | pipe_tables | |\n | pymdownx.keys | | post processing |\n | pymdownx.superfences | backtick_code_blocks | post processing | \n | admonition | fenced_divs | post processing |\n \n3. To troubleshoot just the markdown to html conversion:\n \n ```bash\n mkdocs_translate internal_html manual/docs/contributing/style-guide.md\n mkdocs_translate internal_markdown target/contributing/style-guide.html\n\n diff manual/docs/contributing/style-guide.md target/contributing/style-guide.md\n ```\n \n## Configuration\n\nFor geoserver or core-geonetwork (or other projects following maven conventions) no configuration is required.\n\nTo override configuration on command line add `-concfig <file.yml>` before the command:\n\n```bash\nmkdocs_translate --config translate.yml rst\n```\n\nThe file `mkdocs_translate/config.yml` file contains some settings (defaults are shown below):\n\n* `deepl_base_url`: \"https://api-free.deepl.com\"\n \n Customize if you are paying customer.\n \n* `project_folder`: \".\"\n \n Default assumes you are running from the current directory.\n\n* `rst_folder`: \"source\"\n\n* `docs_folder`: \"docs\"\n\n* `build_folder`: \"target\"\n \n The use of \"target\" follows maven convention, python projects may wish to use \"build\"\n\n* `docs_folder`: \"docs\"\n \n mkdocs convention.\n \n* `anchor_file`: 'anchors.txt'\n \n* `upload_folder`: \"translate\"\n \n Combined with ``build_folder`` to stage html files for translation (example: `build/translate`)\n \n* `convert_folder`: \"convert\"\n\n Combined with ``build_folder`` for rst conversion temporary files (example: `build/convert`).\n Temporary files are required for use by pandoc.\n \n* `download_folder`: \"translate\"\n \n Combined with ``build_folder`` to retrieve translation results (example: `build/translate`)\n Temporary files are required for use by pandoc.\n\n* `substitutions`: dictionary of `|substitutions|` to use when converting config.py rst_epilog common substitutions.\n \n ``` \n project: GeoServer\n author: Open Source Geospatial Foundation\n copyright: 2023, Open Source Geospatial Foundation\n project_copyright: 2023, Open Source Geospatial Foundation\n ```\n \n* The built-in substitutions for `|version|` and `|release|` are changed to `{{ version }}` and `{{ release }}``\n variables for use with `mkdocs-macros-plugin` variable substitution:\n \n Use `mkdocs.yml` to define:\n ```\n extra:\n homepage: https://geoserver.org/\n version: '2.24'\n release: '2.24.2'\n ```\n\n* `extlinks`: dictionary of config.py extlinks substitutions.\n \n To convert sphinx-build config.py:\n \n ```\n extlinks = { \n 'wiki': ('https://github.com/geoserver/geoserver/wiki/%s', None),\n 'user': ('https://docs.geoserver.org/'+branch+'/en/user/%s', None),\n 'geos': ('https://osgeo-org.atlassian.net/browse/GEOS-%s','GEOS-%s')\n }\n ```\n \n Use config.yml (note use of mkdocs-macros-plugin for variable substitution:\n ```\n extlinks:\n wiki: https://github.com/geoserver/geoserver/wiki/%s\n user: https://docs.geoserver.org/{{ branch }}/en/user/%s\n geos: https://osgeo-org.atlassian.net/browse/GEOS-%s|GEOS-%s\n download_release: https://sourceforge.net/projects/geoserver/files/GeoServer/{{ release }}/geoserver-{{ release }}-%s.zip|geoserver-{{ release }}-%s.zip\n ```\n \n",
"bugtrack_url": null,
"license": "MIT-License",
"summary": "Translate from sphinx-build to mkdocs-material compatible markdown",
"version": "0.4.0",
"project_urls": {
"Changelog": "https://github.com/jodygarnett/translate/blob/main/CHANGES",
"Documentation": "https://jodygarnett.github.io/translate/setup/",
"Homepage": "https://jodygarnett.github.io/translate/",
"Repository": "https://github.com/jodygarnett/translate.git"
},
"split_keywords": [
"mkdocs",
" markdown",
" pandoc",
" sphinx-build",
" sphinx",
" rst",
" restructuredtext",
" translate"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f8ddc32fb810d8d40dca753f399dad3e2051533956e49da10bf97869c63af255",
"md5": "c0d0d2b2ccee5ded36088e6a93970914",
"sha256": "beecbc404595b19234d41ad7332334851223f19088a5c508087383d547775fa2"
},
"downloads": -1,
"filename": "mkdocs_translate-0.4.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "c0d0d2b2ccee5ded36088e6a93970914",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.9",
"size": 29910,
"upload_time": "2024-04-08T18:48:34",
"upload_time_iso_8601": "2024-04-08T18:48:34.711609Z",
"url": "https://files.pythonhosted.org/packages/f8/dd/c32fb810d8d40dca753f399dad3e2051533956e49da10bf97869c63af255/mkdocs_translate-0.4.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "04013802d6232742b125b17f57f1e67d60b19fbef68de4f1bda2f6a8164b2f23",
"md5": "bb36b83a7e097f0a8f66edc4e1125a2a",
"sha256": "a3f09392516fa8a8092defffc4724972701ad10b647ebb4bbe28a5c1df0dfb0e"
},
"downloads": -1,
"filename": "mkdocs-translate-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "bb36b83a7e097f0a8f66edc4e1125a2a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 58488,
"upload_time": "2024-04-08T18:48:36",
"upload_time_iso_8601": "2024-04-08T18:48:36.895676Z",
"url": "https://files.pythonhosted.org/packages/04/01/3802d6232742b125b17f57f1e67d60b19fbef68de4f1bda2f6a8164b2f23/mkdocs-translate-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-08 18:48:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jodygarnett",
"github_project": "translate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "mkdocs-translate"
}
JSON
