# Introduction
`recipe2txt` is a CLI-program that you can feed your urls of recipes and it spits out formatted cookbooks containing those recipes. Highlights include:
* asynchronous fetching of recipes
* formatted output either as txt- or markdown-file
* local caching of recipes
* custom recipe formatting via [jinja](https://jinja.palletsprojects.com)
The program is a wrapper for the [recipe-scrapers](https://github.com/hhursev/recipe-scrapers)-library. Please visit their README.md if you would like to know which websites are supported.
# WARNING
THIS SOFTWARE IS AT AN EARLY DEVELOPMENT STAGE.
BE CAREFUL SETTING THE `--output`-FLAG, ANY EXISTING FILES WITH THE SAME NAME WILL BE OVERWRITTEN.
TESTED ONLY ON KUBUNTU.
# Usage
Install with `pip install recipe2txt`. You can either use `recipe2txt` or `re2txt` to run the program.
```
usage: recipes2txt [-h] [--file [FILE ...]] [--output OUTPUT] [--verbosity {debug,info,warning,error,critical}]
[--connections CONNECTIONS] [--cache {only,new,default}] [--debug] [--timeout TIMEOUT]
[--output-format {md,txt}] [--user-agent USER_AGENT] [--erase-appdata] [--version]
[url ...]
Scrapes URLs of recipes into text files
positional arguments:
url URLs whose recipes should be added to the recipe-file (default: '[]')
options:
-h, --help show this help message and exit
--file [FILE ...], -f [FILE ...]
Text-files containing URLs whose recipes should be added to the recipe-file (default:
'[]')
--output OUTPUT, -o OUTPUT
Specifies an output file. THIS WILL OVERWRITE ANY EXISTING FILE WITH THE SAME NAME.
(default: '/home/pc/sciebo/Dokumente/Programming/recipe2txt/recipes')
--verbosity {debug,info,warning,error,critical}, -v {debug,info,warning,error,critical}
Sets the 'chattiness' of the program (default: 'critical')
--connections CONNECTIONS, -con CONNECTIONS
Sets the number of simultaneous connections (default: '4')
--cache {only,new,default}, -c {only,new,default}
Controls how the program should handle its cache: With 'only' no new data will be
downloaded, the recipes will be generated from data that has been downloaded previously.
If a recipe is not in the cache, it will not be written into the final output. 'new' will
make the program ignore any saved data and download the requested recipes even if they
have already been downloaded. Old data will be replaced by the new version, if it is
available. The 'default' will fetch and merge missing data with the data already saved,
only inserting new data into the cache where there was none previously. (default:
'default')
--debug, -d Activates debug-mode: Changes the directory for application data (default: 'False')
--timeout TIMEOUT, -t TIMEOUT
Sets the number of seconds the program waits for an individual website to respond, eg.
sets the connect-value of aiohttp.ClientTimeout (default: '10.0')
--output-format {md,txt}, -of {md,txt}
Sets the format for the output-file. The value defines which .jinja-template will be used
to format the file. The templates are available under
'/home/pc/.config/recipes2txt/templates'. (default: 'txt')
--user-agent USER_AGENT, -ua USER_AGENT
Sets the user-agent to be used for the requests. (default: 'Mozilla/5.0 (Windows NT 10.0;
Win64; x64; rv:115.0) Gecko/20100101 Firefox/115.0')
--erase-appdata Erases all data- and cache-files (e.g. the files listed below) (default: 'False')
--version Displays the version number (SemVer) (default: 'False')
```
## Configuration
When first run the program will generate the config-file `recipe2txt.toml` (use `recipe2txt --help` to locate it).
Every option listed above has a pendant in that file. Uncomment [^1] the line and change the value after the `=`-
sign to change the value this program uses when the option is not specified via the CLI-interface.
[^1]: Remove the leading `#`
### Jinja-Templates
When first run the program will create the config-folder `templates`. Here reside the jinja-templates used for
formatting recipes. Simply modify the existing templates or create your own.
#### Creating your own templates
Jinja-templates are provided with:
* recipes: a Python-list of `Recipe`-objects, each containing
* ingredients: the ingredients, separated by newlines
* instructions: the instructions, separated by newlines
* title
* total_time
* yields: how many people can eat this recipe
* host: which website the recipe is hosted on
* image: an url to an accompanying image
* nutrients: Python-dictionary of the nutrients
* url: the url to the recipe
* status: an integer representing the completeness of the recipe
* scraper_version: the version of the recipe-scrapers-library used to scrape this recipe
* The NA-constant
* used to represent values in the `Recipe`-object that are not available
* The functions of the `recipe2txt.utils.markdown`-module
#### Using your own templates
All `.jinja`-files in the folder are collected and their extentions are stripped to create an identifier for that template.
E.g.: The template is named `rst.jinja` => specify `--output-format rst` to use the template.
# Examples
```bash
recipe2txt www.example-url.com/tastyrecipe www.other-examle-url.org/deliciousmeal -o ~/Documents/great-recipes.txt
```
# Development
## Versioning
This project tries to adhere to (Semantic Versioning)[https://semver.org/]. While using '0.'-version-numbers, '0.N+1'-increases mean API changes (breaking and non-breaking), while '0.N.M+1' means no noticable API-changes.
## Tools
### nox
This project (ab-)uses [nox](https://github.com/wntrblm/nox) as test-(and task-)runner. Install nox from PyPi.org (e.g. `pipx install nox`). Use `nox --list` to get an overview over the different routines the [noxfile](noxfile.py) provides. For example to create the developement enviroment use `nox -s dev`.
### mypy
This project uses [mypy](https://github.com/python/mypy) for type checking. The [configuration file](pyproject.toml) contains all relevant settings, so a simple call to `mypy` from the current directory should be sufficient to typecheck the project.
### black
The project uses [black](https://github.com/psf/black) for code formatting.
## Testing
The project uses Python unittest for unit- and integration testing. The tests are defined in the `test.test_...`-modules. The `test/testfiles`-folder contains permanent and non-permanent testfiles.
Permanent testfiles are used to validate that the output of the program does not drift unintentionally. A reimplementation of the '.jinja'-templates in Python ensures their validity for example. Some of those files are derived from Material not licensed under the GPL3-License. Please see `test/testfiles/permanent/LICENSE` for more information.
Non-permanent testfiles are temporary files and folders generated during testing. They are written into the folder `test/testfiles/tmp_testfiles_re2txt`. Outside of unittest-runs this folder should never appear.
Sytem testing is facilitated via the `test.test4recipe2txt`-module. The module is used to create and run different CLI-parameter configurations of the program. At the end of a testrun it saves all files generated during the run and the initial parameters into a zip-file in `test/reports_test4recipe2txt` for later review. For each run the program uses the `test/reports_test4recipe2txt`-file to optain the urls to use in the testrun. To stress all websites equally and to avoid testing the same websites over and over the urls the file represents a queue, where the urls to use next are at the top and recently used urls will be moved to the bottom. If the file does not exist it will be generated from `test/testfiles/permanent/all_urls.txt`.
Raw data
{
"_id": null,
"home_page": null,
"name": "recipe2txt",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "recipes, cooking, scraping, website",
"author": null,
"author_email": "Jan Philipp Berg <git.7ksst@aleeas.com>",
"download_url": "https://files.pythonhosted.org/packages/b6/02/66c378d7467dbadf372f8301d6deabb007003b78e804ab8850f1144364be/recipe2txt-0.5.2.tar.gz",
"platform": null,
"description": "# Introduction\n\n`recipe2txt` is a CLI-program that you can feed your urls of recipes and it spits out formatted cookbooks containing those recipes. Highlights include:\n\n* asynchronous fetching of recipes\n* formatted output either as txt- or markdown-file\n* local caching of recipes\n* custom recipe formatting via [jinja](https://jinja.palletsprojects.com)\n\nThe program is a wrapper for the [recipe-scrapers](https://github.com/hhursev/recipe-scrapers)-library. Please visit their README.md if you would like to know which websites are supported.\n\n# WARNING\n\nTHIS SOFTWARE IS AT AN EARLY DEVELOPMENT STAGE.\n\nBE CAREFUL SETTING THE `--output`-FLAG, ANY EXISTING FILES WITH THE SAME NAME WILL BE OVERWRITTEN.\n\nTESTED ONLY ON KUBUNTU.\n\n# Usage\n\nInstall with `pip install recipe2txt`. You can either use `recipe2txt` or `re2txt` to run the program.\n\n```\nusage: recipes2txt [-h] [--file [FILE ...]] [--output OUTPUT] [--verbosity {debug,info,warning,error,critical}]\n [--connections CONNECTIONS] [--cache {only,new,default}] [--debug] [--timeout TIMEOUT]\n [--output-format {md,txt}] [--user-agent USER_AGENT] [--erase-appdata] [--version]\n [url ...]\n\nScrapes URLs of recipes into text files\n\npositional arguments:\n url URLs whose recipes should be added to the recipe-file (default: '[]')\n\noptions:\n -h, --help show this help message and exit\n --file [FILE ...], -f [FILE ...]\n Text-files containing URLs whose recipes should be added to the recipe-file (default:\n '[]')\n --output OUTPUT, -o OUTPUT\n Specifies an output file. THIS WILL OVERWRITE ANY EXISTING FILE WITH THE SAME NAME.\n (default: '/home/pc/sciebo/Dokumente/Programming/recipe2txt/recipes')\n --verbosity {debug,info,warning,error,critical}, -v {debug,info,warning,error,critical}\n Sets the 'chattiness' of the program (default: 'critical')\n --connections CONNECTIONS, -con CONNECTIONS\n Sets the number of simultaneous connections (default: '4')\n --cache {only,new,default}, -c {only,new,default}\n Controls how the program should handle its cache: With 'only' no new data will be\n downloaded, the recipes will be generated from data that has been downloaded previously.\n If a recipe is not in the cache, it will not be written into the final output. 'new' will\n make the program ignore any saved data and download the requested recipes even if they\n have already been downloaded. Old data will be replaced by the new version, if it is\n available. The 'default' will fetch and merge missing data with the data already saved,\n only inserting new data into the cache where there was none previously. (default:\n 'default')\n --debug, -d Activates debug-mode: Changes the directory for application data (default: 'False')\n --timeout TIMEOUT, -t TIMEOUT\n Sets the number of seconds the program waits for an individual website to respond, eg.\n sets the connect-value of aiohttp.ClientTimeout (default: '10.0')\n --output-format {md,txt}, -of {md,txt}\n Sets the format for the output-file. The value defines which .jinja-template will be used\n to format the file. The templates are available under\n '/home/pc/.config/recipes2txt/templates'. (default: 'txt')\n --user-agent USER_AGENT, -ua USER_AGENT\n Sets the user-agent to be used for the requests. (default: 'Mozilla/5.0 (Windows NT 10.0;\n Win64; x64; rv:115.0) Gecko/20100101 Firefox/115.0')\n --erase-appdata Erases all data- and cache-files (e.g. the files listed below) (default: 'False')\n --version Displays the version number (SemVer) (default: 'False')\n```\n\n## Configuration\n\nWhen first run the program will generate the config-file `recipe2txt.toml` (use `recipe2txt --help` to locate it).\nEvery option listed above has a pendant in that file. Uncomment [^1] the line and change the value after the `=`-\nsign to change the value this program uses when the option is not specified via the CLI-interface.\n\n[^1]: Remove the leading `#`\n\n### Jinja-Templates\n\nWhen first run the program will create the config-folder `templates`. Here reside the jinja-templates used for\nformatting recipes. Simply modify the existing templates or create your own.\n\n#### Creating your own templates\n\nJinja-templates are provided with:\n\n* recipes: a Python-list of `Recipe`-objects, each containing\n * ingredients: the ingredients, separated by newlines\n * instructions: the instructions, separated by newlines\n * title\n * total_time\n * yields: how many people can eat this recipe\n * host: which website the recipe is hosted on\n * image: an url to an accompanying image\n * nutrients: Python-dictionary of the nutrients\n * url: the url to the recipe\n * status: an integer representing the completeness of the recipe\n * scraper_version: the version of the recipe-scrapers-library used to scrape this recipe\n* The NA-constant\n * used to represent values in the `Recipe`-object that are not available\n* The functions of the `recipe2txt.utils.markdown`-module\n\n#### Using your own templates\n\nAll `.jinja`-files in the folder are collected and their extentions are stripped to create an identifier for that template.\nE.g.: The template is named `rst.jinja` => specify `--output-format rst` to use the template.\n\n\n# Examples\n\n```bash\nrecipe2txt www.example-url.com/tastyrecipe www.other-examle-url.org/deliciousmeal -o ~/Documents/great-recipes.txt\n```\n\n# Development\n\n## Versioning\n\nThis project tries to adhere to (Semantic Versioning)[https://semver.org/]. While using '0.'-version-numbers, '0.N+1'-increases mean API changes (breaking and non-breaking), while '0.N.M+1' means no noticable API-changes.\n\n## Tools\n\n### nox\n\nThis project (ab-)uses [nox](https://github.com/wntrblm/nox) as test-(and task-)runner. Install nox from PyPi.org (e.g. `pipx install nox`). Use `nox --list` to get an overview over the different routines the [noxfile](noxfile.py) provides. For example to create the developement enviroment use `nox -s dev`.\n\n### mypy\n\nThis project uses [mypy](https://github.com/python/mypy) for type checking. The [configuration file](pyproject.toml) contains all relevant settings, so a simple call to `mypy` from the current directory should be sufficient to typecheck the project.\n\n### black\n\nThe project uses [black](https://github.com/psf/black) for code formatting.\n\n## Testing\n\nThe project uses Python unittest for unit- and integration testing. The tests are defined in the `test.test_...`-modules. The `test/testfiles`-folder contains permanent and non-permanent testfiles.\n\nPermanent testfiles are used to validate that the output of the program does not drift unintentionally. A reimplementation of the '.jinja'-templates in Python ensures their validity for example. Some of those files are derived from Material not licensed under the GPL3-License. Please see `test/testfiles/permanent/LICENSE` for more information.\n\nNon-permanent testfiles are temporary files and folders generated during testing. They are written into the folder `test/testfiles/tmp_testfiles_re2txt`. Outside of unittest-runs this folder should never appear.\n\nSytem testing is facilitated via the `test.test4recipe2txt`-module. The module is used to create and run different CLI-parameter configurations of the program. At the end of a testrun it saves all files generated during the run and the initial parameters into a zip-file in `test/reports_test4recipe2txt` for later review. For each run the program uses the `test/reports_test4recipe2txt`-file to optain the urls to use in the testrun. To stress all websites equally and to avoid testing the same websites over and over the urls the file represents a queue, where the urls to use next are at the top and recently used urls will be moved to the bottom. If the file does not exist it will be generated from `test/testfiles/permanent/all_urls.txt`.\n",
"bugtrack_url": null,
"license": "GNU General Public License v3 or later (GPLv3+)",
"summary": "Scrapes recipes and converts them to txt or markdown",
"version": "0.5.2",
"project_urls": {
"Bug Tracker": "https://github.com/jp-berg/recipe2txt/issues",
"Homepage": "https://github.com/jp-berg/recipe2txt"
},
"split_keywords": [
"recipes",
" cooking",
" scraping",
" website"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "877c3dd6e2b87ed44868787379c59735b86c33d869b66943cd04aadb61e963d3",
"md5": "91fe01c008d88dfc42bfc636e58486c7",
"sha256": "1d12843ea8b1f6054f94ef36b775522ef2860ad0b86edfe7e8b8f51a44d5dc63"
},
"downloads": -1,
"filename": "recipe2txt-0.5.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "91fe01c008d88dfc42bfc636e58486c7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 64912,
"upload_time": "2024-05-02T16:09:34",
"upload_time_iso_8601": "2024-05-02T16:09:34.939183Z",
"url": "https://files.pythonhosted.org/packages/87/7c/3dd6e2b87ed44868787379c59735b86c33d869b66943cd04aadb61e963d3/recipe2txt-0.5.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b60266c378d7467dbadf372f8301d6deabb007003b78e804ab8850f1144364be",
"md5": "2946da037dd8db110e135d404f50f950",
"sha256": "b781d60c935a40b7203d0b451b47eb055392d6cb2cca21dec9b46bb915e9024f"
},
"downloads": -1,
"filename": "recipe2txt-0.5.2.tar.gz",
"has_sig": false,
"md5_digest": "2946da037dd8db110e135d404f50f950",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 73536,
"upload_time": "2024-05-02T16:09:37",
"upload_time_iso_8601": "2024-05-02T16:09:37.458923Z",
"url": "https://files.pythonhosted.org/packages/b6/02/66c378d7467dbadf372f8301d6deabb007003b78e804ab8850f1144364be/recipe2txt-0.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-02 16:09:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jp-berg",
"github_project": "recipe2txt",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "recipe2txt"
}
JSON
