| Name | amazon-wishlist-exporter JSON |
| Version |
0.10.0
JSON |
| download |
| home_page | None |
| Summary | Export Amazon wishlist metadata to JSON |
| upload_time | 2024-11-14 23:48:16 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) 2024 nateify Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
amazon
amazon-wishlist
scraper
wishlist
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# amazon-wishlist-exporter
amazon-wishlist-exporter.py - Scrapes Amazon wishlist data to JSON format
usage: amazon_wishlist_exporter.py [-h] (-u URL | -f HTML_FILE) [-t STORE_TLD] [-l STORE_LOCALE] [-p] [-d] [-s SORT_KEYS] [-c] [-y] [-o OUTPUT_FILE] [--debug]
options:
-h, --help show this help message and exit
-u URL, --url URL Amazon wishlist URL
-f HTML_FILE, --html-file HTML_FILE
Amazon wishlist HTML file
-t STORE_TLD, --store-tld STORE_TLD
Amazon store TLD
-l STORE_LOCALE, --store-locale STORE_LOCALE, --locale STORE_LOCALE
Amazon store locale
-p, --priority-is-localized
Return localized priority text instead of numeric value
-d, --iso8601 Convert localized date strings to ISO 8601 format
-s SORT_KEYS, --sort-keys SORT_KEYS
Sort key(s) for JSON output
-c, --compact-json Write compacted JSON
-y, --force Overwrite existing output file without asking
-o OUTPUT_FILE, --output-file OUTPUT_FILE
Output JSON file
--debug Print debug messages
## Installation
Install with pip:
pip install amazon-wishlist-exporter
[uv](https://docs.astral.sh/uv/) is recommended to install the package in a managed environment:
uv tool install amazon-wishlist-exporter
For improved sorting of non-Latin or mixed scripts with the Unicode Collation Algorithm, install with the optional `icu` dependency:
uv tool install amazon-wishlist-exporter[icu]
PyICU may need to be built separately: https://gitlab.pyicu.org/main/pyicu#installing-pyicu
Windows users can use pre-built wheels from here: https://github.com/cgohlke/pyicu-build/releases
Utilizing uv:
uv tool install amazon-wishlist-exporter --python 3.11 --with https://github.com/cgohlke/pyicu-build/releases/download/v2.13/PyICU-2.13-cp311-cp311-win_amd64.whl
## Dependencies
* Python >= 3.9
* amazoncaptcha
* babel
* curl_cffi
* dateparser
* selectolax
* price_parser
* tenacity
* PyICU (optional)
## Options
* `--url`: Alternative to the above, allows whole wishlist URL as input - may need to be quoted
* `--html`: For HTML files generated via below instructions
* `--store-tld`: Optional for `--html`, will be guessed from filename
* `--store-locale`: Optional - Store locale such as en_US, en_GB, de_DE, etc.
* Not all stores support all locales.
* If not specified, the default locale for that store will be chosen.
* This is required for HTML files if the locale is not in the file name
* `--sort-keys`: Optional - A single key or comma separated list of key names to sort the wishlist items by. Example `priority,name` sorts first by priority value highest to lowest, then sorts by name
* Numeric values (such as priority, rating) are sorted largest to smallest
* String values (such as name, comment) are sorted using the specified locale - when the package is installed with PyICU, the Unicode Collation Algorithm for the locale is used
## Limitations
### 403 Errors
Excessive scraping in a short time frame will cause Amazon to serve HTTP 403 errors to the session, but will clear on its own after some time.
### Private Wishlists and "Date Added" Field
This program is not capable of scraping private lists or authentication. Additionally, the "date added" field is only visible when you are authenticated (except for some old lists).
To scrape these lists, you must use the bookmarklet provided in the below section, which will download an HTML file you can pass to the program with the `--html-file` argument.
#### How to generate HTML files from a wishlist
Using "Save As" in the browser will not create a usable HTML file for this program.
To work around this, I created a bookmarklet which you can save and then open on any wishlist page. It will continuously scroll to the bottom until the list is fully loaded, and then save the rendered DOM to an HTML file with a filename understood by this program.
Create a blank/dummy bookmark in your browser and replace the URL with this value:
javascript:(function()%7Bif(window.location.host.startsWith(%22www.amazon.%22))var%20previousCount%3D-1%2CunchangedCount%3D0%2CcheckExist%3DsetInterval(function()%7Bvar%20e%3Ddocument.querySelectorAll(%22.g-item-sortable%22)%3Bif(document.getElementById(%22endOfListMarker%22)%7C%7CunchangedCount%3E%3D3)%7BclearInterval(checkExist)%3Blet%20t%3Ddocument.createElement(%22a%22)%2Cn%3Dnew%20Blob(%5Bdocument.getElementById(%22wishlist-page%22).outerHTML%5D%2C%7Btype%3A%22text%2Fhtml%22%7D)%3Bvar%20o%3Dwindow.location.host%2B%22_%22%2Bdocument.getElementById(%22listId%22).value%2B%22_%22%2Bopts.language%2B%22.html%22%3Bt.href%3DURL.createObjectURL(n)%2Ct.download%3Do%2Ct.click()%2CURL.revokeObjectURL(t.href)%7Delse%20e.length%3D%3D%3DpreviousCount%3FunchangedCount%2B%2B%3AunchangedCount%3D0%2CpreviousCount%3De.length%2C(last%3De%5Be.length-1%5D).scrollIntoView()%7D%2C2e3)%3Belse%20alert(%22This%20bookmarklet%20must%20be%20run%20on%20an%20Amazon%20site!%22)%3B%7D)()%3B
Alternatively, you can open a console using your browser's development tools, and then run the code that way:
```javascript
if(window.location.host.startsWith("www.amazon."))var previousCount=-1,unchangedCount=0,checkExist=setInterval(function(){var e=document.querySelectorAll(".g-item-sortable");if(document.getElementById("endOfListMarker")||unchangedCount>=3){clearInterval(checkExist);let t=document.createElement("a"),n=new Blob([document.getElementById("wishlist-page").outerHTML],{type:"text/html"});var o=window.location.host+"_"+document.getElementById("listId").value+"_"+opts.language+".html";t.href=URL.createObjectURL(n),t.download=o,t.click(),URL.revokeObjectURL(t.href)}else e.length===previousCount?unchangedCount++:unchangedCount=0,previousCount=e.length,(last=e[e.length-1]).scrollIntoView()},2e3);else alert("This bookmarklet must be run on an Amazon site!");
```
Raw data
{
"_id": null,
"home_page": null,
"name": "amazon-wishlist-exporter",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "amazon, amazon-wishlist, scraper, wishlist",
"author": null,
"author_email": "nateify <nateify@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/4d/54/be2e4aa849aaae7edfb5b3533e6757f0367c4876249f7fba01882f24fd2e/amazon_wishlist_exporter-0.10.0.tar.gz",
"platform": null,
"description": "# amazon-wishlist-exporter\namazon-wishlist-exporter.py - Scrapes Amazon wishlist data to JSON format\n\n usage: amazon_wishlist_exporter.py [-h] (-u URL | -f HTML_FILE) [-t STORE_TLD] [-l STORE_LOCALE] [-p] [-d] [-s SORT_KEYS] [-c] [-y] [-o OUTPUT_FILE] [--debug]\n \n options:\n -h, --help show this help message and exit\n -u URL, --url URL Amazon wishlist URL\n -f HTML_FILE, --html-file HTML_FILE\n Amazon wishlist HTML file\n -t STORE_TLD, --store-tld STORE_TLD\n Amazon store TLD\n -l STORE_LOCALE, --store-locale STORE_LOCALE, --locale STORE_LOCALE\n Amazon store locale\n -p, --priority-is-localized\n Return localized priority text instead of numeric value\n -d, --iso8601 Convert localized date strings to ISO 8601 format\n -s SORT_KEYS, --sort-keys SORT_KEYS\n Sort key(s) for JSON output\n -c, --compact-json Write compacted JSON\n -y, --force Overwrite existing output file without asking\n -o OUTPUT_FILE, --output-file OUTPUT_FILE\n Output JSON file\n --debug Print debug messages\n\n## Installation\n\nInstall with pip:\n\n pip install amazon-wishlist-exporter\n\n[uv](https://docs.astral.sh/uv/) is recommended to install the package in a managed environment:\n\n uv tool install amazon-wishlist-exporter\n\nFor improved sorting of non-Latin or mixed scripts with the Unicode Collation Algorithm, install with the optional `icu` dependency:\n\n uv tool install amazon-wishlist-exporter[icu]\n\nPyICU may need to be built separately: https://gitlab.pyicu.org/main/pyicu#installing-pyicu\n\nWindows users can use pre-built wheels from here: https://github.com/cgohlke/pyicu-build/releases\n\nUtilizing uv:\n\n uv tool install amazon-wishlist-exporter --python 3.11 --with https://github.com/cgohlke/pyicu-build/releases/download/v2.13/PyICU-2.13-cp311-cp311-win_amd64.whl\n\n## Dependencies\n\n* Python >= 3.9\n* amazoncaptcha\n* babel\n* curl_cffi\n* dateparser\n* selectolax\n* price_parser\n* tenacity\n* PyICU (optional)\n\n## Options\n\n* `--url`: Alternative to the above, allows whole wishlist URL as input - may need to be quoted\n* `--html`: For HTML files generated via below instructions\n* `--store-tld`: Optional for `--html`, will be guessed from filename\n* `--store-locale`: Optional - Store locale such as en_US, en_GB, de_DE, etc.\n * Not all stores support all locales.\n * If not specified, the default locale for that store will be chosen.\n * This is required for HTML files if the locale is not in the file name\n* `--sort-keys`: Optional - A single key or comma separated list of key names to sort the wishlist items by. Example `priority,name` sorts first by priority value highest to lowest, then sorts by name\n * Numeric values (such as priority, rating) are sorted largest to smallest\n * String values (such as name, comment) are sorted using the specified locale - when the package is installed with PyICU, the Unicode Collation Algorithm for the locale is used\n\n## Limitations\n\n\n### 403 Errors\n\nExcessive scraping in a short time frame will cause Amazon to serve HTTP 403 errors to the session, but will clear on its own after some time.\n\n### Private Wishlists and \"Date Added\" Field\n\nThis program is not capable of scraping private lists or authentication. Additionally, the \"date added\" field is only visible when you are authenticated (except for some old lists).\n\nTo scrape these lists, you must use the bookmarklet provided in the below section, which will download an HTML file you can pass to the program with the `--html-file` argument.\n\n#### How to generate HTML files from a wishlist\n\nUsing \"Save As\" in the browser will not create a usable HTML file for this program.\n\nTo work around this, I created a bookmarklet which you can save and then open on any wishlist page. It will continuously scroll to the bottom until the list is fully loaded, and then save the rendered DOM to an HTML file with a filename understood by this program.\n\nCreate a blank/dummy bookmark in your browser and replace the URL with this value:\n\n javascript:(function()%7Bif(window.location.host.startsWith(%22www.amazon.%22))var%20previousCount%3D-1%2CunchangedCount%3D0%2CcheckExist%3DsetInterval(function()%7Bvar%20e%3Ddocument.querySelectorAll(%22.g-item-sortable%22)%3Bif(document.getElementById(%22endOfListMarker%22)%7C%7CunchangedCount%3E%3D3)%7BclearInterval(checkExist)%3Blet%20t%3Ddocument.createElement(%22a%22)%2Cn%3Dnew%20Blob(%5Bdocument.getElementById(%22wishlist-page%22).outerHTML%5D%2C%7Btype%3A%22text%2Fhtml%22%7D)%3Bvar%20o%3Dwindow.location.host%2B%22_%22%2Bdocument.getElementById(%22listId%22).value%2B%22_%22%2Bopts.language%2B%22.html%22%3Bt.href%3DURL.createObjectURL(n)%2Ct.download%3Do%2Ct.click()%2CURL.revokeObjectURL(t.href)%7Delse%20e.length%3D%3D%3DpreviousCount%3FunchangedCount%2B%2B%3AunchangedCount%3D0%2CpreviousCount%3De.length%2C(last%3De%5Be.length-1%5D).scrollIntoView()%7D%2C2e3)%3Belse%20alert(%22This%20bookmarklet%20must%20be%20run%20on%20an%20Amazon%20site!%22)%3B%7D)()%3B\n\nAlternatively, you can open a console using your browser's development tools, and then run the code that way:\n\n```javascript\nif(window.location.host.startsWith(\"www.amazon.\"))var previousCount=-1,unchangedCount=0,checkExist=setInterval(function(){var e=document.querySelectorAll(\".g-item-sortable\");if(document.getElementById(\"endOfListMarker\")||unchangedCount>=3){clearInterval(checkExist);let t=document.createElement(\"a\"),n=new Blob([document.getElementById(\"wishlist-page\").outerHTML],{type:\"text/html\"});var o=window.location.host+\"_\"+document.getElementById(\"listId\").value+\"_\"+opts.language+\".html\";t.href=URL.createObjectURL(n),t.download=o,t.click(),URL.revokeObjectURL(t.href)}else e.length===previousCount?unchangedCount++:unchangedCount=0,previousCount=e.length,(last=e[e.length-1]).scrollIntoView()},2e3);else alert(\"This bookmarklet must be run on an Amazon site!\");\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 nateify Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Export Amazon wishlist metadata to JSON",
"version": "0.10.0",
"project_urls": {
"Documentation": "https://github.com/nateify/amazon-wishlist-exporter#readme",
"Homepage": "https://github.com/nateify/amazon-wishlist-exporter"
},
"split_keywords": [
"amazon",
" amazon-wishlist",
" scraper",
" wishlist"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "db8ff0319608edfbb2cba8129ba83a13b851455c729f909e5d6d08f38e6c4635",
"md5": "60f7fc4d55bf9139a286a552f07ab2e5",
"sha256": "677f0d8432932e450ee1bf60633cba43469096c222ad44f1ce5dbf2bb2330165"
},
"downloads": -1,
"filename": "amazon_wishlist_exporter-0.10.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "60f7fc4d55bf9139a286a552f07ab2e5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 17726,
"upload_time": "2024-11-14T23:48:14",
"upload_time_iso_8601": "2024-11-14T23:48:14.297554Z",
"url": "https://files.pythonhosted.org/packages/db/8f/f0319608edfbb2cba8129ba83a13b851455c729f909e5d6d08f38e6c4635/amazon_wishlist_exporter-0.10.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4d54be2e4aa849aaae7edfb5b3533e6757f0367c4876249f7fba01882f24fd2e",
"md5": "e75e45f07a1421cb58e79447ef7c7620",
"sha256": "46c1ece0da6603ee35022d26160c9991ec60680baa4dc8d5118778d5682a3455"
},
"downloads": -1,
"filename": "amazon_wishlist_exporter-0.10.0.tar.gz",
"has_sig": false,
"md5_digest": "e75e45f07a1421cb58e79447ef7c7620",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 1550623,
"upload_time": "2024-11-14T23:48:16",
"upload_time_iso_8601": "2024-11-14T23:48:16.864872Z",
"url": "https://files.pythonhosted.org/packages/4d/54/be2e4aa849aaae7edfb5b3533e6757f0367c4876249f7fba01882f24fd2e/amazon_wishlist_exporter-0.10.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-14 23:48:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nateify",
"github_project": "amazon-wishlist-exporter#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "amazon-wishlist-exporter"
}
