fetch-sitemap


Namefetch-sitemap JSON
Version 26 PyPI version JSON
download
home_pageNone
SummaryFetch a given sitemap and retrieve all URLs in it.
upload_time2024-05-10 03:56:55
maintainerNone
docs_urlNone
authorMartin Mahner
requires_python<4.0,>=3.9
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # fetch-sitemap

Retrieves all URLs of a given sitemap.xml URL and fetches each page one by one. 
Useful for (load) testing the entire site for error responses.

![Sample Output](https://raw.githubusercontent.com/bartTC/fetch-sitemap/main/example.png)

## Installation

```bash 
$ pip install fetch-sitemap
```

## Usage 

```
$ fetch-sitemap --help

 Usage: fetch-sitemap [OPTIONS] SITEMAP_URL

 Fetch a given sitemap and retrieve all URLs in it.

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --basic-auth                -a  TEXT              Basic auth information. Format: 'username:password'                                                          │
│ --limit                     -l  INT [>=1]         Maximum number of URLs to fetch from the given sitemap.xml.                                                  │
│ --recursive/--no-recursive                        Recursively fetch all sitemap documents from the given sitemap.xml. [default: recursive]                     │
│ --concurrency-limit         -c  INT [>=1]         Max number of concurrent requests. [default: 5; >=1]                                                         │
│ --request-timeout           -t  INT [>=1]         Timeout for fetching a URL in seconds. [default: 30; >=1]                                                    │
│ --random                    -r                    Append a random string like ?12334232343 to each URL to bypass frontend cache.                               │
│ --random-length                 INT [1 to 100]    Length of the --random hash. [default: 15; 1 to 100]                                                         │
│ --report-path               -p  FILE              Store results in a CSV file. Example: ./report.csv                                                           │
│ --output-dir                -o  DIRECTORY         Store all fetched sitemap documents in this folder. Example: /tmp/my.domain.com/                             │
│ --slow-threshold                FLOAT [>=0.0]     Responses slower than this (in seconds) are considered 'slow'. [default: 5.0; >=0.0]                         │
│ --slow-num                      INTEGER OR "ALL"  How many 'slow' responses to show. [default: 10]                                                             │
│ --user-agent                    TEXT              User-Agent string set in the HTTP header. [default: Mozilla/5.0 (compatible; fetch-sitemap/23)]              │
│ --version                                         Show the version and exit.                                                                                   │
│ --help                                            Show this message and exit.                                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

## 🤺 Local Development

```bash
poetry install
poetry run fetch-sitemap -h
poetry run ./tests.sh
```

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "fetch-sitemap",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.9",
    "maintainer_email": null,
    "keywords": null,
    "author": "Martin Mahner",
    "author_email": "martin@mahner.org",
    "download_url": "https://files.pythonhosted.org/packages/48/23/b3de1f369e7268462750f46a6e6486d66760fa5a28d14d08ccf02003dd1e/fetch_sitemap-26.tar.gz",
    "platform": null,
    "description": "# fetch-sitemap\n\nRetrieves all URLs of a given sitemap.xml URL and fetches each page one by one. \nUseful for (load) testing the entire site for error responses.\n\n![Sample Output](https://raw.githubusercontent.com/bartTC/fetch-sitemap/main/example.png)\n\n## Installation\n\n```bash \n$ pip install fetch-sitemap\n```\n\n## Usage \n\n```\n$ fetch-sitemap --help\n\n Usage: fetch-sitemap [OPTIONS] SITEMAP_URL\n\n Fetch a given sitemap and retrieve all URLs in it.\n\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --basic-auth                -a  TEXT              Basic auth information. Format: 'username:password'                                                          \u2502\n\u2502 --limit                     -l  INT [>=1]         Maximum number of URLs to fetch from the given sitemap.xml.                                                  \u2502\n\u2502 --recursive/--no-recursive                        Recursively fetch all sitemap documents from the given sitemap.xml. [default: recursive]                     \u2502\n\u2502 --concurrency-limit         -c  INT [>=1]         Max number of concurrent requests. [default: 5; >=1]                                                         \u2502\n\u2502 --request-timeout           -t  INT [>=1]         Timeout for fetching a URL in seconds. [default: 30; >=1]                                                    \u2502\n\u2502 --random                    -r                    Append a random string like ?12334232343 to each URL to bypass frontend cache.                               \u2502\n\u2502 --random-length                 INT [1 to 100]    Length of the --random hash. [default: 15; 1 to 100]                                                         \u2502\n\u2502 --report-path               -p  FILE              Store results in a CSV file. Example: ./report.csv                                                           \u2502\n\u2502 --output-dir                -o  DIRECTORY         Store all fetched sitemap documents in this folder. Example: /tmp/my.domain.com/                             \u2502\n\u2502 --slow-threshold                FLOAT [>=0.0]     Responses slower than this (in seconds) are considered 'slow'. [default: 5.0; >=0.0]                         \u2502\n\u2502 --slow-num                      INTEGER OR \"ALL\"  How many 'slow' responses to show. [default: 10]                                                             \u2502\n\u2502 --user-agent                    TEXT              User-Agent string set in the HTTP header. [default: Mozilla/5.0 (compatible; fetch-sitemap/23)]              \u2502\n\u2502 --version                                         Show the version and exit.                                                                                   \u2502\n\u2502 --help                                            Show this message and exit.                                                                                  \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n## \ud83e\udd3a Local Development\n\n```bash\npoetry install\npoetry run fetch-sitemap -h\npoetry run ./tests.sh\n```",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Fetch a given sitemap and retrieve all URLs in it.",
    "version": "26",
    "project_urls": null,
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "e79769c98c06b19021d10c1444ef5efaf059ea3ba892a7d64409ff4fcf1adaae",
                "md5": "d8e399d660718096cbf26616dae176e1",
                "sha256": "b6f30d0dc018674bd44b7fd35fe059a22d13a008668ab91b7c80aa91d92fe97d"
            },
            "downloads": -1,
            "filename": "fetch_sitemap-26-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "d8e399d660718096cbf26616dae176e1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.9",
            "size": 9874,
            "upload_time": "2024-05-10T03:56:53",
            "upload_time_iso_8601": "2024-05-10T03:56:53.777052Z",
            "url": "https://files.pythonhosted.org/packages/e7/97/69c98c06b19021d10c1444ef5efaf059ea3ba892a7d64409ff4fcf1adaae/fetch_sitemap-26-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "4823b3de1f369e7268462750f46a6e6486d66760fa5a28d14d08ccf02003dd1e",
                "md5": "fd4312a7c9491f39238e60395769ca81",
                "sha256": "eca6f7f0f7326721398808bb8d6b4baa8eb17f031949d84e571692ddf96ad82a"
            },
            "downloads": -1,
            "filename": "fetch_sitemap-26.tar.gz",
            "has_sig": false,
            "md5_digest": "fd4312a7c9491f39238e60395769ca81",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.9",
            "size": 9141,
            "upload_time": "2024-05-10T03:56:55",
            "upload_time_iso_8601": "2024-05-10T03:56:55.989177Z",
            "url": "https://files.pythonhosted.org/packages/48/23/b3de1f369e7268462750f46a6e6486d66760fa5a28d14d08ccf02003dd1e/fetch_sitemap-26.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-05-10 03:56:55",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "fetch-sitemap"
}
Martin Mahner
Elapsed time: 0.25861s