.. SPDX-FileCopyrightText: 2023 Ross Patterson <me@rpatterson.net>
..
.. SPDX-License-Identifier: MIT
########################################################################################
Feed Archiver
########################################################################################
Archive the full contents of RSS/Atom syndication feeds including enclosures and assets.
****************************************************************************************
.. list-table::
:class: borderless align-right
* - .. figure:: https://img.shields.io/pypi/v/feed-archiver.svg?logo=pypi&label=PyPI&logoColor=gold
:alt: PyPI latest release version
:target: https://pypi.org/project/feed-archiver/
.. figure:: https://img.shields.io/pypi/pyversions/feed-archiver.svg?logo=python&label=Python&logoColor=gold
:alt: PyPI Python versions
:target: https://pypi.org/project/feed-archiver/
.. figure:: https://img.shields.io/badge/code%20style-black-000000.svg
:alt: Python code style
:target: https://github.com/psf/black
.. figure:: https://api.reuse.software/badge/gitlab.com/rpatterson/feed-archiver
:alt: REUSE license status
:target: https://api.reuse.software/info/gitlab.com/rpatterson/feed-archiver
- .. figure:: https://gitlab.com/rpatterson/feed-archiver/-/badges/release.svg
:alt: GitLab latest release
:target: https://gitlab.com/rpatterson/feed-archiver/-/releases
.. figure:: https://gitlab.com/rpatterson/feed-archiver/badges/main/pipeline.svg
:alt: GitLab CI/CD pipeline status
:target: https://gitlab.com/rpatterson/feed-archiver/-/commits/main
.. figure:: https://gitlab.com/rpatterson/feed-archiver/badges/main/coverage.svg
:alt: GitLab coverage report
:target: https://gitlab.com/rpatterson/feed-archiver/-/commits/main
.. figure:: https://img.shields.io/gitlab/stars/rpatterson/feed-archiver?gitlab_url=https%3A%2F%2Fgitlab.com&logo=gitlab
:alt: GitLab repo stars
:target: https://gitlab.com/rpatterson/feed-archiver
- .. figure:: https://img.shields.io/github/v/release/rpatterson/feed-archiver?logo=github
:alt: GitHub release (latest SemVer)
:target: https://github.com/rpatterson/feed-archiver/releases
.. figure:: https://github.com/rpatterson/feed-archiver/actions/workflows/build-test.yml/badge.svg
:alt: GitHub Actions status
:target: https://github.com/rpatterson/feed-archiver/actions/workflows/build-test.yml
.. figure:: https://codecov.io/github/rpatterson/feed-archiver/branch/main/graph/badge.svg?token=GNKVQ8VYOU
:alt: Codecov test coverage
:target: https://app.codecov.io/github/rpatterson/feed-archiver
.. figure:: https://img.shields.io/github/stars/rpatterson/feed-archiver?logo=github
:alt: GitHub repo stars
:target: https://github.com/rpatterson/feed-archiver/
- .. figure:: https://img.shields.io/docker/v/merpatterson/feed-archiver/main?sort=semver&logo=docker
:alt: Docker Hub image version (latest semver)
:target: https://hub.docker.com/r/merpatterson/feed-archiver
.. figure:: https://img.shields.io/docker/pulls/merpatterson/feed-archiver?logo=docker
:alt: Docker Hub image pulls count
:target: https://hub.docker.com/r/merpatterson/feed-archiver
.. figure:: https://img.shields.io/docker/stars/merpatterson/feed-archiver?logo=docker
:alt: Docker Hub stars
:target: https://hub.docker.com/r/merpatterson/feed-archiver
.. figure:: https://img.shields.io/docker/image-size/merpatterson/feed-archiver?logo=docker
:alt: Docker Hub image size (latest semver)
:target: https://hub.docker.com/r/merpatterson/feed-archiver
- .. figure:: https://img.shields.io/keybase/pgp/rpatterson?logo=keybase
:alt: KeyBase PGP key ID
:target: https://keybase.io/rpatterson
.. figure:: https://img.shields.io/github/followers/rpatterson?style=social
:alt: GitHub followers count
:target: https://github.com/rpatterson
.. figure:: https://img.shields.io/liberapay/receives/rpatterson.svg?logo=liberapay
:alt: LiberaPay donated per week
:target: https://liberapay.com/rpatterson/donate
.. figure:: https://img.shields.io/liberapay/patrons/rpatterson.svg?logo=liberapay
:alt: LiberaPay patrons count
:target: https://liberapay.com/rpatterson/donate
The ``$ feed-archiver`` command aims to archive RSS/Atom feeds as fully as possible in
such a way that the archive can serve (at least) 2 use cases:
#. `Mirror of Feed Enclosures and Assets`_
A mirror of the archived feeds that can be in turn served onto onto feed
clients/subscribers (such as podcatchers). For example, you can subscribe to the
archived feed from your podcatcher app on your phone with auto-download and
auto-delete of podcast episodes while archiving those same episodes on your HTPC
server with large enough storage to keep all episodes. The archived version of the
feed will also reflect the earliest form of feed XML, item XML, enclosures, and
assets that the archive downloaded and as such can be used to reflect the original
version to clients even as the remote feed changes over time.
#. `Ingest Feed Enclosures Into Media Libraries`_
An alternate hierarchy of feed item enclosures better suited for ingestion into
other media software, such as media library servers. For example, your podcast
episodes can also be made available in your `Jellyfin`_/Emby/Plex library.
.. contents:: Table of Contents
****************************************************************************************
Detailed Description
****************************************************************************************
Mirror of Feed Enclosures and Assets
========================================================================================
To serve use case #1, ``feed-archiver`` downloads enclosures and external assets
(e.g. feed and item logos specified as URLs in the feed XMLs) to the archive's local
filesystem, adjusts the URLs of the downloaded items in the feed XML, and saves the feed
XML into the archive as well. This makes the local archive filesystem suitable for
serving to feed clients/subscribers using a simple static site server such as `nginx`_.
All URLs are transformed into file-system paths that are as readable as possible while
avoiding special characters that may cause issues with common file-systems.
Specifically, special characters are ``%xx`` escaped using `Python's
urllib.parse.quote`_ function. Note that this will double-escape any
``%xx`` escapes in the remote URL:
``.../foo?bar=qux%2Fbaz#corge`` -> ``.../foo%3Fbar=qux%252Fbaz#corge``
Then the URL is converted to a corresponding filesystem path:
``https://foo-username:secret@grault.example.com/feeds/garply.rss`` ->
``./https/foo-username%3Asecret@grault.example.com/feeds/garply.rss``
Assuming the archived feeds are all hosted via HTTPS/TLS from an `nginx server_name`_ of
``feeds.example.com``, then subscribing to the archived feed in a syndication client,
such as a pod-catcher app can be done by transforming the URL like so:
``https://foo-username:secret@grault.example.com/feeds/garply.rss`` ->
``https://feeds.example.com/https/foo-username%3Asecret@grault.example.com/feeds/garply.rss``
IOW, it's as close as possible to simply prepending your archives host name to the feed
URL.
As feeds change over time, ``feed-archiver`` preserves the earliest form of feed content
as much as possible. If a feed item is changed in a subsequent retrieval of the feed,
the remote item XML is preserved instead of updating to the newer XML. More
specifically, items will be ignored on subsequent retrievals of the same feed if they
have the same ``guid``/``id`` as items that have previously been archived for that feed.
Ingest Feed Enclosures Into Media Libraries
========================================================================================
To serve use case #2, ``feed-archiver`` links the downloaded feed item enclosures into
an alternate hierarchy based on feed item metadata that better reflects the
show-with-episodes nature of most feeds, such as podcasts, with media enclosures. What
feed item metadata is used and how it's used to assemble the media library path
enclosures are linked into is configurable on a per-feed basis. This can be used, for
example, simply to make your podcasts accessible from your media library software. In a
more complex example, it can be used to link episodes from a podcast about a TV series
as `external alternative audio tracks`_ next to the corresponding TV episode video file.
Multiple linking paths can be configured such that feed item enclosures can be ingested
in multiple locations in media libraries.
Because syndication feeds may have a number of different ways to correspond to library
media, this functionality needs to be highly configurable and in order to be highly
configurable it is more complex to customize to a specific goal. As such, using this
feature requires using `an enclosure plugin`_, or the skill level of a junior developer,
or someone who is comfortable reading and interpreting technical documentation, or
re-using example configurations known to work by others.
****************************************************************************************
Installation
****************************************************************************************
Local/Native Installation
========================================================================================
Install using any tool for installing standard Python 3 distributions such as `pip`_::
$ pip3 install --user feed-archiver
Optional shell tab completion is available via `argcomplete`_.
Docker Container Image Installation
========================================================================================
The recommended way to use the Docker container image is via `Docker Compose`_. See
`the example ./docker-compose.yml file`_ for an example configuration. Once you have
your configuration, you can create and run the container::
$ docker compose up
Alternatively, you make use the image directly. Pull `the Docker image`_::
$ docker pull "registry.gitlab.com/rpatterson/feed-archiver"
And then use the image to create and run a container::
$ docker run --rm -it "registry.gitlab.com/rpatterson/feed-archiver" ...
Images variant tags are published for the Python version, branch, and major/minor
versions so that users can control when they get new images over time,
e.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310-main``. The canonical Python
version is 3.10 which is the version used in tags without ``py###``,
e.g. ``registry.gitlab.com/rpatterson/feed-archiver:main``. Pre-releases are from
``develop`` and final releases are from ``main`` which is also the default for tags
without a branch, e.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310``. The
major/minor version tags are only applied to the final release images and without the
corresponding ``main`` branch tag,
e.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310-v0.8``.
Multi-platform Docker images are published containing images for the following
platforms or architectures in the Python 3.10 ``py310`` variant:
- ``linux/amd64``
- ``linux/arm64``
- ``linux/arm/v7``
****************************************************************************************
Usage
****************************************************************************************
Create a ``./.feed-archiver.yml`` YAML file in a directory to serve as the root
directory for all feeds to be archived. The YAML file must have a top-level
``defaults`` key whose value is an object defining default or global options. In
particular, the ``base-url`` key in that section whose value must be a string which
defines the external base URL at which the archive is served to clients and is used to
assemble absolute URLs where relative URLs can't be used. The file must also have a
top-level ``feeds`` key whose value is an array or list of objects defining the remote
feeds to archive in this directory. Each feed object must contain a ``remote-url`` key
whose value is a string that contains the URL of an individual feed to archive. In the
simplest form, this can just be a file like so::
defaults:
base-url: "https://feeds.example.com"
feeds:
- title: "Garply Podcast Title"
remote-url: "\
https://foo-username:secret@grault.example.com\
/feeds/garply.rss?bar=qux%2Fbaz#corge"
...
Then run the ``$ feed-archiver`` command in that directory to update the archive from
the current version of the feeds and write an HTML index with links to the archived
feeds::
$ cd "/var/www/html/feeds/"
$ feed-archiver
INFO:Retrieving feed URL: https://foo-username:secret@grault.example.com/feeds/garply.rss
...
INFO:Writing HTML index: /var/www/html/feeds/index.html
See also the command-line help for details on options and arguments::
$ feed-archiver --help
usage: feed-archiver [-h] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
[--archive-dir [ARCHIVE_DIR]]
{update,relink} ...
Archive RSS/Atom syndication feeds and their enclosures and assets.
positional arguments:
{update,relink} sub-command
update Request the URL of each feed in the archive and update contents accordingly.
relink Re-link enclosures to the correct locations for the current configuration.
options:
-h, --help show this help message and exit
--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
Select logging verbosity. (default: INFO)
--archive-dir [ARCHIVE_DIR], -a [ARCHIVE_DIR]
the archive root directory into which all feeds, their enclosures and assets
will be downloaded (default: .)
If using the Docker container image, the container can be run from the command-line as
well::
$ docker compose run "feed-archiver" feed-archiver --help
usage: feed-archiver [-h]
To link feed item enclosures into an `alternate hierarchy`_, such as in a media library,
add a ``enclosures`` key to the feed configuration whose value is an list/array of
objects each defining one alternative path to link to the feed item enclosure. Any
``enclosures`` defined in the top-level ``defaults`` key will be used for all feeds.
Configuration to be shared across multiple ``enclosures`` configurations may be placed
in the corresponding ``defaults`` / ``plugins`` / ``enclosures`` / ``{plugin_name}``
object. The actual linking of enclosures is delegated to `plugins`_.
When updating the archive from the remote feed URLs using the ``$ feed-archiver
update`` sub-command, the enclosures of new items are linked as configured. If the
``enclosures`` configuration changes or any of the used plugins refer to external
resources that may change, such as the with the ``sonarr`` plugin when `Sonarr`_ has
upgraded or renamed the corresponding video files, use the ``$ feed-archiver relink``
command to update all existing links.
****************************************************************************************
Plugins
****************************************************************************************
How feed item enclosures are linked into a media library is delegated to plugins or
add-ons. Specifically, the ``plugin`` key in a ``enclosures`` configuration must be a
string which is the name of `a Python entry point`_ registered in the
``feedarchiver.enclosures`` group. The entry point object reference must point to a
``feedarchiver.enclosures.EnclosurePlugin`` subclass which accepts the following arguments
when instantiated:
#. ``parent=dict``
The ``feedarchiver.archive.Archive`` if the plugin is configured in ``defaults`` for
all feeds or the ``feedarchiver.feed.ArchiveFeed`` if defined for a specific feed.
#. ``config=dict``
The Python dictionary object from the de-serialized archive configuration YAML for
this specific enclosure configuration.
and whose instances must be callable and accept the following arguments when called:
#. ``archive_feed=feedarchiver.feed.ArchiveFeed``
The object ``feedarchiver`` uses internally to represent an individual feed in the
archive.
#. ``feed_elem=xml.etree.ElementTree.Element``,
``item_elem=xml.etree.ElementTree.Element``
The `Python XML element object`_ representing the whole feed, for RSS this is the
``<channel>`` child element while for Atom this is the root ``<feed>`` element, and
the a similar object representing the specific feed item.
#. ``feed_parsed=feedparser.util.FeedParserDict``,
``item_parsed=feedparser.util.FeedParserDict``
The `feedparser`_ object representing the whole feed and the specific feed item.
#. ``url_result=lxml.etree._ElementUnicodeResult``
The `lmxl special string object`_ that contains the URL of the specific enclosure.
Can be used to access the specific enclosure element.
#. ``enclosure_path=pathlib.Path``
The path to the enclosure in the archive as a `Python pathlib.Path`_ object with the
best guess at the most correct file basename, including the suffix or extension, for
the given enclosure. This suffix takes into account the suffix from the enclosure
URL, the ``Content-Type`` header of the response to the enclosure URL request, and
finally the value of any ``type`` attribute of the enclosure element XML.
#. ``match=re.Match``
The `Python regular expression match object`_ if the ``match-pattern`` matched the
string expanded from the `Python format string`_ in the ``match-string`` key.
Particularly useful to designate `regular expression groups`_ in the
``match-pattern`` and then use the parts of ``match-string`` that matched those
groups in the format ``template``. If the ``match-pattern`` doesn't match then the
enclosure will not be linked. If there are `symbolic group names`_,
e.g. ``(?P<foo_group_name>.*)`` in the pattern, then they are also available by name
in the format string, e.g ``{foo_group_name.lower()}``. If no ``match-string`` is
provided a default is used combining the feed title, item title, and enclosure
basename with extension::
{utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}
If the plugin returns a value, it must be a list of strings and will be used as the
target paths at which to link the enclosure. Relative paths are resolved against the
archive root. These paths are not escaped, so if escaping is needed it must be a part
of the plugin configuration. If no plugins link a given enclosure, then any plugins
whose ``fallback`` key is ``true`` will be applied. Here's an example ``enclosures``
definition::
defaults:
base-url: "https://feeds.example.com"
plugins:
enclosures:
sonarr:
url: "http://localhost:8989"
api-key: "????????????????????????????????"
enclosures:
# Link all feed item enclosures into the media library under the podcasts
# directory. Link items into an album directory named by series title if
# matching.
- template: "\
/media/Library/Music/Podcasts\
/{utils.quote_sep(feed_parsed.feed.title).strip()}\
/{series_title}\
/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}"
match-string: "{utils.quote_sep(item_parsed.title).strip()}"
match-pattern: "\
(?P<item_title>.+) \\((?P<series_title>.+) \
(?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\)"
# Otherwise link into "self-titled" album directories of the same name as the
# feed.
- template: "\
/media/Library/Music/Podcasts\
/{utils.quote_sep(feed_parsed.feed.title).strip()}\
/{utils.quote_sep(feed_parsed.feed.title).strip()}\
/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}"
fallback: true
feeds:
- remote-url: "\
https://foo-username:secret@grault.example.com\
/feeds/garply.rss?bar=qux%2Fbaz#corge"
enclosures:
# This particular feed is a podcast about a TV series/show. Link enclosures
# from feed items about an individual episode next to the episode video file as
# an external audio track using a non-default plugin.
- plugin: "sonarr"
match-string: "{utils.quote_sep(item_parsed.title).strip()}"
match-pattern: "\
(?P<item_title>.+) \\((?P<series_title>.+) \
(?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\)"
stem-append: "-garply"
...
Default Template Plugin
========================================================================================
If no ``plugin`` key is specified, the ``template`` plugin is used. The link
path config may include the ``template`` key containing a `Python format string`_ which
will be expanded to determine where the feed item enclosure should be linked to. The
default ``template`` is::
./Feeds/{utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}
The format strings may reference any of `the arguments passed into enclosure plugins`_.
Sonarr TV Series Plugin
========================================================================================
The ``sonarr`` plugin uses values from the enclosure configuration and/or the ``match``
groups to lookup a TV series/show managed by `Sonarr`_, then lookup an episode video
file that corresponds to the feed item enclosure, and link the enclosure next to that
video file. The ``enclosures`` configuration or ``match`` groups must contain:
- ``url`` and ``api-key`` used to `connect to the Sonarr API`_
- ``series_id`` or ``series_title`` used to `look up the TV show/series`_, note that
using ``series_id`` saves on Sonarr API request per update
- ``season_number`` used to `lookup the episode file`_
- ``episode_numbers`` used to `lookup the episode file`_, plural to support
multi-episode files
They may also include:
- ``stem-append`` containing a string to append to the episode file stem before the
enclosure suffix/extension
****************************************************************************************
Contributing
****************************************************************************************
NOTE: `This project is hosted on GitLab`_. There's `a mirror on GitHub`_ but please use
GitLab for reporting issues, submitting PRs/MRs and any other development or maintenance
activity.
See `the ./CONTRIBUTING.rst file`_ for more details on how to get started with
development.
.. _alternate hierarchy: `Ingest Feed Enclosures Into Media Libraries`_
.. _an enclosure plugin: `Plugins`_
.. _the arguments passed into enclosure plugins: `Plugins`_
.. _pip: https://pip.pypa.io/en/stable/installation/
.. _argcomplete: https://kislyuk.github.io/argcomplete/#installation
.. _a Python entry point:
https://packaging.python.org/en/latest/specifications/entry-points/#data-model
.. _Python format string: https://docs.python.org/3/library/string.html#formatstrings
.. _Python regular expression match object:
https://docs.python.org/3/library/re.html#match-objects
.. _regular expression groups: https://docs.python.org/3/library/re.html#index-17
.. _symbolic group names: https://docs.python.org/3/library/re.html#index-18
.. _Python's urllib.parse.quote:
https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote
.. _Python pathlib.path:
https://docs.python.org/3/library/pathlib.html#accessing-individual-parts
.. _Python XML element object:
https://docs.python.org/3/library/xml.etree.elementtree.html#element-objects
.. _lmxl special string object: https://lxml.de/xpathxslt.html#xpath-return-values
.. _feedparser: https://pythonhosted.org/feedparser/index.html
.. _nginx: https://nginx.org/en/docs/
.. _nginx server_name: https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/
.. _Jellyfin: https://jellyfin.org/
.. _external alternative audio tracks:
https://jellyfin.org/docs/general/server/media/external-audio-files.html
.. _Sonarr: https://sonarr.tv
.. _connect to the Sonarr API: https://github.com/Sonarr/Sonarr/wiki/API#url
.. _look up the TV show/series: https://github.com/Sonarr/Sonarr/wiki/Series#getid
.. _lookup the episode file: https://github.com/Sonarr/Sonarr/wiki/Episode#get
.. _the Docker image: https://hub.docker.com/r/merpatterson/feed-archiver
.. _`Docker Compose`: https://docs.docker.com/compose/
.. _`the example ./docker-compose.yml file`:
https://gitlab.com/rpatterson/feed-archiver/blob/main/docker-compose.yml
.. _`This project is hosted on GitLab`:
https://gitlab.com/rpatterson/feed-archiver
.. _`a mirror on GitHub`:
https://github.com/rpatterson/feed-archiver
.. _`the ./CONTRIBUTING.rst file`:
https://gitlab.com/rpatterson/feed-archiver/blob/main/CONTRIBUTING.rst
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/rpatterson/feed-archiver",
"name": "feed-archiver",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "feeds,syndication,rss,atom,podcasts,enclosures",
"author": "Ross Patterson",
"author_email": "me@rpatterson.net",
"download_url": "https://files.pythonhosted.org/packages/ad/bc/693c25add432a91860c50788051bcb0c6704ea147214e5322ab30f6d08da/feed-archiver-2.0.6.tar.gz",
"platform": null,
"description": ".. SPDX-FileCopyrightText: 2023 Ross Patterson <me@rpatterson.net>\n..\n.. SPDX-License-Identifier: MIT\n\n########################################################################################\nFeed Archiver\n########################################################################################\nArchive the full contents of RSS/Atom syndication feeds including enclosures and assets.\n****************************************************************************************\n\n.. list-table::\n :class: borderless align-right\n\n * - .. figure:: https://img.shields.io/pypi/v/feed-archiver.svg?logo=pypi&label=PyPI&logoColor=gold\n :alt: PyPI latest release version\n :target: https://pypi.org/project/feed-archiver/\n .. figure:: https://img.shields.io/pypi/pyversions/feed-archiver.svg?logo=python&label=Python&logoColor=gold\n :alt: PyPI Python versions\n :target: https://pypi.org/project/feed-archiver/\n .. figure:: https://img.shields.io/badge/code%20style-black-000000.svg\n :alt: Python code style\n :target: https://github.com/psf/black\n .. figure:: https://api.reuse.software/badge/gitlab.com/rpatterson/feed-archiver\n :alt: REUSE license status\n :target: https://api.reuse.software/info/gitlab.com/rpatterson/feed-archiver\n\n - .. figure:: https://gitlab.com/rpatterson/feed-archiver/-/badges/release.svg\n\t :alt: GitLab latest release\n\t :target: https://gitlab.com/rpatterson/feed-archiver/-/releases\n .. figure:: https://gitlab.com/rpatterson/feed-archiver/badges/main/pipeline.svg\n :alt: GitLab CI/CD pipeline status\n :target: https://gitlab.com/rpatterson/feed-archiver/-/commits/main\n .. figure:: https://gitlab.com/rpatterson/feed-archiver/badges/main/coverage.svg\n :alt: GitLab coverage report\n\t :target: https://gitlab.com/rpatterson/feed-archiver/-/commits/main\n .. figure:: https://img.shields.io/gitlab/stars/rpatterson/feed-archiver?gitlab_url=https%3A%2F%2Fgitlab.com&logo=gitlab\n\t :alt: GitLab repo stars\n\t :target: https://gitlab.com/rpatterson/feed-archiver\n\n - .. figure:: https://img.shields.io/github/v/release/rpatterson/feed-archiver?logo=github\n\t :alt: GitHub release (latest SemVer)\n\t :target: https://github.com/rpatterson/feed-archiver/releases\n .. figure:: https://github.com/rpatterson/feed-archiver/actions/workflows/build-test.yml/badge.svg\n :alt: GitHub Actions status\n :target: https://github.com/rpatterson/feed-archiver/actions/workflows/build-test.yml\n .. figure:: https://codecov.io/github/rpatterson/feed-archiver/branch/main/graph/badge.svg?token=GNKVQ8VYOU\n :alt: Codecov test coverage\n\t :target: https://app.codecov.io/github/rpatterson/feed-archiver\n .. figure:: https://img.shields.io/github/stars/rpatterson/feed-archiver?logo=github\n\t :alt: GitHub repo stars\n\t :target: https://github.com/rpatterson/feed-archiver/\n\n - .. figure:: https://img.shields.io/docker/v/merpatterson/feed-archiver/main?sort=semver&logo=docker\n :alt: Docker Hub image version (latest semver)\n :target: https://hub.docker.com/r/merpatterson/feed-archiver\n .. figure:: https://img.shields.io/docker/pulls/merpatterson/feed-archiver?logo=docker\n :alt: Docker Hub image pulls count\n :target: https://hub.docker.com/r/merpatterson/feed-archiver\n .. figure:: https://img.shields.io/docker/stars/merpatterson/feed-archiver?logo=docker\n\t :alt: Docker Hub stars\n :target: https://hub.docker.com/r/merpatterson/feed-archiver\n .. figure:: https://img.shields.io/docker/image-size/merpatterson/feed-archiver?logo=docker\n\t :alt: Docker Hub image size (latest semver)\n :target: https://hub.docker.com/r/merpatterson/feed-archiver\n\n - .. figure:: https://img.shields.io/keybase/pgp/rpatterson?logo=keybase\n :alt: KeyBase PGP key ID\n :target: https://keybase.io/rpatterson\n .. figure:: https://img.shields.io/github/followers/rpatterson?style=social\n :alt: GitHub followers count\n :target: https://github.com/rpatterson\n .. figure:: https://img.shields.io/liberapay/receives/rpatterson.svg?logo=liberapay\n :alt: LiberaPay donated per week\n :target: https://liberapay.com/rpatterson/donate\n .. figure:: https://img.shields.io/liberapay/patrons/rpatterson.svg?logo=liberapay\n :alt: LiberaPay patrons count\n :target: https://liberapay.com/rpatterson/donate\n\nThe ``$ feed-archiver`` command aims to archive RSS/Atom feeds as fully as possible in\nsuch a way that the archive can serve (at least) 2 use cases:\n\n#. `Mirror of Feed Enclosures and Assets`_\n\n A mirror of the archived feeds that can be in turn served onto onto feed\n clients/subscribers (such as podcatchers). For example, you can subscribe to the\n archived feed from your podcatcher app on your phone with auto-download and\n auto-delete of podcast episodes while archiving those same episodes on your HTPC\n server with large enough storage to keep all episodes. The archived version of the\n feed will also reflect the earliest form of feed XML, item XML, enclosures, and\n assets that the archive downloaded and as such can be used to reflect the original\n version to clients even as the remote feed changes over time.\n\n#. `Ingest Feed Enclosures Into Media Libraries`_\n\n An alternate hierarchy of feed item enclosures better suited for ingestion into\n other media software, such as media library servers. For example, your podcast\n episodes can also be made available in your `Jellyfin`_/Emby/Plex library.\n\n.. contents:: Table of Contents\n\n****************************************************************************************\nDetailed Description\n****************************************************************************************\n\nMirror of Feed Enclosures and Assets\n========================================================================================\n\nTo serve use case #1, ``feed-archiver`` downloads enclosures and external assets\n(e.g. feed and item logos specified as URLs in the feed XMLs) to the archive's local\nfilesystem, adjusts the URLs of the downloaded items in the feed XML, and saves the feed\nXML into the archive as well. This makes the local archive filesystem suitable for\nserving to feed clients/subscribers using a simple static site server such as `nginx`_.\n\nAll URLs are transformed into file-system paths that are as readable as possible while\navoiding special characters that may cause issues with common file-systems.\nSpecifically, special characters are ``%xx`` escaped using `Python's\nurllib.parse.quote`_ function. Note that this will double-escape any\n``%xx`` escapes in the remote URL:\n\n ``.../foo?bar=qux%2Fbaz#corge`` -> ``.../foo%3Fbar=qux%252Fbaz#corge``\n\nThen the URL is converted to a corresponding filesystem path:\n\n ``https://foo-username:secret@grault.example.com/feeds/garply.rss`` ->\n ``./https/foo-username%3Asecret@grault.example.com/feeds/garply.rss``\n\nAssuming the archived feeds are all hosted via HTTPS/TLS from an `nginx server_name`_ of\n``feeds.example.com``, then subscribing to the archived feed in a syndication client,\nsuch as a pod-catcher app can be done by transforming the URL like so:\n\n ``https://foo-username:secret@grault.example.com/feeds/garply.rss`` ->\n ``https://feeds.example.com/https/foo-username%3Asecret@grault.example.com/feeds/garply.rss``\n\nIOW, it's as close as possible to simply prepending your archives host name to the feed\nURL.\n\nAs feeds change over time, ``feed-archiver`` preserves the earliest form of feed content\nas much as possible. If a feed item is changed in a subsequent retrieval of the feed,\nthe remote item XML is preserved instead of updating to the newer XML. More\nspecifically, items will be ignored on subsequent retrievals of the same feed if they\nhave the same ``guid``/``id`` as items that have previously been archived for that feed.\n\nIngest Feed Enclosures Into Media Libraries\n========================================================================================\n\nTo serve use case #2, ``feed-archiver`` links the downloaded feed item enclosures into\nan alternate hierarchy based on feed item metadata that better reflects the\nshow-with-episodes nature of most feeds, such as podcasts, with media enclosures. What\nfeed item metadata is used and how it's used to assemble the media library path\nenclosures are linked into is configurable on a per-feed basis. This can be used, for\nexample, simply to make your podcasts accessible from your media library software. In a\nmore complex example, it can be used to link episodes from a podcast about a TV series\nas `external alternative audio tracks`_ next to the corresponding TV episode video file.\nMultiple linking paths can be configured such that feed item enclosures can be ingested\nin multiple locations in media libraries.\n\nBecause syndication feeds may have a number of different ways to correspond to library\nmedia, this functionality needs to be highly configurable and in order to be highly\nconfigurable it is more complex to customize to a specific goal. As such, using this\nfeature requires using `an enclosure plugin`_, or the skill level of a junior developer,\nor someone who is comfortable reading and interpreting technical documentation, or\nre-using example configurations known to work by others.\n\n\n****************************************************************************************\nInstallation\n****************************************************************************************\n\nLocal/Native Installation\n========================================================================================\n\nInstall using any tool for installing standard Python 3 distributions such as `pip`_::\n\n $ pip3 install --user feed-archiver\n\nOptional shell tab completion is available via `argcomplete`_.\n\nDocker Container Image Installation\n========================================================================================\n\nThe recommended way to use the Docker container image is via `Docker Compose`_. See\n`the example ./docker-compose.yml file`_ for an example configuration. Once you have\nyour configuration, you can create and run the container::\n\n $ docker compose up\n\nAlternatively, you make use the image directly. Pull `the Docker image`_::\n\n $ docker pull \"registry.gitlab.com/rpatterson/feed-archiver\"\n\nAnd then use the image to create and run a container::\n\n $ docker run --rm -it \"registry.gitlab.com/rpatterson/feed-archiver\" ...\n\nImages variant tags are published for the Python version, branch, and major/minor\nversions so that users can control when they get new images over time,\ne.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310-main``. The canonical Python\nversion is 3.10 which is the version used in tags without ``py###``,\ne.g. ``registry.gitlab.com/rpatterson/feed-archiver:main``. Pre-releases are from\n``develop`` and final releases are from ``main`` which is also the default for tags\nwithout a branch, e.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310``. The\nmajor/minor version tags are only applied to the final release images and without the\ncorresponding ``main`` branch tag,\ne.g. ``registry.gitlab.com/rpatterson/feed-archiver:py310-v0.8``.\n\nMulti-platform Docker images are published containing images for the following\nplatforms or architectures in the Python 3.10 ``py310`` variant:\n\n- ``linux/amd64``\n- ``linux/arm64``\n- ``linux/arm/v7``\n\n\n****************************************************************************************\nUsage\n****************************************************************************************\n\nCreate a ``./.feed-archiver.yml`` YAML file in a directory to serve as the root\ndirectory for all feeds to be archived. The YAML file must have a top-level\n``defaults`` key whose value is an object defining default or global options. In\nparticular, the ``base-url`` key in that section whose value must be a string which\ndefines the external base URL at which the archive is served to clients and is used to\nassemble absolute URLs where relative URLs can't be used. The file must also have a\ntop-level ``feeds`` key whose value is an array or list of objects defining the remote\nfeeds to archive in this directory. Each feed object must contain a ``remote-url`` key\nwhose value is a string that contains the URL of an individual feed to archive. In the\nsimplest form, this can just be a file like so::\n\n defaults:\n base-url: \"https://feeds.example.com\"\n feeds:\n - title: \"Garply Podcast Title\"\n remote-url: \"\\\n https://foo-username:secret@grault.example.com\\\n /feeds/garply.rss?bar=qux%2Fbaz#corge\"\n ...\n\nThen run the ``$ feed-archiver`` command in that directory to update the archive from\nthe current version of the feeds and write an HTML index with links to the archived\nfeeds::\n\n $ cd \"/var/www/html/feeds/\"\n $ feed-archiver\n INFO:Retrieving feed URL: https://foo-username:secret@grault.example.com/feeds/garply.rss\n ...\n INFO:Writing HTML index: /var/www/html/feeds/index.html\n\nSee also the command-line help for details on options and arguments::\n\n $ feed-archiver --help\n usage: feed-archiver [-h] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]\n\t\t [--archive-dir [ARCHIVE_DIR]]\n\t\t {update,relink} ...\n\n Archive RSS/Atom syndication feeds and their enclosures and assets.\n\n positional arguments:\n {update,relink} sub-command\n update Request the URL of each feed in the archive and update contents accordingly.\n relink Re-link enclosures to the correct locations for the current configuration.\n\n options:\n -h, --help show this help message and exit\n --log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}\n\t\t\t Select logging verbosity. (default: INFO)\n --archive-dir [ARCHIVE_DIR], -a [ARCHIVE_DIR]\n\t\t\t the archive root directory into which all feeds, their enclosures and assets\n\t\t\t will be downloaded (default: .)\n\nIf using the Docker container image, the container can be run from the command-line as\nwell::\n\n $ docker compose run \"feed-archiver\" feed-archiver --help\n usage: feed-archiver [-h]\n\nTo link feed item enclosures into an `alternate hierarchy`_, such as in a media library,\nadd a ``enclosures`` key to the feed configuration whose value is an list/array of\nobjects each defining one alternative path to link to the feed item enclosure. Any\n``enclosures`` defined in the top-level ``defaults`` key will be used for all feeds.\nConfiguration to be shared across multiple ``enclosures`` configurations may be placed\nin the corresponding ``defaults`` / ``plugins`` / ``enclosures`` / ``{plugin_name}``\nobject. The actual linking of enclosures is delegated to `plugins`_.\n\nWhen updating the archive from the remote feed URLs using the ``$ feed-archiver\nupdate`` sub-command, the enclosures of new items are linked as configured. If the\n``enclosures`` configuration changes or any of the used plugins refer to external\nresources that may change, such as the with the ``sonarr`` plugin when `Sonarr`_ has\nupgraded or renamed the corresponding video files, use the ``$ feed-archiver relink``\ncommand to update all existing links.\n\n\n****************************************************************************************\nPlugins\n****************************************************************************************\n\nHow feed item enclosures are linked into a media library is delegated to plugins or\nadd-ons. Specifically, the ``plugin`` key in a ``enclosures`` configuration must be a\nstring which is the name of `a Python entry point`_ registered in the\n``feedarchiver.enclosures`` group. The entry point object reference must point to a\n``feedarchiver.enclosures.EnclosurePlugin`` subclass which accepts the following arguments\nwhen instantiated:\n\n#. ``parent=dict``\n\n The ``feedarchiver.archive.Archive`` if the plugin is configured in ``defaults`` for\n all feeds or the ``feedarchiver.feed.ArchiveFeed`` if defined for a specific feed.\n\n#. ``config=dict``\n\n The Python dictionary object from the de-serialized archive configuration YAML for\n this specific enclosure configuration.\n\nand whose instances must be callable and accept the following arguments when called:\n\n#. ``archive_feed=feedarchiver.feed.ArchiveFeed``\n\n The object ``feedarchiver`` uses internally to represent an individual feed in the\n archive.\n\n#. ``feed_elem=xml.etree.ElementTree.Element``,\n ``item_elem=xml.etree.ElementTree.Element``\n\n The `Python XML element object`_ representing the whole feed, for RSS this is the\n ``<channel>`` child element while for Atom this is the root ``<feed>`` element, and\n the a similar object representing the specific feed item.\n\n#. ``feed_parsed=feedparser.util.FeedParserDict``,\n ``item_parsed=feedparser.util.FeedParserDict``\n\n The `feedparser`_ object representing the whole feed and the specific feed item.\n\n#. ``url_result=lxml.etree._ElementUnicodeResult``\n\n The `lmxl special string object`_ that contains the URL of the specific enclosure.\n Can be used to access the specific enclosure element.\n\n#. ``enclosure_path=pathlib.Path``\n\n The path to the enclosure in the archive as a `Python pathlib.Path`_ object with the\n best guess at the most correct file basename, including the suffix or extension, for\n the given enclosure. This suffix takes into account the suffix from the enclosure\n URL, the ``Content-Type`` header of the response to the enclosure URL request, and\n finally the value of any ``type`` attribute of the enclosure element XML.\n\n#. ``match=re.Match``\n\n The `Python regular expression match object`_ if the ``match-pattern`` matched the\n string expanded from the `Python format string`_ in the ``match-string`` key.\n Particularly useful to designate `regular expression groups`_ in the\n ``match-pattern`` and then use the parts of ``match-string`` that matched those\n groups in the format ``template``. If the ``match-pattern`` doesn't match then the\n enclosure will not be linked. If there are `symbolic group names`_,\n e.g. ``(?P<foo_group_name>.*)`` in the pattern, then they are also available by name\n in the format string, e.g ``{foo_group_name.lower()}``. If no ``match-string`` is\n provided a default is used combining the feed title, item title, and enclosure\n basename with extension::\n\n {utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}\n\nIf the plugin returns a value, it must be a list of strings and will be used as the\ntarget paths at which to link the enclosure. Relative paths are resolved against the\narchive root. These paths are not escaped, so if escaping is needed it must be a part\nof the plugin configuration. If no plugins link a given enclosure, then any plugins\nwhose ``fallback`` key is ``true`` will be applied. Here's an example ``enclosures``\ndefinition::\n\n defaults:\n base-url: \"https://feeds.example.com\"\n plugins:\n enclosures:\n\tsonarr:\n\t url: \"http://localhost:8989\"\n\t api-key: \"????????????????????????????????\"\n enclosures:\n # Link all feed item enclosures into the media library under the podcasts\n # directory. Link items into an album directory named by series title if\n # matching.\n - template: \"\\\n\t/media/Library/Music/Podcasts\\\n\t/{utils.quote_sep(feed_parsed.feed.title).strip()}\\\n\t/{series_title}\\\n\t/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}\"\n\tmatch-string: \"{utils.quote_sep(item_parsed.title).strip()}\"\n\tmatch-pattern: \"\\\n\t(?P<item_title>.+) \\\\((?P<series_title>.+) \\\n\t(?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\\\)\"\n # Otherwise link into \"self-titled\" album directories of the same name as the\n # feed.\n - template: \"\\\n /media/Library/Music/Podcasts\\\n /{utils.quote_sep(feed_parsed.feed.title).strip()}\\\n /{utils.quote_sep(feed_parsed.feed.title).strip()}\\\n /{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}\"\n\tfallback: true\n feeds:\n - remote-url: \"\\\n https://foo-username:secret@grault.example.com\\\n /feeds/garply.rss?bar=qux%2Fbaz#corge\"\n enclosures:\n\t# This particular feed is a podcast about a TV series/show. Link enclosures\n\t# from feed items about an individual episode next to the episode video file as\n\t# an external audio track using a non-default plugin.\n\t- plugin: \"sonarr\"\n\t match-string: \"{utils.quote_sep(item_parsed.title).strip()}\"\n\t match-pattern: \"\\\n\t (?P<item_title>.+) \\\\((?P<series_title>.+) \\\n\t (?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\\\)\"\n\t stem-append: \"-garply\"\n ...\n\nDefault Template Plugin\n========================================================================================\n\nIf no ``plugin`` key is specified, the ``template`` plugin is used. The link\npath config may include the ``template`` key containing a `Python format string`_ which\nwill be expanded to determine where the feed item enclosure should be linked to. The\ndefault ``template`` is::\n\n ./Feeds/{utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}\n\nThe format strings may reference any of `the arguments passed into enclosure plugins`_.\n\nSonarr TV Series Plugin\n========================================================================================\n\nThe ``sonarr`` plugin uses values from the enclosure configuration and/or the ``match``\ngroups to lookup a TV series/show managed by `Sonarr`_, then lookup an episode video\nfile that corresponds to the feed item enclosure, and link the enclosure next to that\nvideo file. The ``enclosures`` configuration or ``match`` groups must contain:\n\n- ``url`` and ``api-key`` used to `connect to the Sonarr API`_\n- ``series_id`` or ``series_title`` used to `look up the TV show/series`_, note that\n using ``series_id`` saves on Sonarr API request per update\n- ``season_number`` used to `lookup the episode file`_\n- ``episode_numbers`` used to `lookup the episode file`_, plural to support\n multi-episode files\n\nThey may also include:\n\n- ``stem-append`` containing a string to append to the episode file stem before the\n enclosure suffix/extension\n\n\n****************************************************************************************\nContributing\n****************************************************************************************\n\nNOTE: `This project is hosted on GitLab`_. There's `a mirror on GitHub`_ but please use\nGitLab for reporting issues, submitting PRs/MRs and any other development or maintenance\nactivity.\n\nSee `the ./CONTRIBUTING.rst file`_ for more details on how to get started with\ndevelopment.\n\n\n.. _alternate hierarchy: `Ingest Feed Enclosures Into Media Libraries`_\n.. _an enclosure plugin: `Plugins`_\n.. _the arguments passed into enclosure plugins: `Plugins`_\n\n.. _pip: https://pip.pypa.io/en/stable/installation/\n.. _argcomplete: https://kislyuk.github.io/argcomplete/#installation\n.. _a Python entry point:\n https://packaging.python.org/en/latest/specifications/entry-points/#data-model\n.. _Python format string: https://docs.python.org/3/library/string.html#formatstrings\n.. _Python regular expression match object:\n https://docs.python.org/3/library/re.html#match-objects\n.. _regular expression groups: https://docs.python.org/3/library/re.html#index-17\n.. _symbolic group names: https://docs.python.org/3/library/re.html#index-18\n.. _Python's urllib.parse.quote:\n https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote\n.. _Python pathlib.path:\n https://docs.python.org/3/library/pathlib.html#accessing-individual-parts\n.. _Python XML element object:\n https://docs.python.org/3/library/xml.etree.elementtree.html#element-objects\n.. _lmxl special string object: https://lxml.de/xpathxslt.html#xpath-return-values\n.. _feedparser: https://pythonhosted.org/feedparser/index.html\n\n.. _nginx: https://nginx.org/en/docs/\n.. _nginx server_name: https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/\n\n.. _Jellyfin: https://jellyfin.org/\n.. _external alternative audio tracks:\n https://jellyfin.org/docs/general/server/media/external-audio-files.html\n.. _Sonarr: https://sonarr.tv\n.. _connect to the Sonarr API: https://github.com/Sonarr/Sonarr/wiki/API#url\n.. _look up the TV show/series: https://github.com/Sonarr/Sonarr/wiki/Series#getid\n.. _lookup the episode file: https://github.com/Sonarr/Sonarr/wiki/Episode#get\n\n.. _the Docker image: https://hub.docker.com/r/merpatterson/feed-archiver\n.. _`Docker Compose`: https://docs.docker.com/compose/\n.. _`the example ./docker-compose.yml file`:\n https://gitlab.com/rpatterson/feed-archiver/blob/main/docker-compose.yml\n\n.. _`This project is hosted on GitLab`:\n https://gitlab.com/rpatterson/feed-archiver\n.. _`a mirror on GitHub`:\n https://github.com/rpatterson/feed-archiver\n.. _`the ./CONTRIBUTING.rst file`:\n https://gitlab.com/rpatterson/feed-archiver/blob/main/CONTRIBUTING.rst\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Archive the full contents of RSS/Atom syndication feeds including enclosures and assets.",
"version": "2.0.6",
"project_urls": {
"Homepage": "https://gitlab.com/rpatterson/feed-archiver"
},
"split_keywords": [
"feeds",
"syndication",
"rss",
"atom",
"podcasts",
"enclosures"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c77118a41516a1c9580a2a1098b4cd13a5fa411bad7e6ef9dfecd1c363dfd7ad",
"md5": "7a5da4ce93ca455726894a37bd82ca2f",
"sha256": "68acc1efaafd76a6214fc1f712f97b09de214267b79e823b8d1fe53ac98cbf5e"
},
"downloads": -1,
"filename": "feed_archiver-2.0.6-py3-none-any.whl",
"has_sig": true,
"md5_digest": "7a5da4ce93ca455726894a37bd82ca2f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 49740,
"upload_time": "2023-05-10T13:18:25",
"upload_time_iso_8601": "2023-05-10T13:18:25.193757Z",
"url": "https://files.pythonhosted.org/packages/c7/71/18a41516a1c9580a2a1098b4cd13a5fa411bad7e6ef9dfecd1c363dfd7ad/feed_archiver-2.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "adbc693c25add432a91860c50788051bcb0c6704ea147214e5322ab30f6d08da",
"md5": "55e876e41d4c533c8efe71b49c4fa3f8",
"sha256": "57cbdea4f4d783b53aa2a3f8778a11e2fe929282820e400d3aae91cc573aece4"
},
"downloads": -1,
"filename": "feed-archiver-2.0.6.tar.gz",
"has_sig": true,
"md5_digest": "55e876e41d4c533c8efe71b49c4fa3f8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 468803,
"upload_time": "2023-05-10T13:18:28",
"upload_time_iso_8601": "2023-05-10T13:18:28.003468Z",
"url": "https://files.pythonhosted.org/packages/ad/bc/693c25add432a91860c50788051bcb0c6704ea147214e5322ab30f6d08da/feed-archiver-2.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-10 13:18:28",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "rpatterson",
"gitlab_project": "feed-archiver",
"lcname": "feed-archiver"
}
JSON