# fedfind
Fedora Finder finds Fedoras. It provides a CLI and Python module that find and providing identifying information about Fedora images and release trees. Try it out:
fedfind images --release 35
fedfind images --release 36 --milestone Beta
fedfind images --release 36 --milestone Beta --compose 1 --respin 1
fedfind images --dist Fedora-Container --release 33 --compose 20211123 --respin 0
fedfind images --compose 20211210
fedfind images --composeid Fedora-Rawhide-20211210.n.0
fedfind images --release 34 --label Beta-1.1
fedfind images --release 5 --arch x86_64,ppc
fedfind images --release 15 --search desk
Fedora has stable releases, archive releases, really old archive releases, 'milestone' releases (Alpha / Beta), release validation 'candidate' composes, unstable nightly composes, and post-release nightly composes, all in different places, with several different layouts. There is no canonical database of the locations and contents of all the various composes/releases. We in Fedora QA found we had several tools that needed to know where to find various images from various different types of releases, and little bits of knowledge about the locations and layouts of various releases/composes had been added to different tools. fedfind was written to consolidate all this esoteric knowledge in a single codebase with a consistent interface.
fedfind lets you specify a release/compose using five values: 'dist', 'release', 'milestone', 'compose', and 'respin'. It can then find the location for that compose, tell you whether it exists, and give you the locations of all the images that are part of the release and what each image actually contains. As an alternative to this versioning concept, you can also find releases by their Pungi 4 / productmd 'compose ID' or 'compose label' (see examples above).
fedfind runs on Python versions 2.7 and later (including 3.x).
## Installation and use
fedfind is packaged in the official Fedora and EPEL repositories: to install on Fedora run `dnf install fedfind`, on RHEL / CentOS with EPEL enabled, run `yum install fedfind`. You may need to enable the *updates-testing* repository to get the latest version.
You can visit [the fedfind project page on Pagure][1], and clone with `git clone https://pagure.io/fedora-qa/fedfind.git`. Tarballs are released through [PyPI][2].
You can use the fedfind CLI from the tarball without installing it, as `./fedfindlocal.py` from the root of the tarball (you will need `productmd` and `six`, and `cached_property` for Python < 3.8). You can of course copy the Python module anywhere you like and use it in place. To install both CLI and module systemwide, run `python setup.py install`.
## Bugs, pull requests etc.
You can file issues and pull requests on [Pagure][1]. Pull requests must be signed off (use the `-s` git argument). By signing off your pull request you are agreeing to the [Developer's Certificate of Origin][3]:
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
## Release identification
Some usage of fedfind relies on understanding the 'dist', 'release', 'milestone', 'compose', 'respin' versioning concept, so here is a quick primer. Note that if you intend to use fedfind solely with compose IDs or URLs for modern Pungi 4-generated composes, this section will be of less interest to you and you can probably skip it.
In this section we will write release, milestone, compose, dist quints as (release, milestone, compose, respin, dist) with '' indicating an omitted value, e.g. (22, Beta, TC3, '', 'Fedora') or (22, '', '', '', 'Fedora'). Note 'dist' is usually 'Fedora', and this is its default value, used when it is not explicitly specified. Conceptually, 'dist' should come at the front, but in fact fedfind functions which accept these values tend to place it after the others, for historical reasons (it did not exist when fedfind's versioning scheme was created).
* **Dist** is the term fedfind uses for what pungi and productmd refer to as the *shortname*; when fedfind was created this didn't really exist, but now the Fedora project produces many more composes than it used to, some with different *dists* / *shortnames*. In the compose with compose ID 'Fedora-Rawhide-20160301.n.0', the *dist* / *shortname* is *Fedora*; in a compose with compose ID 'Fedora-Container-33-20211123.0', the dist / shortname is *Fedora-Container*. *Fedora* is the **dist** for all mainline composes, and this is the default value. *Fedora-Container* is the **dist** for the nightly container image composes, produced once a day for each of the current stable releases, which at present are found only in their initial location as output by release engineering, and not in their mirrored locations. Similarly, *Fedora-Cloud* and *Fedora-IoT* are the **dists** for the nightly Cloud and IoT composes for the current stable releases. So e.g. (35, '', 20211210, 0, 'Fedora-Cloud') will find the (first) 2021-12-10 nightly Cloud compose for Fedora 35. *FedoraRespin* is the **dist** for the current post-release live respin compose in the `live-respins` directory; there is only ever one of these at a time (and note that these are only semi-official builds, provided by volunteers as a courtesy, they do not have the status of official composes). Any **release** or **compose** passed with the **dist** *FedoraRespin* is used as a *check*: if the existing contents of the `live-respins` directory don't match the expected release (number) or compose (date), fedfind will raise an exception instead of returning the `RespinRelease` instance.
* **Release** is usually a Fedora release number, e.g. 27, 15 or 1. The only non-integer values that are accepted are 'Rawhide', for Rawhide nightly composes, and 'ELN', for ELN composes. These do not, properly speaking, have a definite release number associated with them: they are perpetually rolling trees. The canonical versioning for Rawhide nightly composes is (Rawhide, '', YYYYMMDD, N) (where N is the respin number). Note that [python-wikitcms][4] uses almost the same versioning concept as fedfind, but Wikitcms 'validation events' for Rawhide nightly composes **do** have a release number: this is a property of the *validation event*, not of the *compose*. Thus there may be a Wikitcms *validation event* (24, Rawhide, 20151012, 1, 'Fedora') for the fedfind *compose* (Rawhide, '', 20151012, 1). fedfind and python-wikitcms both recognize this case and will attempt to convert each other's values for convenience.
* **Milestone** indicates the milestone or the type of nightly compose. Valid milestones for current releases are *Beta*, *RC* (or *Final*, which means the same), *Branched*, and *Production*. The *Alpha* milestone existed until Fedora 25, but Fedora 26 and later releases have no Alpha; no Alpha releases can be found any more, as the Fedora 25 and earlier Alphas are now removed. fedfind will accept *Rawhide* as a milestone and convert it to the release - so ('', Rawhide, YYYYMMDD, N, 'Fedora') is not exactly *valid* but will be handled by the CLI and the `get_release` function and converted to (Rawhide, '', YYYYMMDD, N, 'Fedora'). Stable releases do not have a milestone; (23, RC, '', '', 'Fedora') will be accepted by `get_release` and the CLI but is treated internally as (23, '', '', '', 'Fedora'). The *Production* milestone indicates a so-called 'production' compose, which will usually also be an Alpha, Beta or Final 'candidate' compose - you may be able to find the same compose in two different places, for instance, with the *Production* milestone and a date-based **compose** and **respin**, or with the *Alpha*, *Beta* or *Final* milestone and a numeric **compose** and **respin**. It is approximately the same as the difference between searching for a production compose by compose ID and searching for it by compose label. See more on this in the *fedfind / Wikitcms vs. Pungi / productmd versioning* section below. Currently, the milestone value has no meaning for **dists** other than *Fedora*.
The values *Cloud* and *Respin* are accepted for backwards compatibility purposes; they will be translated into a **dist** value of *Fedora-Cloud* or *FedoraRespin* respectively, and a blank **milestone** value. Versions of fedfind before 4.0 overloaded the **milestone** concept to handle these **dist** values, rather than properly handling them as **dists**. This functionality may be removed in a future major release.
* **Compose** is the precise compose identifier (in cases where one is needed). For candidate composes it is always 1. For nightly composes it is a date in YYYYMMDD format. Stable releases and milestone releases do not have a compose.
* **Respin** is an integer that's bumped any time a compose which would otherwise have the same version is repeated. The concept is taken from Pungi. If we attempt to build two Rawhide nightly composes on 2016-03-01, for instance, in fedfind's versioning they are (Rawhide, '', 20160301, 0) and (Rawhide, '', 20160301, 1). The corresponding Pungi / productmd 'compose ID'-style versioning is Fedora-Rawhide-20160301.n.0 and Fedora-Rawhide-20160301.n.1.
Note that, as a convenience, fedfind will attempt to detect when a 'compose' value is in fact a combined compose and respin, and split it up - so you can specify the compose as `1.7` (for compose `1`, respin `7`) or `20160330.0` or `20160330.n.0` (both of which will be treated as compose `20160330`, respin `0`).
Some examples:
* Stable release: (23, '', '', '', 'Fedora')
* Beta release: (24, Beta, '', '', 'Fedora')
* Candidate: (24, Beta, 1, 7, 'Fedora')
* Branched nightly: (24, Branched, 20160330, 0, 'Fedora')
* Rawhide nightly: (Rawhide, '', 20160330, 1, 'Fedora')
The test suite contains a bunch of tests for `get_release()` which incidentally may function as further examples of accepted usage. The fedfind CLI and `get_release()` are designed to guess omitted values in many cases, primarily to aid unattended usage, so e.g. a script can simply specify ('', Branched, '', '', 'Fedora') to run on the date's latest branched compose, without having to know what the current Branched release number is. More detailed information on various cases can be found in the `get_release()` function's docstring.
### fedfind / Wikitcms vs. Pungi / productmd versioning
The fedfind / Wikitcms versioning system was developed prior to the use of Pungi 4 for Fedora composes. The 'respin' concept from Pungi / productmd was then stuffed into the fedfind / Wikitcms versioning concept quite hastily to keep stuff working, and the productmd 'short' concept was also added to fedfind (usually under the name 'dist', which more closely describes its function in Fedora's context).
For now, fedfind attempts to remain compatible with its legacy versioning approach as well as possible, while also supporting release identification using productmd versioning concepts. For instance, non-Pungi 4 composes have a sloppily-faked up *cid* attribute that at least will produce the correct release number when parsed like a 'compose ID', and `get_release()` can parse *most* (we aim for 'all', but we're also realists!) compose IDs, compose URLs and compose labels and return appropriate Release instances.
Pungi has several version-ish concepts. The two most important to fedfind are the 'compose ID' and 'label'. All Pungi composes have a 'compose ID'. Not all have a label - only 'production' composes do (not 'nightly' or 'test' composes).
A compose ID looks like `Fedora-24-20160301.n.0` (nightly), or `Fedora-Rawhide-20160302.t.1` (test), or `Fedora-24-20160303.0` (production). The release number, date and type of compose are always indicated somehow. For Fedora purposes the respin value should always be present. There is no kind of 'milestone' indicator.
A label looks like `Alpha-1.2`, or `Beta-13.16`. There's a list of supported milestones (including Fedora's Alpha and Beta, but not Final - productmd uses RC instead). The first number is a public release number (RHEL has numbered milestone releases, unlike Fedora); the second is the respin value, which is considered more of a private/internal property. The system around which the scheme was designed appears to be that multiple "Alpha 1" respins are produced and tested and the final one is released as the public "Alpha 1" - thus the 'respin' concept covers approximately the same ground as Fedora's "TC" and "RC" composes used to.
Mainline Fedora nightly composes are built just as you'd expect: the short name is 'Fedora', the release number is 'Rawhide' or the actual release number, the type is nightly, and the respin value is incremented if multiple composes are run on the same date (usually only if the first fails and we want to fix it before the next day). So Fedora nightly compose IDs look like `Fedora-Rawhide-20180119.n.0`, for instance.
For milestone releases, Fedora builds 'production' composes with the label `release` number always set as 1, and the `respin` number incremented each time a compose is run. Thus for Fedora 24 Alpha validation testing we built `Alpha-1.1`, `Alpha-1.2`, `Alpha-1.3` and so on, until `Alpha-1.7` was ultimately released as the public Alpha release. Each of these composes also had a compose ID in `Fedora-24-20160323.0` format.
The post-release nightly composes for Cloud, Container and IoT images are built almost as you'd expect, but with their type as 'production', not 'nightly'. So their compose IDs look like `Fedora-Cloud-27-20180119.0` or `Fedora-IoT-26-20171215.1`. Their labels are always `RC-(date).(respin)`, e.g. `RC-20180119.0`.
Current fedfind should be capable in almost all cases of finding any findable Fedora compose by its compose ID or its label. The Fedora 'production' composes initially land in one location (on *kojipkgs*, alongside the nightly Branched and Rawhide composes, in directory names based on the compose ID) and are then mirrored to another location (on *alt*, in directory names based on the compose label). When you search for a given production / candidate compose whether you find its *kojipkgs* location or its *alt* location depends to some extent on how you search for it. Searching by compose label, or with something like *milestone* Alpha, *compose* 1, *respin* 7, will usually find its *alt* location (as a `Compose` class instance). Searching by compose ID, or with something like *milestone* Production, *compose* 20160323, *respin* 0, will usually find its *kojipkgs* location (as a `Production` class instance). However, you can set the `get_fedora_release` argument `promote` to `True` when searching by compose ID-ish values, and fedfind will attempt to find the `Compose` class (*alt* location) if it can. All fedfind release instances for Pungi 4 composes that actually exist should have the compose's compose ID as their `cid` attribute. If the compose has a label, it should be available as the `label` attribute.
## CLI
The `fedfind` CLI command gives you URLs for a release's images. For instance, `fedfind images -r 25` will print the URLs to all Fedora 25 images. You can filter the results in various ways. For more information, use `fedfind -h` and `fedfind images -h`.
## Python module
The Python module provides access to all fedfind's capabilities.
### Example usage
import fedfind.release
comp = fedfind.release.get_release(release=27)
print comp.location
for img in comp.all_images:
print(img['url'])
### Module design and API
The main part of fedfind is the `Release` class, in `fedfind.release`. The primary entry point is `fedfind.release.get_release()` - in almost all cases you would start by getting a release using that function, which takes the `release`, `milestone`, `compose`, `respin`, and `dist` values that identify a release as its arguments and returns an instance of a `Release` subclass. You may also pass `url` (which is expected to be the `/compose` directory of a Pungi 4 compose, or as a special case, `https://dl.fedoraproject.org/pub/alt/live-respins/` for the semi-official post-release live respins which live there), `cid` (a Pungi 4 compose ID), or `label` (a Pungi 4 compose label) as an alternative to release/milestone/compose/respin. If you pass a `url` or `cid`, fedfind will run a cross-check to ensure the URL or CID of the discovered compose actually matches what you requested, and raise an exception if it does not.
Anyone who used fedfind 1.x may remember the `Image` class for describing images and the `Query` class for doing searches. Both of those were removed from fedfind 2.x in favour of productmd-style metadata. All `Release` instances have a `metadata` dict which, if the release exists, will contain an `images` item which is itself a dict containing image metadata in the format of the productmd `images.json` file. For Pungi 4 releases this is read straight in from `images.json`; for pre-Pungi 4 releases fedfind synthesizes metadata in approximately the same format.
You can also use the `all_images` convenience property; this is basically a flattened form of the images metadata. It's a list of image dicts, with each image dict in the same basic form as a productmd image dict, but with a `variant` entry added to indicate its variant (in the original productmd layout, the image dicts are grouped by variant and then by arch, which is kind of a pain to parse for many use cases). Note that since fedfind 3.1.0, from Fedora 9 onwards, `boot.iso` files are not included in `all_images`(or the lower-level `all_paths`).
Since fedfind 3.3.0, image dicts also have `url` and `direct_url` entries added which provide full HTTPS URLs for the image files (so fedfind consumers no longer have to worry about constructing URLs by combining the release `location` or `alt_location` and the image `path`). `url` may go through the https://download.fedoraproject.org mirror redirector, which tries to spread load between mirrors. `direct_url` will always be a direct link. If the image is in the public mirror system, it will use the https://dl.fedoraproject.org mirror. Please use `url` unless you have a strong reason to use `direct_url`, to avoid excessive load on the server.
You're expected to roll your own queries as appropriate for your use case. The reason fedfind 1.x had a dedicated query interface was primarily to try and speed things up for nightly composes by avoiding Koji queries where possible and tailoring them where not; since fedfind no longer ever has to perform slow Koji queries to find images, the need for the `Query` class is no longer there, you can always just operate on the data in `metadata['images']` or `all_images`. Note the image `subvariant` property is extremely useful for identifying images; you may also want to use the `identify_image` function from `productmd` for this purpose.
All methods and functions in fedfind are documented directly: please do refer to the docstrings for information on their purposes. Attributes are documented with comments wherever their purpose is not immediately obvious.
All methods, functions, attributes and properties not prefixed with a `_` are considered 'public'. Public method and function signatures and all public data types will only change in major releases. fedfind has no control over the form or content of the productmd metadata, so when and how that changes is out of our hands; I will make a best effort to keep the synthesized metadata for old composes broadly in line with whatever current Pungi produces, though it is not perfect now and likely never will be (it's kinda tailored to the information I actually need from it).
#### Useful things fedfind can do
fedfind can do some useful stuff that isn't just querying images for releases:
* `Release.check_expected()` sees if all 'expected' images for the release are present.
* `Release.previous_release()` takes a cut at figuring out what the 'previous' release was, though this is difficult and may not always work or return what you expect.
* `helpers.get_current_release()` tells you what the current Fedora release is.
## How it works (roughly)
fedfind has a bunch of knowledge about where Fedora keeps various composes wired in. For Pungi 4 compose types, finding the compose is about all fedfind has to do; it reads all the information about what images are in the compose out from the metadata and exposes it.
For non-Pungi 4 composes (old stable releases) and Pungi 4 composes that are modified and have their metadata stripped (current stable and milestone releases), fedfind uses the `imagelist` files present on dl.fedoraproject.org, which contain lists of every image file in the entire tree. It finds the images for the specific compose being queried, then produces a path relative to the top of the mirror tree from the result. It can then combine that with a known prefix to produce an HTTPS URL.
For metadata, fedfind first tries to see if the compose was originally produced with Pungi 4 and its metadata is available in the metadata archive on kojipkgs. It tries to guess the compose label from the image file names, and then a compose ID from the compose label, and then looks for the metadata for that compose in the archive. If it is successful, it tries to match each image discovered from the `imagelist` file with an image dict from the original metadata, and combine the two so that the path information is correct but all other information for the image is taken from the original metadata, rather than synthesized by fedfind.
For any discovered image no matching original image dict is found for, and for composes where no original metadata is available at all, fedfind synthesizes productmd-style metadata by analyzing the file path, guessing the properties that can be guessed and omitting others.
In all cases, the result is that metadata and derived properties like `all_images` are as similar as possible for the different types of compose, so fedfind consumers can interact with non-Pungi 4 composes in the same way as Pungi 4 ones (to the extent of the metadata discovery and synthesis implementations, some stuff just isn't covered by the synthesis).
## Caveats
### Speed and resource use
With the use of small metadata files (for metadata composes) and still-quite-small image list files that are cached locally (for non-metadata composes), fedfind is much faster than it used to be. It can still take a few seconds to do all its parsing and analysis, though.
When used as a module it caches properties and image lists for the lifetime of the release instance. The image list files are cached (in `~/.cache/fedfind`) based on the 'last modified time' provided by the server they come from: for each new release instance, fedfind will hit the server to retrieve the last modified time header, but if the cached copy matches that time, it will not re-download the file. These files are also quite small so will not take long to download in any case. If the download fails for some reason, but we do have a cached copy of the necessary lists, fedfind will work but log a warning that it's using cached data which may be outdated.
If `~/.cache/fedfind` is not writeable for the user running fedfind, we fall back to using a temporary cache location that is only valid for the life of the process (and deleted on exit). This at least ensures fedfind will work in this case, but results in it being slower and doing more round trips.
You can set the environment variable `FEDFIND_NO_CACHE` to any value to disable this caching. File lists will still be written to `~/.cache/fedfind`, but the cached copy will never be used, the list will be re-downloaded each time it is requested. This is intended for downstream projects whose test suites are affected by the caching (specifically, one that uses pyvcr).
It shouldn't use too much bandwidth (though I haven't really measured), but obviously the server admins won't be happy with me if the servers get inundated with fedfind requests, so don't go *completely* crazy with it - if you want to do something script-y, please at least use the module and re-use release instances so the queries get cached.
### Can't find what ain't there
All releases other than stable releases disappear. fedfind can find stable releases all the way back to Fedora Core 1, but it is not going to find Fedora 14 Alpha, Fedora 19 Beta TC3, or nightlies from more than 2-3 weeks ago. This isn't a bug in fedfind - those images literally are not publicly available any more. Nightlies only stick around for a few weeks, candidate composes for a given milestone usually disappear once we've moved on another couple of milestones, and pre-releases (Alphas and Betas) usually disappear some time after the release in question goes stable. fedfind will only find what's actually there.
Also note that fedfind is not designed to find even *notional* locations for old non-stable releases. Due to their ephemeral nature, the patterns it uses for nightly builds and candidate composes only reflect current practice, and will simply be updated any time that practice changes. It doesn't have a big store of knowledge of what exact naming conventions we used for old composes. If you do `comp = fedfind.release.Compose(12, 'Final', 'TC4')` and read out `comp.location` or something what you get is almost certainly *not* the location where Fedora 12 Final TC4 actually lived when it was around.
### No secondary arches
fedfind does not, for the present, handle secondary arches at all. It *will* find PPC images for releases where PPC was a primary arch and i686 images for releases where i686 was a primary arch, though.
## Credits
This is pretty much all my fault. Note that aside from its external deps, older versions of fedfind (up to 1.1.2) included a copy of the `cached_property` implementation maintained [here][5] by Daniel Greenfield. The bundled copy was dropped with version 1.1.3.
## Licensing
Fedora Finder is available under the GPL, version 3 or any later version. A copy is included as COPYING.
[1]: https://pagure.io/fedora-qa/fedfind
[2]: https://pypi.python.org/pypi/fedfind
[3]: https://developercertificate.org/
[4]: https://pagure.io/fedora-qa/python-wikitcms
[5]: https://github.com/pydanny
Raw data
{
"_id": null,
"home_page": "https://pagure.io/fedora-qa/fedfind",
"name": "fedfind",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "fedora release image media iso",
"author": "Adam Williamson",
"author_email": "awilliam@redhat.com",
"download_url": "https://files.pythonhosted.org/packages/e1/41/addb70da1e3f5ec50606954d74c34869c4ce54b6f3848a5eafbefbaf7e9b/fedfind-6.0.4.tar.gz",
"platform": null,
"description": "# fedfind\n\nFedora Finder finds Fedoras. It provides a CLI and Python module that find and providing identifying information about Fedora images and release trees. Try it out:\n\n fedfind images --release 35\n fedfind images --release 36 --milestone Beta\n fedfind images --release 36 --milestone Beta --compose 1 --respin 1\n fedfind images --dist Fedora-Container --release 33 --compose 20211123 --respin 0\n fedfind images --compose 20211210\n fedfind images --composeid Fedora-Rawhide-20211210.n.0\n fedfind images --release 34 --label Beta-1.1\n fedfind images --release 5 --arch x86_64,ppc\n fedfind images --release 15 --search desk\n\nFedora has stable releases, archive releases, really old archive releases, 'milestone' releases (Alpha / Beta), release validation 'candidate' composes, unstable nightly composes, and post-release nightly composes, all in different places, with several different layouts. There is no canonical database of the locations and contents of all the various composes/releases. We in Fedora QA found we had several tools that needed to know where to find various images from various different types of releases, and little bits of knowledge about the locations and layouts of various releases/composes had been added to different tools. fedfind was written to consolidate all this esoteric knowledge in a single codebase with a consistent interface.\n\nfedfind lets you specify a release/compose using five values: 'dist', 'release', 'milestone', 'compose', and 'respin'. It can then find the location for that compose, tell you whether it exists, and give you the locations of all the images that are part of the release and what each image actually contains. As an alternative to this versioning concept, you can also find releases by their Pungi 4 / productmd 'compose ID' or 'compose label' (see examples above).\n\nfedfind runs on Python versions 2.7 and later (including 3.x).\n\n## Installation and use\n\nfedfind is packaged in the official Fedora and EPEL repositories: to install on Fedora run `dnf install fedfind`, on RHEL / CentOS with EPEL enabled, run `yum install fedfind`. You may need to enable the *updates-testing* repository to get the latest version.\n\nYou can visit [the fedfind project page on Pagure][1], and clone with `git clone https://pagure.io/fedora-qa/fedfind.git`. Tarballs are released through [PyPI][2].\n\nYou can use the fedfind CLI from the tarball without installing it, as `./fedfindlocal.py` from the root of the tarball (you will need `productmd` and `six`, and `cached_property` for Python < 3.8). You can of course copy the Python module anywhere you like and use it in place. To install both CLI and module systemwide, run `python setup.py install`.\n\n## Bugs, pull requests etc.\n\nYou can file issues and pull requests on [Pagure][1]. Pull requests must be signed off (use the `-s` git argument). By signing off your pull request you are agreeing to the [Developer's Certificate of Origin][3]:\n\n Developer's Certificate of Origin 1.1\n\n By making a contribution to this project, I certify that:\n\n (a) The contribution was created in whole or in part by me and I\n have the right to submit it under the open source license\n indicated in the file; or\n\n (b) The contribution is based upon previous work that, to the best\n of my knowledge, is covered under an appropriate open source\n license and I have the right under that license to submit that\n work with modifications, whether created in whole or in part\n by me, under the same open source license (unless I am\n permitted to submit under a different license), as indicated\n in the file; or\n\n (c) The contribution was provided directly to me by some other\n person who certified (a), (b) or (c) and I have not modified\n it.\n\n (d) I understand and agree that this project and the contribution\n are public and that a record of the contribution (including all\n personal information I submit with it, including my sign-off) is\n maintained indefinitely and may be redistributed consistent with\n this project or the open source license(s) involved.\n\n## Release identification\n\nSome usage of fedfind relies on understanding the 'dist', 'release', 'milestone', 'compose', 'respin' versioning concept, so here is a quick primer. Note that if you intend to use fedfind solely with compose IDs or URLs for modern Pungi 4-generated composes, this section will be of less interest to you and you can probably skip it.\n\nIn this section we will write release, milestone, compose, dist quints as (release, milestone, compose, respin, dist) with '' indicating an omitted value, e.g. (22, Beta, TC3, '', 'Fedora') or (22, '', '', '', 'Fedora'). Note 'dist' is usually 'Fedora', and this is its default value, used when it is not explicitly specified. Conceptually, 'dist' should come at the front, but in fact fedfind functions which accept these values tend to place it after the others, for historical reasons (it did not exist when fedfind's versioning scheme was created).\n\n* **Dist** is the term fedfind uses for what pungi and productmd refer to as the *shortname*; when fedfind was created this didn't really exist, but now the Fedora project produces many more composes than it used to, some with different *dists* / *shortnames*. In the compose with compose ID 'Fedora-Rawhide-20160301.n.0', the *dist* / *shortname* is *Fedora*; in a compose with compose ID 'Fedora-Container-33-20211123.0', the dist / shortname is *Fedora-Container*. *Fedora* is the **dist** for all mainline composes, and this is the default value. *Fedora-Container* is the **dist** for the nightly container image composes, produced once a day for each of the current stable releases, which at present are found only in their initial location as output by release engineering, and not in their mirrored locations. Similarly, *Fedora-Cloud* and *Fedora-IoT* are the **dists** for the nightly Cloud and IoT composes for the current stable releases. So e.g. (35, '', 20211210, 0, 'Fedora-Cloud') will find the (first) 2021-12-10 nightly Cloud compose for Fedora 35. *FedoraRespin* is the **dist** for the current post-release live respin compose in the `live-respins` directory; there is only ever one of these at a time (and note that these are only semi-official builds, provided by volunteers as a courtesy, they do not have the status of official composes). Any **release** or **compose** passed with the **dist** *FedoraRespin* is used as a *check*: if the existing contents of the `live-respins` directory don't match the expected release (number) or compose (date), fedfind will raise an exception instead of returning the `RespinRelease` instance.\n\n* **Release** is usually a Fedora release number, e.g. 27, 15 or 1. The only non-integer values that are accepted are 'Rawhide', for Rawhide nightly composes, and 'ELN', for ELN composes. These do not, properly speaking, have a definite release number associated with them: they are perpetually rolling trees. The canonical versioning for Rawhide nightly composes is (Rawhide, '', YYYYMMDD, N) (where N is the respin number). Note that [python-wikitcms][4] uses almost the same versioning concept as fedfind, but Wikitcms 'validation events' for Rawhide nightly composes **do** have a release number: this is a property of the *validation event*, not of the *compose*. Thus there may be a Wikitcms *validation event* (24, Rawhide, 20151012, 1, 'Fedora') for the fedfind *compose* (Rawhide, '', 20151012, 1). fedfind and python-wikitcms both recognize this case and will attempt to convert each other's values for convenience.\n\n* **Milestone** indicates the milestone or the type of nightly compose. Valid milestones for current releases are *Beta*, *RC* (or *Final*, which means the same), *Branched*, and *Production*. The *Alpha* milestone existed until Fedora 25, but Fedora 26 and later releases have no Alpha; no Alpha releases can be found any more, as the Fedora 25 and earlier Alphas are now removed. fedfind will accept *Rawhide* as a milestone and convert it to the release - so ('', Rawhide, YYYYMMDD, N, 'Fedora') is not exactly *valid* but will be handled by the CLI and the `get_release` function and converted to (Rawhide, '', YYYYMMDD, N, 'Fedora'). Stable releases do not have a milestone; (23, RC, '', '', 'Fedora') will be accepted by `get_release` and the CLI but is treated internally as (23, '', '', '', 'Fedora'). The *Production* milestone indicates a so-called 'production' compose, which will usually also be an Alpha, Beta or Final 'candidate' compose - you may be able to find the same compose in two different places, for instance, with the *Production* milestone and a date-based **compose** and **respin**, or with the *Alpha*, *Beta* or *Final* milestone and a numeric **compose** and **respin**. It is approximately the same as the difference between searching for a production compose by compose ID and searching for it by compose label. See more on this in the *fedfind / Wikitcms vs. Pungi / productmd versioning* section below. Currently, the milestone value has no meaning for **dists** other than *Fedora*.\n\nThe values *Cloud* and *Respin* are accepted for backwards compatibility purposes; they will be translated into a **dist** value of *Fedora-Cloud* or *FedoraRespin* respectively, and a blank **milestone** value. Versions of fedfind before 4.0 overloaded the **milestone** concept to handle these **dist** values, rather than properly handling them as **dists**. This functionality may be removed in a future major release.\n\n* **Compose** is the precise compose identifier (in cases where one is needed). For candidate composes it is always 1. For nightly composes it is a date in YYYYMMDD format. Stable releases and milestone releases do not have a compose.\n\n* **Respin** is an integer that's bumped any time a compose which would otherwise have the same version is repeated. The concept is taken from Pungi. If we attempt to build two Rawhide nightly composes on 2016-03-01, for instance, in fedfind's versioning they are (Rawhide, '', 20160301, 0) and (Rawhide, '', 20160301, 1). The corresponding Pungi / productmd 'compose ID'-style versioning is Fedora-Rawhide-20160301.n.0 and Fedora-Rawhide-20160301.n.1.\n\nNote that, as a convenience, fedfind will attempt to detect when a 'compose' value is in fact a combined compose and respin, and split it up - so you can specify the compose as `1.7` (for compose `1`, respin `7`) or `20160330.0` or `20160330.n.0` (both of which will be treated as compose `20160330`, respin `0`).\n\nSome examples:\n\n* Stable release: (23, '', '', '', 'Fedora')\n* Beta release: (24, Beta, '', '', 'Fedora')\n* Candidate: (24, Beta, 1, 7, 'Fedora')\n* Branched nightly: (24, Branched, 20160330, 0, 'Fedora')\n* Rawhide nightly: (Rawhide, '', 20160330, 1, 'Fedora')\n\nThe test suite contains a bunch of tests for `get_release()` which incidentally may function as further examples of accepted usage. The fedfind CLI and `get_release()` are designed to guess omitted values in many cases, primarily to aid unattended usage, so e.g. a script can simply specify ('', Branched, '', '', 'Fedora') to run on the date's latest branched compose, without having to know what the current Branched release number is. More detailed information on various cases can be found in the `get_release()` function's docstring.\n\n### fedfind / Wikitcms vs. Pungi / productmd versioning\n\nThe fedfind / Wikitcms versioning system was developed prior to the use of Pungi 4 for Fedora composes. The 'respin' concept from Pungi / productmd was then stuffed into the fedfind / Wikitcms versioning concept quite hastily to keep stuff working, and the productmd 'short' concept was also added to fedfind (usually under the name 'dist', which more closely describes its function in Fedora's context).\n\nFor now, fedfind attempts to remain compatible with its legacy versioning approach as well as possible, while also supporting release identification using productmd versioning concepts. For instance, non-Pungi 4 composes have a sloppily-faked up *cid* attribute that at least will produce the correct release number when parsed like a 'compose ID', and `get_release()` can parse *most* (we aim for 'all', but we're also realists!) compose IDs, compose URLs and compose labels and return appropriate Release instances.\n\nPungi has several version-ish concepts. The two most important to fedfind are the 'compose ID' and 'label'. All Pungi composes have a 'compose ID'. Not all have a label - only 'production' composes do (not 'nightly' or 'test' composes).\n\nA compose ID looks like `Fedora-24-20160301.n.0` (nightly), or `Fedora-Rawhide-20160302.t.1` (test), or `Fedora-24-20160303.0` (production). The release number, date and type of compose are always indicated somehow. For Fedora purposes the respin value should always be present. There is no kind of 'milestone' indicator.\n\nA label looks like `Alpha-1.2`, or `Beta-13.16`. There's a list of supported milestones (including Fedora's Alpha and Beta, but not Final - productmd uses RC instead). The first number is a public release number (RHEL has numbered milestone releases, unlike Fedora); the second is the respin value, which is considered more of a private/internal property. The system around which the scheme was designed appears to be that multiple \"Alpha 1\" respins are produced and tested and the final one is released as the public \"Alpha 1\" - thus the 'respin' concept covers approximately the same ground as Fedora's \"TC\" and \"RC\" composes used to.\n\nMainline Fedora nightly composes are built just as you'd expect: the short name is 'Fedora', the release number is 'Rawhide' or the actual release number, the type is nightly, and the respin value is incremented if multiple composes are run on the same date (usually only if the first fails and we want to fix it before the next day). So Fedora nightly compose IDs look like `Fedora-Rawhide-20180119.n.0`, for instance.\n\nFor milestone releases, Fedora builds 'production' composes with the label `release` number always set as 1, and the `respin` number incremented each time a compose is run. Thus for Fedora 24 Alpha validation testing we built `Alpha-1.1`, `Alpha-1.2`, `Alpha-1.3` and so on, until `Alpha-1.7` was ultimately released as the public Alpha release. Each of these composes also had a compose ID in `Fedora-24-20160323.0` format.\n\nThe post-release nightly composes for Cloud, Container and IoT images are built almost as you'd expect, but with their type as 'production', not 'nightly'. So their compose IDs look like `Fedora-Cloud-27-20180119.0` or `Fedora-IoT-26-20171215.1`. Their labels are always `RC-(date).(respin)`, e.g. `RC-20180119.0`.\n\nCurrent fedfind should be capable in almost all cases of finding any findable Fedora compose by its compose ID or its label. The Fedora 'production' composes initially land in one location (on *kojipkgs*, alongside the nightly Branched and Rawhide composes, in directory names based on the compose ID) and are then mirrored to another location (on *alt*, in directory names based on the compose label). When you search for a given production / candidate compose whether you find its *kojipkgs* location or its *alt* location depends to some extent on how you search for it. Searching by compose label, or with something like *milestone* Alpha, *compose* 1, *respin* 7, will usually find its *alt* location (as a `Compose` class instance). Searching by compose ID, or with something like *milestone* Production, *compose* 20160323, *respin* 0, will usually find its *kojipkgs* location (as a `Production` class instance). However, you can set the `get_fedora_release` argument `promote` to `True` when searching by compose ID-ish values, and fedfind will attempt to find the `Compose` class (*alt* location) if it can. All fedfind release instances for Pungi 4 composes that actually exist should have the compose's compose ID as their `cid` attribute. If the compose has a label, it should be available as the `label` attribute.\n\n## CLI\n\nThe `fedfind` CLI command gives you URLs for a release's images. For instance, `fedfind images -r 25` will print the URLs to all Fedora 25 images. You can filter the results in various ways. For more information, use `fedfind -h` and `fedfind images -h`.\n\n## Python module\n\nThe Python module provides access to all fedfind's capabilities.\n\n### Example usage\n\n import fedfind.release\n \n comp = fedfind.release.get_release(release=27)\n print comp.location\n for img in comp.all_images:\n print(img['url'])\n\n### Module design and API\n\nThe main part of fedfind is the `Release` class, in `fedfind.release`. The primary entry point is `fedfind.release.get_release()` - in almost all cases you would start by getting a release using that function, which takes the `release`, `milestone`, `compose`, `respin`, and `dist` values that identify a release as its arguments and returns an instance of a `Release` subclass. You may also pass `url` (which is expected to be the `/compose` directory of a Pungi 4 compose, or as a special case, `https://dl.fedoraproject.org/pub/alt/live-respins/` for the semi-official post-release live respins which live there), `cid` (a Pungi 4 compose ID), or `label` (a Pungi 4 compose label) as an alternative to release/milestone/compose/respin. If you pass a `url` or `cid`, fedfind will run a cross-check to ensure the URL or CID of the discovered compose actually matches what you requested, and raise an exception if it does not.\n\nAnyone who used fedfind 1.x may remember the `Image` class for describing images and the `Query` class for doing searches. Both of those were removed from fedfind 2.x in favour of productmd-style metadata. All `Release` instances have a `metadata` dict which, if the release exists, will contain an `images` item which is itself a dict containing image metadata in the format of the productmd `images.json` file. For Pungi 4 releases this is read straight in from `images.json`; for pre-Pungi 4 releases fedfind synthesizes metadata in approximately the same format.\n\nYou can also use the `all_images` convenience property; this is basically a flattened form of the images metadata. It's a list of image dicts, with each image dict in the same basic form as a productmd image dict, but with a `variant` entry added to indicate its variant (in the original productmd layout, the image dicts are grouped by variant and then by arch, which is kind of a pain to parse for many use cases). Note that since fedfind 3.1.0, from Fedora 9 onwards, `boot.iso` files are not included in `all_images`(or the lower-level `all_paths`).\n\nSince fedfind 3.3.0, image dicts also have `url` and `direct_url` entries added which provide full HTTPS URLs for the image files (so fedfind consumers no longer have to worry about constructing URLs by combining the release `location` or `alt_location` and the image `path`). `url` may go through the https://download.fedoraproject.org mirror redirector, which tries to spread load between mirrors. `direct_url` will always be a direct link. If the image is in the public mirror system, it will use the https://dl.fedoraproject.org mirror. Please use `url` unless you have a strong reason to use `direct_url`, to avoid excessive load on the server.\n\nYou're expected to roll your own queries as appropriate for your use case. The reason fedfind 1.x had a dedicated query interface was primarily to try and speed things up for nightly composes by avoiding Koji queries where possible and tailoring them where not; since fedfind no longer ever has to perform slow Koji queries to find images, the need for the `Query` class is no longer there, you can always just operate on the data in `metadata['images']` or `all_images`. Note the image `subvariant` property is extremely useful for identifying images; you may also want to use the `identify_image` function from `productmd` for this purpose.\n\nAll methods and functions in fedfind are documented directly: please do refer to the docstrings for information on their purposes. Attributes are documented with comments wherever their purpose is not immediately obvious.\n\nAll methods, functions, attributes and properties not prefixed with a `_` are considered 'public'. Public method and function signatures and all public data types will only change in major releases. fedfind has no control over the form or content of the productmd metadata, so when and how that changes is out of our hands; I will make a best effort to keep the synthesized metadata for old composes broadly in line with whatever current Pungi produces, though it is not perfect now and likely never will be (it's kinda tailored to the information I actually need from it).\n\n#### Useful things fedfind can do\n\nfedfind can do some useful stuff that isn't just querying images for releases:\n\n* `Release.check_expected()` sees if all 'expected' images for the release are present.\n* `Release.previous_release()` takes a cut at figuring out what the 'previous' release was, though this is difficult and may not always work or return what you expect.\n* `helpers.get_current_release()` tells you what the current Fedora release is.\n\n## How it works (roughly)\n\nfedfind has a bunch of knowledge about where Fedora keeps various composes wired in. For Pungi 4 compose types, finding the compose is about all fedfind has to do; it reads all the information about what images are in the compose out from the metadata and exposes it.\n\nFor non-Pungi 4 composes (old stable releases) and Pungi 4 composes that are modified and have their metadata stripped (current stable and milestone releases), fedfind uses the `imagelist` files present on dl.fedoraproject.org, which contain lists of every image file in the entire tree. It finds the images for the specific compose being queried, then produces a path relative to the top of the mirror tree from the result. It can then combine that with a known prefix to produce an HTTPS URL.\n\nFor metadata, fedfind first tries to see if the compose was originally produced with Pungi 4 and its metadata is available in the metadata archive on kojipkgs. It tries to guess the compose label from the image file names, and then a compose ID from the compose label, and then looks for the metadata for that compose in the archive. If it is successful, it tries to match each image discovered from the `imagelist` file with an image dict from the original metadata, and combine the two so that the path information is correct but all other information for the image is taken from the original metadata, rather than synthesized by fedfind.\n\nFor any discovered image no matching original image dict is found for, and for composes where no original metadata is available at all, fedfind synthesizes productmd-style metadata by analyzing the file path, guessing the properties that can be guessed and omitting others.\n\nIn all cases, the result is that metadata and derived properties like `all_images` are as similar as possible for the different types of compose, so fedfind consumers can interact with non-Pungi 4 composes in the same way as Pungi 4 ones (to the extent of the metadata discovery and synthesis implementations, some stuff just isn't covered by the synthesis).\n\n## Caveats\n\n### Speed and resource use\n\nWith the use of small metadata files (for metadata composes) and still-quite-small image list files that are cached locally (for non-metadata composes), fedfind is much faster than it used to be. It can still take a few seconds to do all its parsing and analysis, though.\n\nWhen used as a module it caches properties and image lists for the lifetime of the release instance. The image list files are cached (in `~/.cache/fedfind`) based on the 'last modified time' provided by the server they come from: for each new release instance, fedfind will hit the server to retrieve the last modified time header, but if the cached copy matches that time, it will not re-download the file. These files are also quite small so will not take long to download in any case. If the download fails for some reason, but we do have a cached copy of the necessary lists, fedfind will work but log a warning that it's using cached data which may be outdated.\n\nIf `~/.cache/fedfind` is not writeable for the user running fedfind, we fall back to using a temporary cache location that is only valid for the life of the process (and deleted on exit). This at least ensures fedfind will work in this case, but results in it being slower and doing more round trips.\n\nYou can set the environment variable `FEDFIND_NO_CACHE` to any value to disable this caching. File lists will still be written to `~/.cache/fedfind`, but the cached copy will never be used, the list will be re-downloaded each time it is requested. This is intended for downstream projects whose test suites are affected by the caching (specifically, one that uses pyvcr).\n\nIt shouldn't use too much bandwidth (though I haven't really measured), but obviously the server admins won't be happy with me if the servers get inundated with fedfind requests, so don't go *completely* crazy with it - if you want to do something script-y, please at least use the module and re-use release instances so the queries get cached.\n\n### Can't find what ain't there\n\nAll releases other than stable releases disappear. fedfind can find stable releases all the way back to Fedora Core 1, but it is not going to find Fedora 14 Alpha, Fedora 19 Beta TC3, or nightlies from more than 2-3 weeks ago. This isn't a bug in fedfind - those images literally are not publicly available any more. Nightlies only stick around for a few weeks, candidate composes for a given milestone usually disappear once we've moved on another couple of milestones, and pre-releases (Alphas and Betas) usually disappear some time after the release in question goes stable. fedfind will only find what's actually there.\n\nAlso note that fedfind is not designed to find even *notional* locations for old non-stable releases. Due to their ephemeral nature, the patterns it uses for nightly builds and candidate composes only reflect current practice, and will simply be updated any time that practice changes. It doesn't have a big store of knowledge of what exact naming conventions we used for old composes. If you do `comp = fedfind.release.Compose(12, 'Final', 'TC4')` and read out `comp.location` or something what you get is almost certainly *not* the location where Fedora 12 Final TC4 actually lived when it was around.\n\n### No secondary arches\n\nfedfind does not, for the present, handle secondary arches at all. It *will* find PPC images for releases where PPC was a primary arch and i686 images for releases where i686 was a primary arch, though.\n\n## Credits\n\nThis is pretty much all my fault. Note that aside from its external deps, older versions of fedfind (up to 1.1.2) included a copy of the `cached_property` implementation maintained [here][5] by Daniel Greenfield. The bundled copy was dropped with version 1.1.3.\n\n## Licensing\n\nFedora Finder is available under the GPL, version 3 or any later version. A copy is included as COPYING.\n\n[1]: https://pagure.io/fedora-qa/fedfind\n[2]: https://pypi.python.org/pypi/fedfind\n[3]: https://developercertificate.org/\n[4]: https://pagure.io/fedora-qa/python-wikitcms\n[5]: https://github.com/pydanny\n",
"bugtrack_url": null,
"license": "GPLv3+",
"summary": "Fedora Finder finds Fedora - images, more to come?",
"version": "6.0.4",
"project_urls": {
"Homepage": "https://pagure.io/fedora-qa/fedfind"
},
"split_keywords": [
"fedora",
"release",
"image",
"media",
"iso"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "973b6d2f5ff28927a87ead932c84903f4b9bd41bb627ae0b9b9d23eb87e6a6c5",
"md5": "1fa4a9e1a21fdc52ad323f147c2ca871",
"sha256": "0b6625755a5f17906cdaf7687a28e0992839c8e382ea335c3606d82d48ff5382"
},
"downloads": -1,
"filename": "fedfind-6.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1fa4a9e1a21fdc52ad323f147c2ca871",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 68858,
"upload_time": "2024-10-21T21:39:21",
"upload_time_iso_8601": "2024-10-21T21:39:21.437142Z",
"url": "https://files.pythonhosted.org/packages/97/3b/6d2f5ff28927a87ead932c84903f4b9bd41bb627ae0b9b9d23eb87e6a6c5/fedfind-6.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e141addb70da1e3f5ec50606954d74c34869c4ce54b6f3848a5eafbefbaf7e9b",
"md5": "6f1be9c8940a9509cc53f15908ad830e",
"sha256": "9feec0fd032b6a8fd131bbee4108c81948b69395dfb2b391d617f1ae4bf54727"
},
"downloads": -1,
"filename": "fedfind-6.0.4.tar.gz",
"has_sig": false,
"md5_digest": "6f1be9c8940a9509cc53f15908ad830e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 360925,
"upload_time": "2024-10-21T21:39:23",
"upload_time_iso_8601": "2024-10-21T21:39:23.173347Z",
"url": "https://files.pythonhosted.org/packages/e1/41/addb70da1e3f5ec50606954d74c34869c4ce54b6f3848a5eafbefbaf7e9b/fedfind-6.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-21 21:39:23",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "fedfind"
}
JSON