# On Demand Compose Service - ODCS
## What is ODCS
The main goal of ODCS is to allow generation of temporary composes using the REST API calls. By a compose, we mainly mean RPM repository with packages taken from different sources, but in the future, generation of other output types could be possible too.
ODCS can take RPMs for a compose from multiple sources like Koji tag, module built in Koji or external repository provided by Pulp tool.
## Using ODCS - client library
There is client library written in Python which allows easy access to REST API provided by ODCS server.
### Installing the ODCS client
On Fedora:
```
$ sudo dnf install odcs-client
```
If you want to install using `pip`, you can run following:
```
$ sudo pip install odcs[client]
```
In case you want your python project to depend on ODCS client library and add it to your `requirements.txt`, you can just use following to depend on ODCS client:
```
odcs[client]
```
### ODCS authentication system
ODCS server can be configured to authenticate using OpenIDC, Kerberos or SSL. Eventually it can be set in NoAuth mode to support anonymous access. Depending on the ODCS server configuration, you have to set your authentication method when creating ODCS class instance.
#### Using OpenIDC for authentication
To use OpenIDC, you have to provide the OpenIDC token to ODCS client class constructor. To obtain that OpenIDC token, you can either use `python-openidc-client`, or ask the OpenIDC provider for service token which does not have to be refreshed. Once you have the token, you can create the ODCS instance like this:
```
from odcs.client.odcs import ODCS, AuthMech
odcs = ODCS("https://odcs.fedoraproject.org",
auth_mech=AuthMech.OpenIDC,
openidc_token="your_openidc_token")
```
Getting the `openidc_token` using `python-openidc-client` library can be done like this:
```
import openidc_client
staging = False
if staging:
id_provider = 'https://id.stg.fedoraproject.org/openidc/'
else:
id_provider = 'https://id.fedoraproject.org/openidc/'
# Get the auth token using the OpenID client.
oidc = openidc_client.OpenIDCClient(
'odcs',
id_provider,
{'Token': 'Token', 'Authorization': 'Authorization'},
'odcs-authorizer',
'notsecret',
)
scopes = [
'openid',
'https://id.fedoraproject.org/scope/groups',
'https://pagure.io/odcs/new-compose',
'https://pagure.io/odcs/renew-compose',
'https://pagure.io/odcs/delete-compose',
]
try:
token = oidc.get_token(scopes, new_token=True)
token = oidc.report_token_issue()
except requests.exceptions.HTTPError as e:
print(e.response.text)
raise
```
#### Using Kerberos for authentication
To use Kerberos, you have to have valid Kerberos ticket or you need to have the Kerberos keytab file. If you want to use ODCS client library with Kerberos keytab, you have to set the `KRB5_CLIENT_KTNAME` environment variable to full path to the keytab file you want to use. You can for example do it like this:
```
from odcs.client.odcs import ODCS, AuthMech
from os import environ
environ["KRB5_CLIENT_KTNAME"] = "/full/path/to/ketab"
odcs = ODCS("https://odcs.fedoraproject.org",
auth_mech=AuthMech.Kerberos)
```
#### Using SSL for authentication
To use SSL, you have to have SSL client certificate and key files. You then have to choose SSL AuthMech and pass the paths to SSL client certificate and key like this:
```
from odcs.client.odcs import ODCS, AuthMech
odcs = ODCS("https://odcs.fedoraproject.org",
auth_mech=AuthMech.SSL,
ssl_cert="/path/to/ssl-crt.pem",
ssl_key="/path/to/ssl-key.pem")
```
### Requesting new compose
The general way how to request new ODCS compose is following:
```
compose = odcs.new_compose(sources, source_type)
```
Both `sources` and `source_type` are strings. Depending on `source_type` value, the `sources` have following meaning:
| `source_type` | `source` |
|---------------|----------|
| tag | Name of Koji tag to take RPMs from. |
| module | White-space separated NAME:STREAM or NAME:STREAM:VERSION of modules to include in compose. |
| pulp | White-space separated list of content-sets or repository ids. Repositories will be included in a compose. |
| raw_config | String in `name#commit` hash format. The `name` must match one of the raw config locations defined in ODCS server config as `raw_config_urls`. The `commit` is commit hash defining the version of raw config to use. This config is then used as input config for Pungi. |
| build | Source should be omitted in the request. The list of Koji builds included in a compose is defined by `builds` attribute. |
| pungi_compose | URL to variant repository of external compose generated by the Pungi. For example https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Server/. The generated compose will contain the same set of RPMs as the given external compose variant. The packages will be taken from the configured Koji instance. |
There are also additional optional attributes you can pass to `new_compose(...)` method:
- `seconds_to_live` - Number of seconds after which the generated compose should expire and will be removed.
- `packages` - List of packages which should be included in a compose. This is used only when `source_type` is set to `tag` or `build` to further limit the compose repository.
If the `packages` is not set, all packages in Koji tag or all packages in a `builds` list will be included in a final compose.
- `flags` - List of flags to further modify the compose output:
- `no_deps` - For `tag` `source_type`, do not resolve dependencies between packages and include only packages listed in the `packages` in the compose. For `module` `source_type`, do not resolve dependencies between modules and include only the requested module in the compose.
- `include_unpublished_pulp_repos` - For `pulp` `source_type`, include also unpublished repositories for input content-sets.
- `check_deps` - When set, abort the compose when some package has broken dependencies.
- `no_reuse` - When set, do not try to reuse old compose.
- `ignore_absent_pulp_repos` - For `pulp` `source_type`, ignore any content set that does not exist in Pulp
- `use_only_compatible_arch` - For `pulp` `source_type` only. When this flag is set, architecture hardcoded in URL returned from Pulp will be replaced with `$basearch` variable. The repository definition will also define `skip_if_unavailable = 1`. This could be useful when multiple content sets are included in the repofile to completly ignore packages from repositories for incompatible architectures.
- `sigkeys` - List of signature keys IDs. Only packages signed by one of these keys will be included in a compose. If there is no signed version of a package, compose will fail. It is also possible to pass an empty-string in a list meaning unsigned packages are allowed. For example if you want to prefer packages signed by key with ID `123` and also allow unsigned packages to appear in a compose, you can do it by setting sigkeys to `["123", ""]`.
- `results` - List of additional results which will be generated as part of a compose. Valid keys are:
- `iso` - Generates non-installable ISO files with RPMs from a compose.
- `boot.iso` - Generates `images/boot.iso` file which is needed to build base container images from resulting compose.
- `arches` - List of additional Koji arches to build this compose for. By default, the compose is built only for "x86_64" arch.
- `multilib_arches` - Subset of `arches` for which the multilib should be enabled. For each architecture in the `multilib_arches` list, ODCS will include also packages from other compatible architectures in a compose. For example when "x86_64" is included `multilib_arches`, ODCS will include also "i686" packages in a compose. The set of packages included in a composes is influenced by `multilib_method` list.
- `multilib_method` - List defining the method used to determine whether consider package as multilib. Defaults to empty list. The list can have following values:
- `iso` - Generates non-installable ISO files with RPMs from a compose.
- `runtime` - Packages whose name ends with "-devel" or "-static" suffix will be considered as multilib.
- `devel` - Packages that install some shared object file "*.so.*" will be considered as multilib.
- `all` - All pakages will be considered as multilib.
- `builds` - List of NVRs defining the Koji builds to include in a compose. Only valid for `tag` and `build` source types. For `tag` source type, the NVRs will be considered
for inclusion in a compose on top of Koji tag defined by `source`. For `build` source type, only the Koji builds defined by the NVRs will be considered for inclusion. The `packages` still need to be set to include particular packages from the Koji builds in a compose.
- `lookaside_repos` - List of URLs pointing to RPM repositories with packages which will be used by internal dependency resolver to resolve dependencies. Packages from these repositories will not appear in the resulting ODCS compose, but they are considered while checking whether the RPM dependencies are satisfied in the resulting compose when `check_deps` ODCS flag.
- `module_defaults_url` - List with URL to git repository with Module defaults data and the branch name or commit hash. For example ["https://pagure.io/releng/fedora-module-defaults.git", "master"]. This is used only when creating modular compose including non-modular RPMs.
- `modular_koji_tags` - List of Koji tags in which the modular Koji Content Generator builds are tagged. Such builds will be included in a compose.
The `new_compose` method returns `dict` object describing the compose, for example:
```
{
"arches": "x86_64 ppc64",
"flags": [
"no_deps"
],
"id": 1,
"owner": "jkaluza",
"packages": "gofer-package",
"removed_by": null,
"result_repo": "https://odcs.fedoraproject.org/composes/latest-odcs-1-1/compose/Temporary",
"result_repofile": "https://odcs.fedoraproject.org/composes/latest-odcs-1-1/compose/Temporary/odcs-1.repo",
"results": [
"repository"
],
"sigkeys": "",
"source": "f26",
"source_type": 1,
"state": 3,
"state_name": "wait",
"time_done": "2017-10-13T17:03:13Z",
"time_removed": "2017-10-14T17:00:00Z",
"time_submitted": "2017-10-13T16:59:51Z",
"time_to_expire": "2017-10-14T16:59:51Z"
}
```
The most useful data there is `result_repofile`, which points to the .repo file with URLs for generated compose. Another very important data there is the `state` and `state_name` field. There are following states of a compose:
| `state` | `state_name` | Description |
|---------|--------------|-------------|
| 0 | wait | Compose is waiting in a queue to be generated |
| 1 | generating | Compose is being generated |
| 2 | done | Compose is generated - done |
| 3 | removed | Compose has expired and is removed |
| 4 | failed | Compose generation has failed |
As you can see in our example, compose is in `wait` state and therefore we have to wait until the ODCS generates the compose.
### Waiting until the compose is generated
There are two ways how to wait for the compose generation. The preferred one is listening on Fedora messaging bus for `odcs.state.change` message with `done` or `failed` state and another one is using HTTP polling implemented in `wait_for_compose(...)` method.
If your application does not allow listening on the bus for some reason, you can use `wait_for_compose(...)` method like this:
```
compose = odcs.new_compose(sources, source_type)
# Blocks until the compose is ready, but maximally for 600 seconds.
compose = odcs.wait_for_compose(compose["id"], timeout=600)
if compose["state_name"] == "done":
print "Compose done, URL with repo file", compose["result_repofile"]
else:
print "Failed to generate compose"
```
### Checking the state of existing ODCS compose
Once you have the compose ready, you might want to check its state later. This can be done using the `get_compose(...)` method like this:
```
compose = odcs.get_compose(compose["id"])
```
### Renewing the compose
If the `time_to_expire` for your compose is getting closer and you know you want to continue using the compose, you can increase the `time_to_expire` using the `renew_compose(...)` method. You can also use this method to regenerate an expired compose in the `removed` state. Such compose will have the same versions of packages as the original compose.
```
compose = odcs.renew_compose(compose["id"])
```
## Development
### Code Convention
The code must be well formatted via ``black`` and pass ``flake8`` checking.
Run ``tox -e black,flake8`` to do the check.
### Unit-testing
Install packages required by pip to compile some python packages:
```
$ sudo dnf install -y gcc swig redhat-rpm-config python-devel openssl-devel openldap-devel \
zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel tk-devel \
git python3-cairo-devel cairo-gobject-devel gobject-introspection-devel
```
A lot of these dependencies come from the module
[pygobject](https://pygobject.readthedocs.io/en/latest/devguide/dev_environ.html#fedora-dep).
Run the tests:
```
$ make check
```
### Testing local composes from plain RPM repositories
You can test ODCS by generating compose from the `./server/tests/repo` repository using following commands:
```
$ ./create_sqlite_db
$ ./start_odcs_from_here
```
Before executing the command `start_odcs_from_here`, a messaging broker supporting AMQP protocol is required to run locally as well in order to run asynchronous tasks to generate composes. Here is an example to run a RabbitMQ broker:
```
sudo dnf install -y rabbitmq-server
sudo systemctl start rabbitmq-server
```
Add the `repo` source type to the server configuration in `./server/odcs/server/config.py`. (This will cause some tests to fail, so it needs to be reverted back after you are done with your changes!)
And in another terminal, submit a request to frontend:
```
$ ./submit_test_compose repo `pwd`/server/tests/repo ed
{
"id": 1,
"owner": "Unknown",
"result_repo": null,
"source": "/home/hanzz/code/fedora-modularization/odcs/tests/repo",
"source_type": 3,
"state": 0,
"state_name": "wait",
"time_done": null,
"time_removed": null,
"time_submitted": "2017-06-12T14:18:19Z"
}
```
You should then see the backend process generating the compose and once it's done, the resulting compose in `./test_composes/latest-Unknown-1/compose/Temporary` directory.
### Using docker-compose for creating a local setup
You can create test setup for ODCS with the docker-compose file. This yaml file creates docker container for the backend and frontend setup of ODCS and run multiple services together. These services are;
* **rabbitmq** (Handles the communication between backend and frontend)
* **postgres** (Creates the database where the localy generated composes are stored)
* **backend** (Backend service of ODCS)
* **frontend** (Frontend service of ODCS that handles the REST API)
* **static** (Apache service for making storage available)
* **beat** (Cronjob for backend to check the service status)
In addition to these, there are also three docker volumes (`odcs_odcs-composes`, `odcs_odcs-postgres` and `odcs_rabbitmq`) are created. These are providing persistent storage for the services.
This yaml file requires also an **.env** file that specified some enviroment variables for the configuration of frontend and backend. Such an **.env** file should be in the same path with the **docker-compose.yml** file and here are the necessary variables that need to be specified in this file;
```
# Pyhton path
PYTHONPATH=/src/common:/src/server:/src/client
# POSTGRES
POSTGRES_USER=odcs
POSTGRES_PASSWORD=password
# PULP CONFIGURATION
PULP_SERVER_URL=<URL of the pulp server>
PULP_USERNAME=<Username for the pulp server>
PULP_PASSWORD=<Credentials for the pulp server>
RAW_CONFIG_URLS=<Raw config settings in JSON format>
# ODCS CONFIGURATION
ODCS_CONFIG_SECTION=DevConfiguration
ODCS_CONFIG_DIR=/src/server/conf/
ODCS_CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/
ODCS_DB_URL=postgresql+psycopg2://odcs:password@postgres/odcs
# Directory where the generated composes are stored. This hast to match
# location where odcs-composes volume is mounted in frontend and backend
# containers.
ODCS_TARGET_DIR=/mnt/odcs
TARGET_DIR_URL=http://localhost:8080
# FLASK SETTINGS
# Force flask to reload application on change of source files.
FLASK_ENV=development
```
The services can be start with `sudo docker-compose up` command. If you have face an error or something that does not work correctly please use the following steps;
1. Check whether there is already created docker volume.
$ sudo docker volume ls
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
DRIVER VOLUME NAME
If the output is like above, it means there isn't any volume yet and it is sufficient to run `sudo docker-compose up` and then `sudo docker-compose down` command. This creates the volumes and if you run the above command again the output becomes as follows;
$ sudo docker volume ls
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
DRIVER VOLUME NAME
local odcs_odcs-composes
local odcs_odcs-postgres
local odcs_odcs-rabbitmq
2. After this point we need to set the correct permission to `ocds_odcs-composes` volume. In order to find where is the actual location of the volume type the following
$ sudo docker volume inspect odcs_odcs-composes
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
[
{
"Name": "odcs_odcs-composes",
"Driver": "local",
"Mountpoint": "/var/lib/containers/storage/volumes/odcs_odcs-composes/_data",
"CreatedAt": "2021-10-04T13:19:36.478636851+02:00",
"Labels": {
"io.podman.compose.project": "odcs"
},
"Scope": "local",
"Options": {}
}
]
`"Mountpoint": "/var/lib/containers/storage/volumes/odcs_odcs-composes/_data"` shows the exact location of the corresponding volume.
Add group write permission to the Mountpoint by
$ sudo chmod 775 /var/lib/containers/storage/volumes/odcs_odcs-composes/_data
3. In this step it is sufficient to run `sudo docker-compose up` command to start the services properly.
Here are some REST calls for checking the ODCS.
* `$ curl -s http://localhost:5000/api/1/composes/ | jq .` This call shows all the composes in db.
* `$ odcs --server http://localhost:5000/ create-pulp <Pulp content set>` This call starts a pulp compose that match the given pulp content set
* `$ odcs --server http://localhost:5000/ create-raw-config --compose-type test my_raw_config master` This call starts a compose with the configuration defined as my_raw_config
Raw data
{
"_id": null,
"home_page": "https://pagure.io/odcs/",
"name": "odcs",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "on demand compose service modularity fedora",
"author": "The Compose Team",
"author_email": "odcs-owner@fedoraproject.org",
"download_url": "https://files.pythonhosted.org/packages/2c/d8/f5cad90fc6db4a0a26bf072a504a6badf58e411eb568952d7faf864f0bc2/odcs-0.8.0.tar.gz",
"platform": null,
"description": "# On Demand Compose Service - ODCS\n\n\n\n## What is ODCS\n\nThe main goal of ODCS is to allow generation of temporary composes using the REST API calls. By a compose, we mainly mean RPM repository with packages taken from different sources, but in the future, generation of other output types could be possible too.\n\nODCS can take RPMs for a compose from multiple sources like Koji tag, module built in Koji or external repository provided by Pulp tool.\n\n## Using ODCS - client library\n\nThere is client library written in Python which allows easy access to REST API provided by ODCS server.\n\n### Installing the ODCS client\n\nOn Fedora:\n\n```\n$ sudo dnf install odcs-client\n```\n\nIf you want to install using `pip`, you can run following:\n\n```\n$ sudo pip install odcs[client]\n```\n\nIn case you want your python project to depend on ODCS client library and add it to your `requirements.txt`, you can just use following to depend on ODCS client:\n\n```\nodcs[client]\n```\n\n### ODCS authentication system\n\nODCS server can be configured to authenticate using OpenIDC, Kerberos or SSL. Eventually it can be set in NoAuth mode to support anonymous access. Depending on the ODCS server configuration, you have to set your authentication method when creating ODCS class instance.\n\n#### Using OpenIDC for authentication\n\nTo use OpenIDC, you have to provide the OpenIDC token to ODCS client class constructor. To obtain that OpenIDC token, you can either use `python-openidc-client`, or ask the OpenIDC provider for service token which does not have to be refreshed. Once you have the token, you can create the ODCS instance like this:\n\n```\nfrom odcs.client.odcs import ODCS, AuthMech\n\nodcs = ODCS(\"https://odcs.fedoraproject.org\",\n auth_mech=AuthMech.OpenIDC,\n openidc_token=\"your_openidc_token\")\n```\n\nGetting the `openidc_token` using `python-openidc-client` library can be done like this:\n\n```\nimport openidc_client\nstaging = False\n\nif staging:\n id_provider = 'https://id.stg.fedoraproject.org/openidc/'\nelse:\n id_provider = 'https://id.fedoraproject.org/openidc/'\n\n# Get the auth token using the OpenID client.\noidc = openidc_client.OpenIDCClient(\n 'odcs',\n id_provider,\n {'Token': 'Token', 'Authorization': 'Authorization'},\n 'odcs-authorizer',\n 'notsecret',\n)\n\nscopes = [\n 'openid',\n 'https://id.fedoraproject.org/scope/groups',\n 'https://pagure.io/odcs/new-compose',\n 'https://pagure.io/odcs/renew-compose',\n 'https://pagure.io/odcs/delete-compose',\n]\ntry:\n token = oidc.get_token(scopes, new_token=True)\n token = oidc.report_token_issue()\nexcept requests.exceptions.HTTPError as e:\n print(e.response.text)\n raise\n```\n\n#### Using Kerberos for authentication\n\nTo use Kerberos, you have to have valid Kerberos ticket or you need to have the Kerberos keytab file. If you want to use ODCS client library with Kerberos keytab, you have to set the `KRB5_CLIENT_KTNAME` environment variable to full path to the keytab file you want to use. You can for example do it like this:\n\n```\nfrom odcs.client.odcs import ODCS, AuthMech\nfrom os import environ\nenviron[\"KRB5_CLIENT_KTNAME\"] = \"/full/path/to/ketab\"\n\nodcs = ODCS(\"https://odcs.fedoraproject.org\",\n auth_mech=AuthMech.Kerberos)\n```\n\n#### Using SSL for authentication\n\nTo use SSL, you have to have SSL client certificate and key files. You then have to choose SSL AuthMech and pass the paths to SSL client certificate and key like this:\n\n```\nfrom odcs.client.odcs import ODCS, AuthMech\n\nodcs = ODCS(\"https://odcs.fedoraproject.org\",\n auth_mech=AuthMech.SSL,\n ssl_cert=\"/path/to/ssl-crt.pem\",\n ssl_key=\"/path/to/ssl-key.pem\")\n```\n\n### Requesting new compose\n\nThe general way how to request new ODCS compose is following:\n\n```\ncompose = odcs.new_compose(sources, source_type)\n```\n\nBoth `sources` and `source_type` are strings. Depending on `source_type` value, the `sources` have following meaning:\n\n| `source_type` | `source` |\n|---------------|----------|\n| tag | Name of Koji tag to take RPMs from. |\n| module | White-space separated NAME:STREAM or NAME:STREAM:VERSION of modules to include in compose. |\n| pulp | White-space separated list of content-sets or repository ids. Repositories will be included in a compose. |\n| raw_config | String in `name#commit` hash format. The `name` must match one of the raw config locations defined in ODCS server config as `raw_config_urls`. The `commit` is commit hash defining the version of raw config to use. This config is then used as input config for Pungi. |\n| build | Source should be omitted in the request. The list of Koji builds included in a compose is defined by `builds` attribute. |\n| pungi_compose | URL to variant repository of external compose generated by the Pungi. For example https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Server/. The generated compose will contain the same set of RPMs as the given external compose variant. The packages will be taken from the configured Koji instance. |\n\nThere are also additional optional attributes you can pass to `new_compose(...)` method:\n\n- `seconds_to_live` - Number of seconds after which the generated compose should expire and will be removed.\n- `packages` - List of packages which should be included in a compose. This is used only when `source_type` is set to `tag` or `build` to further limit the compose repository.\nIf the `packages` is not set, all packages in Koji tag or all packages in a `builds` list will be included in a final compose.\n- `flags` - List of flags to further modify the compose output:\n - `no_deps` - For `tag` `source_type`, do not resolve dependencies between packages and include only packages listed in the `packages` in the compose. For `module` `source_type`, do not resolve dependencies between modules and include only the requested module in the compose.\n - `include_unpublished_pulp_repos` - For `pulp` `source_type`, include also unpublished repositories for input content-sets.\n - `check_deps` - When set, abort the compose when some package has broken dependencies.\n - `no_reuse` - When set, do not try to reuse old compose.\n - `ignore_absent_pulp_repos` - For `pulp` `source_type`, ignore any content set that does not exist in Pulp\n - `use_only_compatible_arch` - For `pulp` `source_type` only. When this flag is set, architecture hardcoded in URL returned from Pulp will be replaced with `$basearch` variable. The repository definition will also define `skip_if_unavailable = 1`. This could be useful when multiple content sets are included in the repofile to completly ignore packages from repositories for incompatible architectures.\n- `sigkeys` - List of signature keys IDs. Only packages signed by one of these keys will be included in a compose. If there is no signed version of a package, compose will fail. It is also possible to pass an empty-string in a list meaning unsigned packages are allowed. For example if you want to prefer packages signed by key with ID `123` and also allow unsigned packages to appear in a compose, you can do it by setting sigkeys to `[\"123\", \"\"]`.\n- `results` - List of additional results which will be generated as part of a compose. Valid keys are:\n - `iso` - Generates non-installable ISO files with RPMs from a compose.\n - `boot.iso` - Generates `images/boot.iso` file which is needed to build base container images from resulting compose.\n- `arches` - List of additional Koji arches to build this compose for. By default, the compose is built only for \"x86_64\" arch.\n- `multilib_arches` - Subset of `arches` for which the multilib should be enabled. For each architecture in the `multilib_arches` list, ODCS will include also packages from other compatible architectures in a compose. For example when \"x86_64\" is included `multilib_arches`, ODCS will include also \"i686\" packages in a compose. The set of packages included in a composes is influenced by `multilib_method` list.\n- `multilib_method` - List defining the method used to determine whether consider package as multilib. Defaults to empty list. The list can have following values:\n - `iso` - Generates non-installable ISO files with RPMs from a compose.\n - `runtime` - Packages whose name ends with \"-devel\" or \"-static\" suffix will be considered as multilib.\n - `devel` - Packages that install some shared object file \"*.so.*\" will be considered as multilib.\n - `all` - All pakages will be considered as multilib.\n- `builds` - List of NVRs defining the Koji builds to include in a compose. Only valid for `tag` and `build` source types. For `tag` source type, the NVRs will be considered\nfor inclusion in a compose on top of Koji tag defined by `source`. For `build` source type, only the Koji builds defined by the NVRs will be considered for inclusion. The `packages` still need to be set to include particular packages from the Koji builds in a compose.\n- `lookaside_repos` - List of URLs pointing to RPM repositories with packages which will be used by internal dependency resolver to resolve dependencies. Packages from these repositories will not appear in the resulting ODCS compose, but they are considered while checking whether the RPM dependencies are satisfied in the resulting compose when `check_deps` ODCS flag.\n- `module_defaults_url` - List with URL to git repository with Module defaults data and the branch name or commit hash. For example [\"https://pagure.io/releng/fedora-module-defaults.git\", \"master\"]. This is used only when creating modular compose including non-modular RPMs.\n- `modular_koji_tags` - List of Koji tags in which the modular Koji Content Generator builds are tagged. Such builds will be included in a compose.\n\nThe `new_compose` method returns `dict` object describing the compose, for example:\n\n```\n{\n \"arches\": \"x86_64 ppc64\",\n \"flags\": [\n \"no_deps\"\n ],\n \"id\": 1,\n \"owner\": \"jkaluza\",\n \"packages\": \"gofer-package\",\n \"removed_by\": null,\n \"result_repo\": \"https://odcs.fedoraproject.org/composes/latest-odcs-1-1/compose/Temporary\",\n \"result_repofile\": \"https://odcs.fedoraproject.org/composes/latest-odcs-1-1/compose/Temporary/odcs-1.repo\",\n \"results\": [\n \"repository\"\n ],\n \"sigkeys\": \"\",\n \"source\": \"f26\",\n \"source_type\": 1,\n \"state\": 3,\n \"state_name\": \"wait\",\n \"time_done\": \"2017-10-13T17:03:13Z\",\n \"time_removed\": \"2017-10-14T17:00:00Z\",\n \"time_submitted\": \"2017-10-13T16:59:51Z\",\n \"time_to_expire\": \"2017-10-14T16:59:51Z\"\n}\n```\n\nThe most useful data there is `result_repofile`, which points to the .repo file with URLs for generated compose. Another very important data there is the `state` and `state_name` field. There are following states of a compose:\n\n| `state` | `state_name` | Description |\n|---------|--------------|-------------|\n| 0 | wait | Compose is waiting in a queue to be generated |\n| 1 | generating | Compose is being generated |\n| 2 | done | Compose is generated - done |\n| 3 | removed | Compose has expired and is removed |\n| 4 | failed | Compose generation has failed |\n\nAs you can see in our example, compose is in `wait` state and therefore we have to wait until the ODCS generates the compose.\n\n### Waiting until the compose is generated\n\nThere are two ways how to wait for the compose generation. The preferred one is listening on Fedora messaging bus for `odcs.state.change` message with `done` or `failed` state and another one is using HTTP polling implemented in `wait_for_compose(...)` method.\n\nIf your application does not allow listening on the bus for some reason, you can use `wait_for_compose(...)` method like this:\n\n```\ncompose = odcs.new_compose(sources, source_type)\n\n# Blocks until the compose is ready, but maximally for 600 seconds.\ncompose = odcs.wait_for_compose(compose[\"id\"], timeout=600)\n\nif compose[\"state_name\"] == \"done\":\n print \"Compose done, URL with repo file\", compose[\"result_repofile\"]\nelse:\n print \"Failed to generate compose\"\n```\n\n### Checking the state of existing ODCS compose\n\nOnce you have the compose ready, you might want to check its state later. This can be done using the `get_compose(...)` method like this:\n\n```\ncompose = odcs.get_compose(compose[\"id\"])\n```\n\n### Renewing the compose\n\nIf the `time_to_expire` for your compose is getting closer and you know you want to continue using the compose, you can increase the `time_to_expire` using the `renew_compose(...)` method. You can also use this method to regenerate an expired compose in the `removed` state. Such compose will have the same versions of packages as the original compose.\n\n```\ncompose = odcs.renew_compose(compose[\"id\"])\n```\n\n## Development\n\n### Code Convention\n\nThe code must be well formatted via ``black`` and pass ``flake8`` checking.\n\nRun ``tox -e black,flake8`` to do the check.\n\n### Unit-testing\n\nInstall packages required by pip to compile some python packages:\n\n```\n$ sudo dnf install -y gcc swig redhat-rpm-config python-devel openssl-devel openldap-devel \\\n zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel tk-devel \\\n git python3-cairo-devel cairo-gobject-devel gobject-introspection-devel\n```\n\nA lot of these dependencies come from the module\n[pygobject](https://pygobject.readthedocs.io/en/latest/devguide/dev_environ.html#fedora-dep).\n\nRun the tests:\n\n```\n$ make check\n```\n\n### Testing local composes from plain RPM repositories\n\nYou can test ODCS by generating compose from the `./server/tests/repo` repository using following commands:\n\n```\n$ ./create_sqlite_db\n$ ./start_odcs_from_here\n```\n\nBefore executing the command `start_odcs_from_here`, a messaging broker supporting AMQP protocol is required to run locally as well in order to run asynchronous tasks to generate composes. Here is an example to run a RabbitMQ broker:\n\n```\nsudo dnf install -y rabbitmq-server\nsudo systemctl start rabbitmq-server\n```\n\nAdd the `repo` source type to the server configuration in `./server/odcs/server/config.py`. (This will cause some tests to fail, so it needs to be reverted back after you are done with your changes!)\n\nAnd in another terminal, submit a request to frontend:\n\n```\n$ ./submit_test_compose repo `pwd`/server/tests/repo ed\n{\n \"id\": 1,\n \"owner\": \"Unknown\",\n \"result_repo\": null,\n \"source\": \"/home/hanzz/code/fedora-modularization/odcs/tests/repo\",\n \"source_type\": 3,\n \"state\": 0,\n \"state_name\": \"wait\",\n \"time_done\": null,\n \"time_removed\": null,\n \"time_submitted\": \"2017-06-12T14:18:19Z\"\n}\n```\n\nYou should then see the backend process generating the compose and once it's done, the resulting compose in `./test_composes/latest-Unknown-1/compose/Temporary` directory.\n\n### Using docker-compose for creating a local setup\n\nYou can create test setup for ODCS with the docker-compose file. This yaml file creates docker container for the backend and frontend setup of ODCS and run multiple services together. These services are;\n\n* **rabbitmq** (Handles the communication between backend and frontend)\n* **postgres** (Creates the database where the localy generated composes are stored)\n* **backend** (Backend service of ODCS)\n* **frontend** (Frontend service of ODCS that handles the REST API)\n* **static** (Apache service for making storage available)\n* **beat** (Cronjob for backend to check the service status)\n\nIn addition to these, there are also three docker volumes (`odcs_odcs-composes`, `odcs_odcs-postgres` and `odcs_rabbitmq`) are created. These are providing persistent storage for the services.\n\nThis yaml file requires also an **.env** file that specified some enviroment variables for the configuration of frontend and backend. Such an **.env** file should be in the same path with the **docker-compose.yml** file and here are the necessary variables that need to be specified in this file;\n```\n# Pyhton path\nPYTHONPATH=/src/common:/src/server:/src/client\n\n# POSTGRES\nPOSTGRES_USER=odcs\nPOSTGRES_PASSWORD=password\n\n# PULP CONFIGURATION\nPULP_SERVER_URL=<URL of the pulp server>\nPULP_USERNAME=<Username for the pulp server>\nPULP_PASSWORD=<Credentials for the pulp server>\nRAW_CONFIG_URLS=<Raw config settings in JSON format>\n\n# ODCS CONFIGURATION\nODCS_CONFIG_SECTION=DevConfiguration\nODCS_CONFIG_DIR=/src/server/conf/\nODCS_CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/\nODCS_DB_URL=postgresql+psycopg2://odcs:password@postgres/odcs\n# Directory where the generated composes are stored. This hast to match\n# location where odcs-composes volume is mounted in frontend and backend\n# containers.\nODCS_TARGET_DIR=/mnt/odcs\nTARGET_DIR_URL=http://localhost:8080\n\n# FLASK SETTINGS\n# Force flask to reload application on change of source files.\nFLASK_ENV=development\n```\n\nThe services can be start with `sudo docker-compose up` command. If you have face an error or something that does not work correctly please use the following steps;\n\n1. Check whether there is already created docker volume.\n\n\n $ sudo docker volume ls\n Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.\n DRIVER VOLUME NAME\n\n\n If the output is like above, it means there isn't any volume yet and it is sufficient to run `sudo docker-compose up` and then `sudo docker-compose down` command. This creates the volumes and if you run the above command again the output becomes as follows;\n\n\n $ sudo docker volume ls\n Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.\n DRIVER VOLUME NAME\n local odcs_odcs-composes\n local odcs_odcs-postgres\n local odcs_odcs-rabbitmq\n\n\n2. After this point we need to set the correct permission to `ocds_odcs-composes` volume. In order to find where is the actual location of the volume type the following\n\n\n $ sudo docker volume inspect odcs_odcs-composes\n Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.\n [\n {\n \"Name\": \"odcs_odcs-composes\",\n \"Driver\": \"local\",\n \"Mountpoint\": \"/var/lib/containers/storage/volumes/odcs_odcs-composes/_data\",\n \"CreatedAt\": \"2021-10-04T13:19:36.478636851+02:00\",\n \"Labels\": {\n \"io.podman.compose.project\": \"odcs\"\n },\n \"Scope\": \"local\",\n \"Options\": {}\n }\n ]\n\n `\"Mountpoint\": \"/var/lib/containers/storage/volumes/odcs_odcs-composes/_data\"` shows the exact location of the corresponding volume.\n\n Add group write permission to the Mountpoint by\n\n\n $ sudo chmod 775 /var/lib/containers/storage/volumes/odcs_odcs-composes/_data\n\n\n3. In this step it is sufficient to run `sudo docker-compose up` command to start the services properly.\n\nHere are some REST calls for checking the ODCS.\n\n* `$ curl -s http://localhost:5000/api/1/composes/ | jq .` This call shows all the composes in db.\n* `$ odcs --server http://localhost:5000/ create-pulp <Pulp content set>` This call starts a pulp compose that match the given pulp content set\n* `$ odcs --server http://localhost:5000/ create-raw-config --compose-type test my_raw_config master` This call starts a compose with the configuration defined as my_raw_config\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "On Demand Compose Service",
"version": "0.8.0",
"project_urls": {
"Homepage": "https://pagure.io/odcs/"
},
"split_keywords": [
"on",
"demand",
"compose",
"service",
"modularity",
"fedora"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2cd8f5cad90fc6db4a0a26bf072a504a6badf58e411eb568952d7faf864f0bc2",
"md5": "db301e7310bf4d28d8852c93e6579999",
"sha256": "2910f002acc52f851c761798bb4448daaa0ced85b80a448891ac171a0b016c8a"
},
"downloads": -1,
"filename": "odcs-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "db301e7310bf4d28d8852c93e6579999",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 425496,
"upload_time": "2024-01-23T02:25:44",
"upload_time_iso_8601": "2024-01-23T02:25:44.429772Z",
"url": "https://files.pythonhosted.org/packages/2c/d8/f5cad90fc6db4a0a26bf072a504a6badf58e411eb568952d7faf864f0bc2/odcs-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-23 02:25:44",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "odcs"
}
JSON