The `tc-admin` library supports administration of the runtime configuration of a Taskcluster deployment.
This means creation and maintenance of resources such as roles, hooks, and worker pools, based on version-controlled specifications..
# Usage
This library is used as a dependency of a Python application containing code and configuration specific to the taskcluster deployment(s) being administered.
The project should contain a `tc-admin.py` that serves to define the application configuration.
Assuming that is in place, the tool is easy to use:
After installing the app, run `tc-admin generate` to generate the expected set of resources (use `--json` to get JSON output).
This will require `TASKCLUSTER_ROOT_URL` to be set in the environment, to know which deployment to talk to.
Similarly, `tc-admin current` will generate the current set of resources (optionally with `--json`).
To compare them, run `tc-admin diff`.
If the configuration includes secrets, you may want to pass the `--without-secrets` option.
This option skips managing the content of secrets, and thus needs neither access to secret values nor Taskcluster credentials to fetch secrets.
Run `tc-admin apply` to apply the changes.
Note that only `apply` will require Taskcluster credentials, and it's a good practice to only set TC credentials when running this command.
See `tc-admin <command> --help` for more useful options.
## Checks
Checks are a way to double-check that purpose-specific invariants are satisfied in a Taskcluster deployment.
For example, it may be important to check that only specific repository roles have scopes to create tasks in critical worker pools.
Checks are defined as normal Python tests, and have access to the current and generated configurations.
If the app has checks set up, then `tc-admin check` will run those checks.
# Quick Guide to Library Operation
The operation of this tool is pretty simple: it generates a set of expected Taskcluster resources, downloads existing resources from the Taskcluster API, and compares them.
A collection of resources also specifies the set of "managed" resources -- this allows deletion of resources that are no longer expected, without risk of deleting *everything* in the Taskcluster API.
After generation, resources can be "modified".
This is typically used to make minor changes to resources depending on environment.
For exmaple, in a staging environment, hook schedules might be removed.
# App Configuration
## `tc-admin` and `tc-admin.py`
A tc-admin app is configured by `tc-admin.py`.
This is a Python file which is responsible for creating and customizing an `AppConfig` object.
```python
from tcadmin.appconfig import AppConfig
appconfig = AppConfig()
# .. customize
```
The `tc-admin` command looks for `tc-admin.py` in the current directory, or in the directory given by `$TC_ADMIN_PY` or command-line argument `--tc-admin-py`.
Like most Python modules, the global `__file__` is set when `tc-admin.py` is executed, and can be used to determine relative paths.
Before `tc-admin.py` is executed, the current working directory is changed to the directory containing it.
This enables relative imports as well as loading files with relative paths (such as with LocalLoader, below).
## Programmatic Interface
This library can also be used programmatically.
Simply create an AppConfig object directly and call `tcadmin.main.main` with it:
```python
from tcadmin.appconfig import AppConfig
from tcadmin.main import main
def boot():
appconfig = AppConfig()
# .. customize
main(appconfig)
if __name__ == "__main__":
boot()
```
Note that the current directory is not automatically set in this case.
## AppConfig
The AppConfig object contains a number of properties that can be customized, described below.
During execution, the current AppConfig object is available from `AppConfig.current()`.
This can be useful when generators or modifiers are defined in separate Python modules.
### Generators
Generators generate expected Taskcluster resources and defined the managed resource names.
Each generator is an async function that is given a `Resources` object and is expected to call `resources.manage` and `resources.update`.
Generators are registered with `appconfig.generators.register`, most easily with a decorator:
```python
@appconfig.generators.register
async def update_resources(resources):
# modify in place ...
```
When generating secrets, respect the `--with-secrets` option, and generate secrets without values when it is false.
You can also use this option to determine whether the generation process requires access to secret values.
This allows generation runs with `--without-secrets` to occur without any credentials or access to secret values.
```python
@appconfig.generators.register
async def update_resources(resources):
# modify in place ...
if appconfig.options.get('--with-secrets'):
secretstore = load_secret_values()
resources.add(Secret(
name="top-secret/cookie-recipe",
secret=secretstore.decrypt('recipes/double-chocolate-chip')))
else:
resources.add(Secret(name="top-secret/cookie-recipe"))
```
### Modifiers
Modifiers are responsible for modifying an existing set of resources.
Since resources are immutable, the signature differs slightly from generators:
```python
@appconfig.modifiers.register
async def modify_resources(resources):
# return new set of resources
return resources.map(..)
```
Modifiers are called sequentially in the order in which they were registered.
### Callbacks
Callbacks are external function from your own application that can be executed at specific times during the `tc-admin apply` execution:
* A `before_apply` callback will run before a resource is created, updated or deleted,
* A `after_apply` callback will run after a resource has beencreated, updated or deleted.
Supported actions are :
* create
* update
* delete
By default all actions are used.
All resources are supported by callbacks, and enabled by default. If you want to limit your callback to some resources, you need to specify them using their class (not a string).
You can declare your callbacks as:
```python
from tcadmin.resources import Secret
async def my_action(action, resource):
print("Got a callback on", action, resource)
# Will call your function when a secret has been updated or deleted
appconfig.callbacks.add("after_apply", my_action, actions=["update", "delete"], resources=[Secret, ])
```
### Command-Line Options
Apps can add additional command-line options, the values of which are then available during resource generation.
To register an option, call `appconfig.options.add`, with the full option name and any of the following keyword options:
* `required` - if True, the option is required
* `help` - help string to be shown in `tc-admin generate --help`
* `default` - default value for the option
To retrieve the option value during generation, call `appconfig.options.get(name)`.
All together, then:
```python
appconfig.options.add("--branch", help="configuration branch to read from")
@appconfig.generators.register
async def update_resources(resources):
branch = appconfig.options.get("--branch")
# ...
```
As a special case, the `--with-secrets` secret is available through this same mechanism.
### Checks
The `appconfig.check_path` property gives the path of the checks to run for `tc-admin check`, relative to the current directory.
This directory is a "normal" pytest directory.
To help distinguish checks from tests, include a `pytest.ini` in this directory:
```ini
[pytest]
python_classes = Check*
python_files = check_*.py
python_functions = check_*
```
### description_prefix
The `appconfig.description_prefix` property allows the users to customize the prefix of the description.
This can be customized in the `tc-admin.py` as follows:
```python
from tcadmin.appconfig import AppConfig
appconfig = AppConfig()
appconfig.description_prefix = "YOUR_CUSTOM_PREFIX"
```
The DEFAULT value of the description_prefix is `*DO NOT EDIT* - This resource is configured automatically.\n\n`
### Root URL
For the common case of a configuration that applies to only one Taskcluster deployment, specify that deployment's root URL in `tc-admin.py`:
```python
from tcadmin.appconfig import AppConfig
appconfig = AppConfig()
appconfig.root_url = "https://taskcluster.example.com"
```
To support more complex cases, this value can also be an async callable.
It will be invoked once, after the `click` options have been processed, so it can access `appconfig.options` if necessary.
The current root URL is available from an async helper function:
```python
from tcadmin.util.root_url import root_url
async def foo():
print(await root_url())
```
This will retrieve the value from the AppConfig or, if that is not set, from `TASKCLUSTER_ROOT_URL`.
If both are set, and the values do not match, it will produce an error message.
### Loading Config Sources
Most uses of this library load configuration data from some easily-modified YAML files.
The `tcadmin.util.config` package provides some support for loading and parsing these files.
All of this is entirely optional; use what is appropriate to the purpose.
#### Loaders
First, define a loader that can load data from files.
```python
from tcadmin.util.config import LocalLoader
loader = LocalLoader()
```
The LocalLoader class knows how to load configuration from files relative to `tc-admin.py`.
It has a `load` method that will load data, optionally parsing it as YAML:
```python
data = loader.load("data.bin")
aliases = await loader.load("aliases.yml", parse="yaml")
```
You can also define your own loader class.
Just implement the `load_raw` method to return bytes, given a filename.
#### Config
YAML data is inconvenient to deal with in Python, introducig a lot of `[".."]` noise.
Commonly, config files are either a top-level array, or a top-level object with named "stanzas" of configuration.
The ConfigList and ConfigDict classes support these formats.
We suggest using these with the Python `attrs` library.
Define a class that inherits from either of these classes, specifies the filename to load from, and has an `Item` class for the items in the collection:
```python
from tcadmin.util.config import ConfigList
class Workers(ConfigList):
filename = "workers.yml"
@attr.s
class Item:
workerId = attr.ib(type=str)
bigness = attr.ib(type=int, default=1)
```
Then simply call `await Workers.load(loader)` to load a `workers.yml` that looks something like
```yaml
- workerId: small
bigness: 5
- workerId: huge
bigness: 5000
```
The ConfigDict class is similar, but parses files like
```yaml
small:
bigness: 5
huge:
bigness: 5000
```
ConfigList creates new `Item` instances from array elements `item` with `Item(**item)`.
ConfigDict creates new `Item` instances from `k: item` with `Item(k, **item)`.
This approach is compatible with `attrs`, where in the latter case `k` should be the first attribute defined.
If array elements or object values are not themselves YAML objects, add a class method named `transform_item` to transform the data in the YAML file into a Python dictionary.
For example:
```python
class Workers(ConfigList):
@classmethod
def transform_item(cls, item):
# given a simple string, assume that is the workerId and apply defaults
if isinstance(item, str):
return {"workerId": item}
return item
...
```
## Resources
The `tcadmin.resources` package contains clasess for defining Taskcluster resources and collections.
```python
from tcadmin.resources import Resources
```
The `Resources` class defines a collection of resources and tracks what resources are managed.
Resources found in the Taskcluster deployment that match the "managed" patterns but are not generated will be deleted on `apply`.
The class has the following methods:
* `resources.add(resource)` - add a resource to the collection. The resource must be managed.
* `resources.update(iterable)` - add an iterable full of resources to the collection. All resources must be managed.
* `resources.manage(pattern)` - consider reources matching regular expression string `pattern` to be managed
* `resources.is_managed(id)` - return true if the given resource is managed
* `resources.filter(pattern)` - return a new Resources object containing only resources matching the given regular expression string
* `resources.map(functor)` - return a new Resources object, with fuctor applied to each resource. This is typically used in modifiers.
Resources must be unique -- tc-admin cannot manage multiple hooks with the same name, for example.
However, some resource kinds support merging, where adding a resource with the same identity as one that already exists "merges" it into the existing resource.
See the description of roles, below.
The remaining classes represent individual resources.
Each has an `id` formed from its kind and the unique identifier for the resource in the Taskcluster.
For example, `Hook=release-hooks/beta-release`.
Resources are immutable once created, but can be "evolved" (returning a new resource) with `rsrc.evolve(**updates)`.
Resources with descriptions automatically prepend a "DO NOT EDIT" prefix to dissuade users from editing them in the Taskcluster UI.
### Hook
```python
from tcadmin.resources import Hook, Binding
hook = Hook(
hookGroupId=..,
hookId=..,
name=..,
description=..,
owner=..,
emailOnError=..,
schedule=(.., ..),
bindings=(.., ..),
task={..},
triggerSchema={..})
```
Most of these fields correspond directly to the Taskcluster definition.
Both `schedule` and `bindings` must be tuples, not lists (as lists are mutable).
The items in `schedule` are cron-like strings.
The items in `bindings` are instances of `Binding(exchange=.., routingKeyPattern=..)`.
### Secret
```python
from tcadmin.resources import Secret
secret = Secret(
name=..,
secret=..)
# or, when not managing secret values
secret = Secret(name=..)
```
Secrets are managed using the Secret resource type.
While Taskcluster supports expiration times on secrets, this library sets those times the far future, effectively creating non-expiring secrets
This library is careful to not display secret values in its output.
Instead, it displays `<unknown>` when not managing secret values, and displays a salted hash of the secret value when managing secret values.
The salted hash allows `tc-admin diff` to show that a secret value has changed, without revealing the value of that secret.
The salt includes a per-run salt, and the name of the secret, with the result that even if two secrets have the same value, they will be shown with different hashes in `tc-admin generate`.
### Role
```python
from tcadmin.resources import Role
role = Role(
roleId=..,
description=..,
scopes=(.., ..))
```
As with hooks, `scopes` must be a tuple (not a list) of strings.
Roles can be merged if their descriptions match.
The resulting role contains the union of the scopes of the input roles.
This functionality makes management of roles easier in cases where different parts of the generation process may add scopes to the same role.
For example:
```python
resources.add(Role(roleId="my-role", description="My Role", scopes=["scope1"]))
resources.add(Role(roleId="my-role", description="My Role", scopes=["scope2"]))
```
This will result in a single Role with scopes `["scope1", "scope2"]`.
### Client
```python
from tcadmin.resources import Client
client = Client(
clientId=..,
description=..,
scopes=(.., ..))
```
Clients work much like roles.
As with roles, `scopes` must be a tuple (not a list) of strings.
This library does not manage access tokens: it discards them from the response to `auth.createClient`.
The expectation is that project admins who need credentials for the managed clients will call `auth.resetAccessToken` and use the returned token.
Clients configured by this library have an expiration date far in the future.
Like roles, the clients managed here last "forever".
### WorkerPool
```python
from tcadmin.resources import WorkerPool
workerPool = WorkerPool(
workerPoolId=..,
providerId=..,
description=..,
owner=..,
config={..},
emailOnError=..)
```
All attributes of this class match the Taskcluster definition.
## Utiliites
The library provides a number of utilities for common application requirements.
*NOTE*: only functions described in this README are considered stable.
Other functions defined by the library may change without notice.
### Scopes
As an aid to writing checks, tc-admin supplies local implementations of various scope-related algorithms.
```python
from tcadmin.util.scopes import satisfies, normalizeScopes, Resolver
```
The `satisfies` function determines scope satisfaction, without any expansion.
Satisfaction means that the first argument contains all scopes in the second argument.
```python
assert satisfies(['balloons:*', 'cake:birthday'], ['baloons:mylar:happy-birthday'])
```
The `normalizeScopes` function normalizes a scopeset, removing redundant scopes and sorting.
```python
assert normalizedScopes(['balloons:*', 'balloons:mylar:*']) == ['baloons:*']
```
Finally, `Resolver` can perform scope expansion.
It is initialized with a dictionary mapping roleIds to scope lists.
Alternately, it can be initialized from a `Resources` instance using `Resolver.from_resources(resources)`.
Its `expandScopes` method behaves identically to the remote call `auth.expandScopes`.
```python
resolver = Resolver.from_resources(resources)
assert resolver.expandScopes(['assume:clown:grimaldi']) == ['assume:clown:grimaldi', 'ruffle:full']
```
### aiohttp session
The library uses `aiohttp` to communicate with Taskcluster, and establishes a single session for efficiency.
Applications can use the same session for any other HTTP operations.
```python
from tcadmin.util.session import aiohttp_session
async def foo():
# ...
async with aiohttp_session().get(url) as response:
response.raise_for_status()
result = await response.read()
```
Tests and checks can set this value using `with_aiohttp_session`:
```python
from tcadmin.util.sessions import with_aiohttp_session
import pytest
@pytest.mark.asyncio
@with_aiohttp_session
async def test_something():
# ...
```
### MatchList
A MatchList is a list of regular expressions that can determine whether a given
string matches one of those patterns. Patterns are rooted at the left, but
should use `$` where required to match the end of the string.
```python
from tcadmin.util.matchlist import MatchList
ml = MatchList()
ml.add("fo+$")
ml.add("ba+r$")
assert ml.matches("foo")
```
This functionality is used to track managed resources, but may be useful otherwise.
# Development
To install for development, in a virtualenv:
```
pip install -e [path]
```
And to run flake8 and tests:
```
python setup.py flake8
python setup.py test
```
The library uses [Black](https://black.readthedocs.io/en/stable/) to format code.
```
pip install black
black tcadmin
```
## Releasing
To release:
* update version in `setup.py` and `git commit -m "vX.Y.Z"`
* `git tag vX.Y.z`
* push those changes to `main`
* `./release.sh --real` and enter your pypi credentials when prompted (omit the `--real` to try it against the testing pypi, if you're not sure)
* Find the tag in https://github.com/taskcluster/tc-admin/releases and create a new release with a brief description of the changes
## Docker image
To build and publish new version of tc-admin docker image run:
```bash
docker build -t tc-admin:X.Y.Z .
docker push tc-admin:X.Y.Z
```
Raw data
{
"_id": null,
"home_page": "https://github.com/taskcluster/tc-admin",
"name": "tc-admin",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Dustin Mitchell",
"author_email": "dustin@mozilla.com",
"download_url": "https://files.pythonhosted.org/packages/93/b7/72e08e6d077567ab39a8729a9905e1e351b9cee78235574b3b787b7f8f1f/tc_admin-4.0.1.tar.gz",
"platform": null,
"description": "The `tc-admin` library supports administration of the runtime configuration of a Taskcluster deployment.\nThis means creation and maintenance of resources such as roles, hooks, and worker pools, based on version-controlled specifications..\n\n# Usage\n\nThis library is used as a dependency of a Python application containing code and configuration specific to the taskcluster deployment(s) being administered.\nThe project should contain a `tc-admin.py` that serves to define the application configuration.\n\nAssuming that is in place, the tool is easy to use:\n\nAfter installing the app, run `tc-admin generate` to generate the expected set of resources (use `--json` to get JSON output).\nThis will require `TASKCLUSTER_ROOT_URL` to be set in the environment, to know which deployment to talk to.\nSimilarly, `tc-admin current` will generate the current set of resources (optionally with `--json`).\nTo compare them, run `tc-admin diff`.\n\nIf the configuration includes secrets, you may want to pass the `--without-secrets` option.\nThis option skips managing the content of secrets, and thus needs neither access to secret values nor Taskcluster credentials to fetch secrets.\n\nRun `tc-admin apply` to apply the changes.\nNote that only `apply` will require Taskcluster credentials, and it's a good practice to only set TC credentials when running this command.\n\nSee `tc-admin <command> --help` for more useful options.\n\n## Checks\n\nChecks are a way to double-check that purpose-specific invariants are satisfied in a Taskcluster deployment.\nFor example, it may be important to check that only specific repository roles have scopes to create tasks in critical worker pools.\nChecks are defined as normal Python tests, and have access to the current and generated configurations.\n\nIf the app has checks set up, then `tc-admin check` will run those checks.\n\n# Quick Guide to Library Operation\n\nThe operation of this tool is pretty simple: it generates a set of expected Taskcluster resources, downloads existing resources from the Taskcluster API, and compares them.\nA collection of resources also specifies the set of \"managed\" resources -- this allows deletion of resources that are no longer expected, without risk of deleting *everything* in the Taskcluster API.\n\nAfter generation, resources can be \"modified\".\nThis is typically used to make minor changes to resources depending on environment.\nFor exmaple, in a staging environment, hook schedules might be removed.\n\n# App Configuration\n\n## `tc-admin` and `tc-admin.py`\n\nA tc-admin app is configured by `tc-admin.py`.\nThis is a Python file which is responsible for creating and customizing an `AppConfig` object.\n\n```python\nfrom tcadmin.appconfig import AppConfig\n\nappconfig = AppConfig()\n# .. customize\n```\n\nThe `tc-admin` command looks for `tc-admin.py` in the current directory, or in the directory given by `$TC_ADMIN_PY` or command-line argument `--tc-admin-py`.\nLike most Python modules, the global `__file__` is set when `tc-admin.py` is executed, and can be used to determine relative paths.\n\nBefore `tc-admin.py` is executed, the current working directory is changed to the directory containing it.\nThis enables relative imports as well as loading files with relative paths (such as with LocalLoader, below).\n\n## Programmatic Interface\n\nThis library can also be used programmatically.\nSimply create an AppConfig object directly and call `tcadmin.main.main` with it:\n\n```python\nfrom tcadmin.appconfig import AppConfig\nfrom tcadmin.main import main\n\ndef boot():\n appconfig = AppConfig()\n # .. customize\n main(appconfig)\n\nif __name__ == \"__main__\":\n boot()\n```\n\nNote that the current directory is not automatically set in this case.\n\n## AppConfig\n\nThe AppConfig object contains a number of properties that can be customized, described below.\nDuring execution, the current AppConfig object is available from `AppConfig.current()`.\nThis can be useful when generators or modifiers are defined in separate Python modules.\n\n### Generators\n\nGenerators generate expected Taskcluster resources and defined the managed resource names.\nEach generator is an async function that is given a `Resources` object and is expected to call `resources.manage` and `resources.update`.\n\nGenerators are registered with `appconfig.generators.register`, most easily with a decorator:\n\n```python\n@appconfig.generators.register\nasync def update_resources(resources):\n # modify in place ...\n```\n\nWhen generating secrets, respect the `--with-secrets` option, and generate secrets without values when it is false.\nYou can also use this option to determine whether the generation process requires access to secret values.\nThis allows generation runs with `--without-secrets` to occur without any credentials or access to secret values.\n\n```python\n@appconfig.generators.register\nasync def update_resources(resources):\n # modify in place ...\n if appconfig.options.get('--with-secrets'):\n secretstore = load_secret_values()\n resources.add(Secret(\n name=\"top-secret/cookie-recipe\",\n secret=secretstore.decrypt('recipes/double-chocolate-chip')))\n else:\n resources.add(Secret(name=\"top-secret/cookie-recipe\"))\n```\n\n### Modifiers\n\nModifiers are responsible for modifying an existing set of resources.\nSince resources are immutable, the signature differs slightly from generators:\n\n```python\n@appconfig.modifiers.register\nasync def modify_resources(resources):\n # return new set of resources\n return resources.map(..)\n```\n\nModifiers are called sequentially in the order in which they were registered.\n\n### Callbacks\n\nCallbacks are external function from your own application that can be executed at specific times during the `tc-admin apply` execution:\n\n* A `before_apply` callback will run before a resource is created, updated or deleted,\n* A `after_apply` callback will run after a resource has beencreated, updated or deleted.\n\nSupported actions are :\n\n* create\n* update\n* delete\n\nBy default all actions are used.\n\nAll resources are supported by callbacks, and enabled by default. If you want to limit your callback to some resources, you need to specify them using their class (not a string).\n\nYou can declare your callbacks as:\n\n```python\nfrom tcadmin.resources import Secret\n\nasync def my_action(action, resource):\n print(\"Got a callback on\", action, resource)\n\n# Will call your function when a secret has been updated or deleted\nappconfig.callbacks.add(\"after_apply\", my_action, actions=[\"update\", \"delete\"], resources=[Secret, ])\n```\n\n### Command-Line Options\n\nApps can add additional command-line options, the values of which are then available during resource generation.\n\nTo register an option, call `appconfig.options.add`, with the full option name and any of the following keyword options:\n * `required` - if True, the option is required\n * `help` - help string to be shown in `tc-admin generate --help`\n * `default` - default value for the option\n\nTo retrieve the option value during generation, call `appconfig.options.get(name)`.\nAll together, then:\n\n```python\nappconfig.options.add(\"--branch\", help=\"configuration branch to read from\")\n\n@appconfig.generators.register\nasync def update_resources(resources):\n branch = appconfig.options.get(\"--branch\")\n # ...\n```\n\nAs a special case, the `--with-secrets` secret is available through this same mechanism.\n\n### Checks\n\nThe `appconfig.check_path` property gives the path of the checks to run for `tc-admin check`, relative to the current directory.\nThis directory is a \"normal\" pytest directory.\n\nTo help distinguish checks from tests, include a `pytest.ini` in this directory:\n\n```ini\n[pytest]\npython_classes = Check*\npython_files = check_*.py\npython_functions = check_*\n```\n\n### description_prefix\n\nThe `appconfig.description_prefix` property allows the users to customize the prefix of the description.\nThis can be customized in the `tc-admin.py` as follows:\n\n```python\nfrom tcadmin.appconfig import AppConfig\n\nappconfig = AppConfig()\nappconfig.description_prefix = \"YOUR_CUSTOM_PREFIX\"\n```\n\nThe DEFAULT value of the description_prefix is `*DO NOT EDIT* - This resource is configured automatically.\\n\\n`\n\n### Root URL\n\nFor the common case of a configuration that applies to only one Taskcluster deployment, specify that deployment's root URL in `tc-admin.py`:\n\n```python\nfrom tcadmin.appconfig import AppConfig\n\nappconfig = AppConfig()\nappconfig.root_url = \"https://taskcluster.example.com\"\n```\n\nTo support more complex cases, this value can also be an async callable.\nIt will be invoked once, after the `click` options have been processed, so it can access `appconfig.options` if necessary.\n\nThe current root URL is available from an async helper function:\n\n```python\nfrom tcadmin.util.root_url import root_url\n\nasync def foo():\n print(await root_url())\n```\n\nThis will retrieve the value from the AppConfig or, if that is not set, from `TASKCLUSTER_ROOT_URL`.\nIf both are set, and the values do not match, it will produce an error message.\n\n### Loading Config Sources\n\nMost uses of this library load configuration data from some easily-modified YAML files.\nThe `tcadmin.util.config` package provides some support for loading and parsing these files.\nAll of this is entirely optional; use what is appropriate to the purpose.\n\n#### Loaders\n\nFirst, define a loader that can load data from files.\n\n```python\nfrom tcadmin.util.config import LocalLoader\n\nloader = LocalLoader()\n```\n\nThe LocalLoader class knows how to load configuration from files relative to `tc-admin.py`.\nIt has a `load` method that will load data, optionally parsing it as YAML:\n\n```python\ndata = loader.load(\"data.bin\")\naliases = await loader.load(\"aliases.yml\", parse=\"yaml\")\n```\n\nYou can also define your own loader class.\nJust implement the `load_raw` method to return bytes, given a filename.\n\n#### Config\n\nYAML data is inconvenient to deal with in Python, introducig a lot of `[\"..\"]` noise.\nCommonly, config files are either a top-level array, or a top-level object with named \"stanzas\" of configuration.\nThe ConfigList and ConfigDict classes support these formats.\nWe suggest using these with the Python `attrs` library.\n\nDefine a class that inherits from either of these classes, specifies the filename to load from, and has an `Item` class for the items in the collection:\n\n```python\nfrom tcadmin.util.config import ConfigList\n\nclass Workers(ConfigList):\n filename = \"workers.yml\"\n\n @attr.s\n class Item:\n workerId = attr.ib(type=str)\n bigness = attr.ib(type=int, default=1)\n```\n\nThen simply call `await Workers.load(loader)` to load a `workers.yml` that looks something like\n\n```yaml\n- workerId: small\n bigness: 5\n- workerId: huge\n bigness: 5000\n```\n\nThe ConfigDict class is similar, but parses files like\n\n```yaml\nsmall:\n bigness: 5\nhuge:\n bigness: 5000\n```\n\nConfigList creates new `Item` instances from array elements `item` with `Item(**item)`.\nConfigDict creates new `Item` instances from `k: item` with `Item(k, **item)`.\nThis approach is compatible with `attrs`, where in the latter case `k` should be the first attribute defined.\n\nIf array elements or object values are not themselves YAML objects, add a class method named `transform_item` to transform the data in the YAML file into a Python dictionary.\nFor example:\n\n```python\nclass Workers(ConfigList):\n\n @classmethod\n def transform_item(cls, item):\n # given a simple string, assume that is the workerId and apply defaults\n if isinstance(item, str):\n return {\"workerId\": item}\n return item\n\n ...\n```\n\n## Resources\n\nThe `tcadmin.resources` package contains clasess for defining Taskcluster resources and collections.\n\n```python\nfrom tcadmin.resources import Resources\n```\n\nThe `Resources` class defines a collection of resources and tracks what resources are managed.\nResources found in the Taskcluster deployment that match the \"managed\" patterns but are not generated will be deleted on `apply`.\nThe class has the following methods:\n\n* `resources.add(resource)` - add a resource to the collection. The resource must be managed.\n* `resources.update(iterable)` - add an iterable full of resources to the collection. All resources must be managed.\n* `resources.manage(pattern)` - consider reources matching regular expression string `pattern` to be managed\n* `resources.is_managed(id)` - return true if the given resource is managed\n* `resources.filter(pattern)` - return a new Resources object containing only resources matching the given regular expression string\n* `resources.map(functor)` - return a new Resources object, with fuctor applied to each resource. This is typically used in modifiers.\n\nResources must be unique -- tc-admin cannot manage multiple hooks with the same name, for example.\nHowever, some resource kinds support merging, where adding a resource with the same identity as one that already exists \"merges\" it into the existing resource.\nSee the description of roles, below.\n\nThe remaining classes represent individual resources.\nEach has an `id` formed from its kind and the unique identifier for the resource in the Taskcluster.\nFor example, `Hook=release-hooks/beta-release`.\nResources are immutable once created, but can be \"evolved\" (returning a new resource) with `rsrc.evolve(**updates)`.\n\nResources with descriptions automatically prepend a \"DO NOT EDIT\" prefix to dissuade users from editing them in the Taskcluster UI.\n\n### Hook\n\n```python\nfrom tcadmin.resources import Hook, Binding\n\nhook = Hook(\n hookGroupId=..,\n hookId=..,\n name=..,\n description=..,\n owner=..,\n emailOnError=..,\n schedule=(.., ..),\n bindings=(.., ..),\n task={..},\n triggerSchema={..})\n```\n\nMost of these fields correspond directly to the Taskcluster definition.\nBoth `schedule` and `bindings` must be tuples, not lists (as lists are mutable).\nThe items in `schedule` are cron-like strings.\nThe items in `bindings` are instances of `Binding(exchange=.., routingKeyPattern=..)`.\n\n### Secret\n\n```python\nfrom tcadmin.resources import Secret\n\nsecret = Secret(\n name=..,\n secret=..)\n\n# or, when not managing secret values\n\nsecret = Secret(name=..)\n```\n\nSecrets are managed using the Secret resource type.\nWhile Taskcluster supports expiration times on secrets, this library sets those times the far future, effectively creating non-expiring secrets\n\nThis library is careful to not display secret values in its output.\nInstead, it displays `<unknown>` when not managing secret values, and displays a salted hash of the secret value when managing secret values.\nThe salted hash allows `tc-admin diff` to show that a secret value has changed, without revealing the value of that secret.\nThe salt includes a per-run salt, and the name of the secret, with the result that even if two secrets have the same value, they will be shown with different hashes in `tc-admin generate`.\n\n### Role\n\n```python\nfrom tcadmin.resources import Role\n\nrole = Role(\n roleId=..,\n description=..,\n scopes=(.., ..))\n```\n\nAs with hooks, `scopes` must be a tuple (not a list) of strings.\n\nRoles can be merged if their descriptions match.\nThe resulting role contains the union of the scopes of the input roles.\nThis functionality makes management of roles easier in cases where different parts of the generation process may add scopes to the same role.\n\nFor example:\n\n```python\nresources.add(Role(roleId=\"my-role\", description=\"My Role\", scopes=[\"scope1\"]))\nresources.add(Role(roleId=\"my-role\", description=\"My Role\", scopes=[\"scope2\"]))\n```\n\nThis will result in a single Role with scopes `[\"scope1\", \"scope2\"]`.\n\n### Client\n\n```python\nfrom tcadmin.resources import Client\n\nclient = Client(\n clientId=..,\n description=..,\n scopes=(.., ..))\n```\n\nClients work much like roles.\nAs with roles, `scopes` must be a tuple (not a list) of strings.\nThis library does not manage access tokens: it discards them from the response to `auth.createClient`.\nThe expectation is that project admins who need credentials for the managed clients will call `auth.resetAccessToken` and use the returned token.\n\nClients configured by this library have an expiration date far in the future.\nLike roles, the clients managed here last \"forever\".\n\n### WorkerPool\n\n```python\nfrom tcadmin.resources import WorkerPool\n\nworkerPool = WorkerPool(\n workerPoolId=..,\n providerId=..,\n description=..,\n owner=..,\n config={..},\n emailOnError=..)\n```\n\nAll attributes of this class match the Taskcluster definition.\n\n## Utiliites\n\nThe library provides a number of utilities for common application requirements.\n\n*NOTE*: only functions described in this README are considered stable.\nOther functions defined by the library may change without notice.\n\n### Scopes\n\nAs an aid to writing checks, tc-admin supplies local implementations of various scope-related algorithms.\n\n```python\nfrom tcadmin.util.scopes import satisfies, normalizeScopes, Resolver\n```\n\nThe `satisfies` function determines scope satisfaction, without any expansion.\nSatisfaction means that the first argument contains all scopes in the second argument.\n\n```python\nassert satisfies(['balloons:*', 'cake:birthday'], ['baloons:mylar:happy-birthday'])\n```\n\nThe `normalizeScopes` function normalizes a scopeset, removing redundant scopes and sorting.\n\n```python\nassert normalizedScopes(['balloons:*', 'balloons:mylar:*']) == ['baloons:*']\n```\n\nFinally, `Resolver` can perform scope expansion.\nIt is initialized with a dictionary mapping roleIds to scope lists.\nAlternately, it can be initialized from a `Resources` instance using `Resolver.from_resources(resources)`.\n\nIts `expandScopes` method behaves identically to the remote call `auth.expandScopes`.\n\n```python\nresolver = Resolver.from_resources(resources)\nassert resolver.expandScopes(['assume:clown:grimaldi']) == ['assume:clown:grimaldi', 'ruffle:full']\n```\n\n### aiohttp session\n\nThe library uses `aiohttp` to communicate with Taskcluster, and establishes a single session for efficiency.\nApplications can use the same session for any other HTTP operations.\n\n```python\nfrom tcadmin.util.session import aiohttp_session\n\nasync def foo():\n # ...\n async with aiohttp_session().get(url) as response:\n response.raise_for_status()\n result = await response.read()\n```\n\nTests and checks can set this value using `with_aiohttp_session`:\n\n```python\nfrom tcadmin.util.sessions import with_aiohttp_session\nimport pytest\n\n@pytest.mark.asyncio\n@with_aiohttp_session\nasync def test_something():\n # ...\n```\n\n### MatchList\n\nA MatchList is a list of regular expressions that can determine whether a given\nstring matches one of those patterns. Patterns are rooted at the left, but\nshould use `$` where required to match the end of the string.\n\n```python\nfrom tcadmin.util.matchlist import MatchList\n\nml = MatchList()\nml.add(\"fo+$\")\nml.add(\"ba+r$\")\nassert ml.matches(\"foo\")\n```\n\nThis functionality is used to track managed resources, but may be useful otherwise.\n\n# Development\n\nTo install for development, in a virtualenv:\n\n```\npip install -e [path]\n```\n\nAnd to run flake8 and tests:\n\n```\npython setup.py flake8\npython setup.py test\n```\n\nThe library uses [Black](https://black.readthedocs.io/en/stable/) to format code.\n\n```\npip install black\nblack tcadmin\n```\n\n## Releasing\n\nTo release:\n\n * update version in `setup.py` and `git commit -m \"vX.Y.Z\"`\n * `git tag vX.Y.z`\n * push those changes to `main`\n * `./release.sh --real` and enter your pypi credentials when prompted (omit the `--real` to try it against the testing pypi, if you're not sure)\n * Find the tag in https://github.com/taskcluster/tc-admin/releases and create a new release with a brief description of the changes\n\n\n## Docker image\n\nTo build and publish new version of tc-admin docker image run:\n\n```bash\ndocker build -t tc-admin:X.Y.Z .\ndocker push tc-admin:X.Y.Z\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Administration of Taskcluster runtime configuration",
"version": "4.0.1",
"project_urls": {
"Homepage": "https://github.com/taskcluster/tc-admin"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "758f1fbcb13f90f98dc81d1a3d98af65ddcceac079b19d3b71318dabf10eaea8",
"md5": "880590ada026a14270fd7069ab91d983",
"sha256": "22888b3d1856dc9b92b50964c1427c0d42a314ea95c2122f7bd2a32013d85e95"
},
"downloads": -1,
"filename": "tc_admin-4.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "880590ada026a14270fd7069ab91d983",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 45701,
"upload_time": "2024-08-07T15:48:49",
"upload_time_iso_8601": "2024-08-07T15:48:49.719403Z",
"url": "https://files.pythonhosted.org/packages/75/8f/1fbcb13f90f98dc81d1a3d98af65ddcceac079b19d3b71318dabf10eaea8/tc_admin-4.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "93b772e08e6d077567ab39a8729a9905e1e351b9cee78235574b3b787b7f8f1f",
"md5": "418ac827f5b81430ade901e92139a739",
"sha256": "7f6d4903cf45d412e9d3c0e204bda054238217bf05096fd766bafbc0750e3802"
},
"downloads": -1,
"filename": "tc_admin-4.0.1.tar.gz",
"has_sig": false,
"md5_digest": "418ac827f5b81430ade901e92139a739",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 32052,
"upload_time": "2024-08-07T15:48:51",
"upload_time_iso_8601": "2024-08-07T15:48:51.494309Z",
"url": "https://files.pythonhosted.org/packages/93/b7/72e08e6d077567ab39a8729a9905e1e351b9cee78235574b3b787b7f8f1f/tc_admin-4.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-07 15:48:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "taskcluster",
"github_project": "tc-admin",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "tc-admin"
}
JSON
