# Taskcluster Client for Python
[](https://pypi.python.org/pypi/taskcluster)
[](http://mozilla.org/MPL/2.0)
**A Taskcluster client library for Python.**
This library is a complete interface to Taskcluster in Python. It provides
both synchronous and asynchronous interfaces for all Taskcluster API methods.
## Usage
For a general guide to using Taskcluster clients, see [Calling Taskcluster APIs](https://docs.taskcluster.net/docs/manual/using/api).
### Setup
Before calling an API end-point, you'll need to create a client instance.
There is a class for each service, e.g., `Queue` and `Auth`. Each takes the
same options, described below. Note that only `rootUrl` is
required, and it's unusual to configure any other options aside from
`credentials`.
For each service, there are sync and async variants. The classes under
`taskcluster` (e.g., `taskcluster.Queue`) operate synchronously. The classes
under `taskcluster.aio` (e.g., `taskcluster.aio.Queue`) are asynchronous.
#### Authentication Options
Here is a simple set-up of an Index client:
```python
import taskcluster
index = taskcluster.Index({
'rootUrl': 'https://tc.example.com',
'credentials': {'clientId': 'id', 'accessToken': 'accessToken'},
})
```
The `rootUrl` option is required as it gives the Taskcluster deployment to
which API requests should be sent. Credentials are only required if the
request is to be authenticated -- many Taskcluster API methods do not require
authentication.
In most cases, the root URL and Taskcluster credentials should be provided in [standard environment variables](https://docs.taskcluster.net/docs/manual/design/env-vars). Use `taskcluster.optionsFromEnvironment()` to read these variables automatically:
```python
auth = taskcluster.Auth(taskcluster.optionsFromEnvironment())
```
Note that this function does not respect `TASKCLUSTER_PROXY_URL`. To use the Taskcluster Proxy from within a task:
```python
auth = taskcluster.Auth({'rootUrl': os.environ['TASKCLUSTER_PROXY_URL']})
```
#### Authorized Scopes
If you wish to perform requests on behalf of a third-party that has small set
of scopes than you do. You can specify [which scopes your request should be
allowed to
use](https://docs.taskcluster.net/docs/manual/design/apis/hawk/authorized-scopes),
in the `authorizedScopes` option.
```python
opts = taskcluster.optionsFromEnvironment()
opts['authorizedScopes'] = ['queue:create-task:highest:my-provisioner/my-worker-type']
queue = taskcluster.Queue(opts)
```
#### Other Options
The following additional options are accepted when constructing a client object:
* `signedUrlExpiration` - default value for the `expiration` argument to `buildSignedUrl`
* `maxRetries` - maximum number of times to retry a failed request
### Calling API Methods
API methods are available as methods on the corresponding client object. For
sync clients, these are sync methods, and for async clients they are async
methods; the calling convention is the same in either case.
There are four calling conventions for methods:
```python
client.method(v1, v1, payload)
client.method(payload, k1=v1, k2=v2)
client.method(payload=payload, query=query, params={k1: v1, k2: v2})
client.method(v1, v2, payload=payload, query=query)
```
Here, `v1` and `v2` are URL parameters (named `k1` and `k2`), `payload` is the
request payload, and `query` is a dictionary of query arguments.
For example, in order to call an API method with query-string arguments:
```python
await queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g',
query={'continuationToken': previousResponse.get('continuationToken')})
```
### Generating URLs
It is often necessary to generate the URL for an API method without actually calling the method.
To do so, use `buildUrl` or, for an API method that requires authentication, `buildSignedUrl`.
```python
import taskcluster
index = taskcluster.Index(taskcluster.optionsFromEnvironment())
print(index.buildUrl('findTask', 'builds.v1.latest'))
secrets = taskcluster.Secrets(taskcluster.optionsFromEnvironment())
print(secret.buildSignedUrl('get', 'my-secret'))
```
Note that signed URLs are time-limited; the expiration can be set with the `signedUrlExpiration` option to the client constructor, or with the `expiration` keyword arguement to `buildSignedUrl`, both given in seconds.
### Generating Temporary Credentials
If you have non-temporary taskcluster credentials you can generate a set of
[temporary credentials](https://docs.taskcluster.net/docs/manual/design/apis/hawk/temporary-credentials) as follows. Notice that the credentials cannot last more
than 31 days, and you can only revoke them by revoking the credentials that was
used to issue them (this takes up to one hour).
It is not the responsibility of the caller to apply any clock drift adjustment
to the start or expiry time - this is handled by the auth service directly.
```python
import datetime
start = datetime.datetime.now()
expiry = start + datetime.timedelta(0,60)
scopes = ['ScopeA', 'ScopeB']
name = 'foo'
credentials = taskcluster.createTemporaryCredentials(
# issuing clientId
clientId,
# issuing accessToken
accessToken,
# Validity of temporary credentials starts here, in timestamp
start,
# Expiration of temporary credentials, in timestamp
expiry,
# Scopes to grant the temporary credentials
scopes,
# credential name (optional)
name
)
```
You cannot use temporary credentials to issue new temporary credentials. You
must have `auth:create-client:<name>` to create a named temporary credential,
but unnamed temporary credentials can be created regardless of your scopes.
### Handling Timestamps
Many taskcluster APIs require ISO 8601 time stamps offset into the future
as way of providing expiration, deadlines, etc. These can be easily created
using `datetime.datetime.isoformat()`, however, it can be rather error prone
and tedious to offset `datetime.datetime` objects into the future. Therefore
this library comes with two utility functions for this purposes.
```python
dateObject = taskcluster.fromNow("2 days 3 hours 1 minute")
# -> datetime.datetime(2017, 1, 21, 17, 8, 1, 607929)
dateString = taskcluster.fromNowJSON("2 days 3 hours 1 minute")
# -> '2017-01-21T17:09:23.240178Z'
```
By default it will offset the date time into the future, if the offset strings
are prefixed minus (`-`) the date object will be offset into the past. This is
useful in some corner cases.
```python
dateObject = taskcluster.fromNow("- 1 year 2 months 3 weeks 5 seconds");
# -> datetime.datetime(2015, 10, 30, 18, 16, 50, 931161)
```
The offset string is ignorant of whitespace and case insensitive. It may also
optionally be prefixed plus `+` (if not prefixed minus), any `+` prefix will be
ignored. However, entries in the offset string must be given in order from
high to low, ie. `2 years 1 day`. Additionally, various shorthands may be
employed, as illustrated below.
```
years, year, yr, y
months, month, mo
weeks, week, w
days, day, d
hours, hour, h
minutes, minute, min
seconds, second, sec, s
```
The `fromNow` method may also be given a date to be relative to as a second
argument. This is useful if offset the task expiration relative to the the task
deadline or doing something similar. This argument can also be passed as the
kwarg `dateObj`
```python
dateObject1 = taskcluster.fromNow("2 days 3 hours");
dateObject2 = taskcluster.fromNow("1 year", dateObject1);
taskcluster.fromNow("1 year", dateObj=dateObject1);
# -> datetime.datetime(2018, 1, 21, 17, 59, 0, 328934)
```
### Generating SlugIDs
To generate slugIds (Taskcluster's client-generated unique IDs), use
`taskcluster.slugId()`, which will return a unique slugId on each call.
In some cases it is useful to be able to create a mapping from names to
slugIds, with the ability to generate the same slugId multiple times.
The `taskcluster.stableSlugId()` function returns a callable that does
just this.
```python
gen = taskcluster.stableSlugId()
sometask = gen('sometask')
assert gen('sometask') == sometask # same input generates same output
assert gen('sometask') != gen('othertask')
gen2 = taskcluster.stableSlugId()
sometask2 = gen('sometask')
assert sometask2 != sometask # but different slugId generators produce
# different output
```
### Scope Analysis
The `scopeMatch(assumedScopes, requiredScopeSets)` function determines
whether one or more of a set of required scopes are satisfied by the assumed
scopes, taking *-expansion into account. This is useful for making local
decisions on scope satisfaction, but note that `assumed_scopes` must be the
*expanded* scopes, as this function cannot perform expansion.
It takes a list of a assumed scopes, and a list of required scope sets on
disjunctive normal form, and checks if any of the required scope sets are
satisfied.
Example:
```python
requiredScopeSets = [
["scopeA", "scopeB"],
["scopeC:*"]
]
assert scopesMatch(['scopeA', 'scopeB'], requiredScopeSets)
assert scopesMatch(['scopeC:xyz'], requiredScopeSets)
assert not scopesMatch(['scopeA'], requiredScopeSets)
assert not scopesMatch(['scopeC'], requiredScopeSets)
```
### Pagination
Many Taskcluster API methods are paginated. There are two ways to handle
pagination easily with the python client. The first is to implement pagination
in your code:
```python
import taskcluster
queue = taskcluster.Queue({'rootUrl': 'https://tc.example.com'})
i = 0
tasks = 0
outcome = queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g')
while outcome.get('continuationToken'):
print('Response %d gave us %d more tasks' % (i, len(outcome['tasks'])))
if outcome.get('continuationToken'):
outcome = queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g', query={'continuationToken': outcome.get('continuationToken')})
i += 1
tasks += len(outcome.get('tasks', []))
print('Task Group %s has %d tasks' % (outcome['taskGroupId'], tasks))
```
There's also an experimental feature to support built in automatic pagination
in the sync client. This feature allows passing a callback as the
'paginationHandler' keyword-argument. This function will be passed the
response body of the API method as its sole positional arugment.
This example of the built in pagination shows how a list of tasks could be
built and then counted:
```python
import taskcluster
queue = taskcluster.Queue({'rootUrl': 'https://tc.example.com'})
responses = []
def handle_page(y):
print("%d tasks fetched" % len(y.get('tasks', [])))
responses.append(y)
queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g', paginationHandler=handle_page)
tasks = 0
for response in responses:
tasks += len(response.get('tasks', []))
print("%d requests fetch %d tasks" % (len(responses), tasks))
```
### Pulse Events
This library can generate exchange patterns for Pulse messages based on the
Exchanges definitions provded by each service. This is done by instantiating a
`<service>Events` class and calling a method with the name of the vent.
Options for the topic exchange methods can be in the form of either a single
dictionary argument or keyword arguments. Only one form is allowed.
```python
from taskcluster import client
qEvt = client.QueueEvents({rootUrl: 'https://tc.example.com'})
# The following calls are equivalent
print(qEvt.taskCompleted({'taskId': 'atask'}))
print(qEvt.taskCompleted(taskId='atask'))
```
Note that the client library does *not* provide support for interfacing with a Pulse server.
### Logging
Logging is set up in `taskcluster/__init__.py`. If the special
`DEBUG_TASKCLUSTER_CLIENT` environment variable is set, the `__init__.py`
module will set the `logging` module's level for its logger to `logging.DEBUG`
and if there are no existing handlers, add a `logging.StreamHandler()`
instance. This is meant to assist those who do not wish to bother figuring out
how to configure the python logging module but do want debug messages
## Uploading and Downloading Objects
The Object service provides an API for reliable uploads and downloads of large objects.
This library provides convenience methods to implement the client portion of those APIs, providing well-tested, resilient upload and download functionality.
These methods will negotiate the appropriate method with the object service and perform the required steps to transfer the data.
All methods are available in both sync and async versions, with identical APIs except for the `async`/`await` keywords.
In either case, you will need to provide a configured `Object` instance with appropriate credentials for the operation.
NOTE: There is an helper function to upload `s3` artifacts, `taskcluster.helper.upload_artifact`, but it is deprecated as it only supports the `s3` artifact type.
### Uploads
To upload, use any of the following:
* `await taskcluster.aio.upload.uploadFromBuf(projectId=.., name=.., contentType=.., contentLength=.., uploadId=.., expires=.., maxRetries=.., objectService=.., data=..)` - asynchronously upload data from a buffer full of bytes.
* `await taskcluster.aio.upload.uploadFromFile(projectId=.., name=.., contentType=.., contentLength=.., uploadId=.., expires=.., maxRetries=.., objectService=.., file=..)` - asynchronously upload data from a standard Python file.
Note that this is [probably what you want](https://github.com/python/asyncio/wiki/ThirdParty#filesystem), even in an async context.
* `await taskcluster.aio.upload(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., readerFactory=..)` - asynchronously upload data from an async reader factory.
* `taskcluster.upload.uploadFromBuf(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., data=..)` - upload data from a buffer full of bytes.
* `taskcluster.upload.uploadFromFile(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., file=..)` - upload data from a standard Python file.
* `taskcluster.upload(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., readerFactory=..)` - upload data from a sync reader factory.
A "reader" is an object with a `read(max_size=-1)` method which reads and returns a chunk of 1 .. `max_size` bytes, or returns an empty string at EOF, async for the async functions and sync for the remainder.
A "reader factory" is an async callable which returns a fresh reader, ready to read the first byte of the object.
When uploads are retried, the reader factory may be called more than once.
The `uploadId` parameter may be omitted, in which case a new slugId will be generated.
### Downloads
To download, use any of the following:
* `await taskcluster.aio.download.downloadToBuf(name=.., maxRetries=.., objectService=..)` - asynchronously download an object to an in-memory buffer, returning a tuple (buffer, content-type).
If the file is larger than available memory, this will crash.
* `await taskcluster.aio.download.downloadToFile(name=.., maxRetries=.., objectService=.., file=..)` - asynchronously download an object to a standard Python file, returning the content type.
* `await taskcluster.aio.download.download(name=.., maxRetries=.., objectService=.., writerFactory=..)` - asynchronously download an object to an async writer factory, returning the content type.
* `taskcluster.download.downloadToBuf(name=.., maxRetries=.., objectService=..)` - download an object to an in-memory buffer, returning a tuple (buffer, content-type).
If the file is larger than available memory, this will crash.
* `taskcluster.download.downloadToFile(name=.., maxRetries=.., objectService=.., file=..)` - download an object to a standard Python file, returning the content type.
* `taskcluster.download.download(name=.., maxRetries=.., objectService=.., writerFactory=..)` - download an object to a sync writer factory, returning the content type.
A "writer" is an object with a `write(data)` method which writes the given data, async for the async functions and sync for the remainder.
A "writer factory" is a callable (again either async or sync) which returns a fresh writer, ready to write the first byte of the object.
When uploads are retried, the writer factory may be called more than once.
### Artifact Downloads
Artifacts can be downloaded from the queue service with similar functions to those above.
These functions support all of the queue's storage types, raising an error for `error` artifacts.
In each case, if `runId` is omitted then the most recent run will be used.
* `await taskcluster.aio.download.downloadArtifactToBuf(taskId=.., runId=.., name=.., maxRetries=.., queueService=..)` - asynchronously download an object to an in-memory buffer, returning a tuple (buffer, content-type).
If the file is larger than available memory, this will crash.
* `await taskcluster.aio.download.downloadArtifactToFile(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., file=..)` - asynchronously download an object to a standard Python file, returning the content type.
* `await taskcluster.aio.download.downloadArtifact(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., writerFactory=..)` - asynchronously download an object to an async writer factory, returning the content type.
* `taskcluster.download.downloadArtifactToBuf(taskId=.., runId=.., name=.., maxRetries=.., queueService=..)` - download an object to an in-memory buffer, returning a tuple (buffer, content-type).
If the file is larger than available memory, this will crash.
* `taskcluster.download.downloadArtifactToFile(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., file=..)` - download an object to a standard Python file, returning the content type.
* `taskcluster.download.downloadArtifact(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., writerFactory=..)` - download an object to a sync writer factory, returning the content type.
## Integration Helpers
The Python Taskcluster client has a module `taskcluster.helper` with utilities which allows you to easily share authentication options across multiple services in your project.
Generally a project using this library will face different use cases and authentication options:
* No authentication for a new contributor without Taskcluster access,
* Specific client credentials through environment variables on a developer's computer,
* Taskcluster Proxy when running inside a task.
### Shared authentication
The class `taskcluster.helper.TaskclusterConfig` is made to be instantiated once in your project, usually in a top level module. That singleton is then accessed by different parts of your projects, whenever a Taskcluster service is needed.
Here is a sample usage:
1. in `project/__init__.py`, no call to Taskcluster is made at that point:
```python
from taskcluster.helper import Taskcluster config
tc = TaskclusterConfig('https://community-tc.services.mozilla.com')
```
2. in `project/boot.py`, we authenticate on Taskcuster with provided credentials, or environment variables, or taskcluster proxy (in that order):
```python
from project import tc
tc.auth(client_id='XXX', access_token='YYY')
```
3. at that point, you can load any service using the authenticated wrapper from anywhere in your code:
```python
from project import tc
def sync_usage():
queue = tc.get_service('queue')
queue.ping()
async def async_usage():
hooks = tc.get_service('hooks', use_async=True) # Asynchronous service class
await hooks.ping()
```
Supported environment variables are:
- `TASKCLUSTER_ROOT_URL` to specify your Taskcluster instance base url. You can either use that variable or instanciate `TaskclusterConfig` with the base url.
- `TASKCLUSTER_CLIENT_ID` & `TASKCLUSTER_ACCESS_TOKEN` to specify your client credentials instead of providing them to `TaskclusterConfig.auth`
- `TASKCLUSTER_PROXY_URL` to specify the proxy address used to reach Taskcluster in a task. It defaults to `http://taskcluster` when not specified.
For more details on Taskcluster environment variables, [here is the documentation](https://docs.taskcluster.net/docs/manual/design/env-vars).
### Loading secrets across multiple authentications
Another available utility is `taskcluster.helper.load_secrets` which allows you to retrieve a secret using an authenticated `taskcluster.Secrets` instance (using `TaskclusterConfig.get_service` or the synchronous class directly).
This utility loads a secret, but allows you to:
1. share a secret across multiple projects, by using key prefixes inside the secret,
2. check that some required keys are present in the secret,
3. provide some default values,
4. provide a local secret source instead of using the Taskcluster service (useful for local development or sharing _secrets_ with contributors)
Let's say you have a secret on a Taskcluster instance named `project/foo/prod-config`, which is needed by a backend and some tasks. Here is its content:
```yaml
common:
environment: production
remote_log: https://log.xx.com/payload
backend:
bugzilla_token: XXXX
task:
backend_url: https://backend.foo.mozilla.com
```
In your backend, you would do:
```python
from taskcluster import Secrets
from taskcluster.helper import load_secrets
prod_config = load_secrets(
Secrets({...}),
'project/foo/prod-config',
# We only need the common & backend parts
prefixes=['common', 'backend'],
# We absolutely need a bugzilla token to run
required=['bugzilla_token'],
# Let's provide some default value for the environment
existing={
'environment': 'dev',
}
)
# -> prod_config == {
# "environment": "production"
# "remote_log": "https://log.xx.com/payload",
# "bugzilla_token": "XXXX",
# }
```
In your task, you could do the following using `TaskclusterConfig` mentionned above (the class has a shortcut to use an authenticated `Secrets` service automatically):
```python
from project import tc
prod_config = tc.load_secrets(
'project/foo/prod-config',
# We only need the common & bot parts
prefixes=['common', 'bot'],
# Let's provide some default value for the environment and backend_url
existing={
'environment': 'dev',
'backend_url': 'http://localhost:8000',
}
)
# -> prod_config == {
# "environment": "production"
# "remote_log": "https://log.xx.com/payload",
# "backend_url": "https://backend.foo.mozilla.com",
# }
```
To provide local secrets value, you first need to load these values as a dictionary (usually by reading a local file in your format of choice : YAML, JSON, ...) and providing the dictionary to `load_secrets` by using the `local_secrets` parameter:
```python
import os
import yaml
from taskcluster import Secrets
from taskcluster.helper import load_secrets
local_path = 'path/to/file.yml'
prod_config = load_secrets(
Secrets({...}),
'project/foo/prod-config',
# We support an optional local file to provide some configuration without reaching Taskcluster
local_secrets=yaml.safe_load(open(local_path)) if os.path.exists(local_path) else None,
)
```
## Compatibility
This library is co-versioned with Taskcluster itself.
That is, a client with version x.y.z contains API methods corresponding to Taskcluster version x.y.z.
Taskcluster is careful to maintain API compatibility, and guarantees it within a major version.
That means that any client with version x.* will work against any Taskcluster services at version x.*, and is very likely to work for many other major versions of the Taskcluster services.
Any incompatibilities are noted in the [Changelog](https://github.com/taskcluster/taskcluster/blob/main/CHANGELOG.md).
Raw data
{
"_id": null,
"home_page": "https://github.com/taskcluster/taskcluster",
"name": "taskcluster",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Mozilla Taskcluster and Release Engineering",
"author_email": "release+python@mozilla.com",
"download_url": "https://files.pythonhosted.org/packages/40/27/1e5a81c563131bbf5c34867030df98aa22494fbd3ef52cf3251625370b96/taskcluster-64.2.5.tar.gz",
"platform": null,
"description": "# Taskcluster Client for Python\n\n[](https://pypi.python.org/pypi/taskcluster)\n[](http://mozilla.org/MPL/2.0)\n\n**A Taskcluster client library for Python.**\n\nThis library is a complete interface to Taskcluster in Python. It provides\nboth synchronous and asynchronous interfaces for all Taskcluster API methods.\n\n## Usage\n\nFor a general guide to using Taskcluster clients, see [Calling Taskcluster APIs](https://docs.taskcluster.net/docs/manual/using/api).\n\n### Setup\n\nBefore calling an API end-point, you'll need to create a client instance.\nThere is a class for each service, e.g., `Queue` and `Auth`. Each takes the\nsame options, described below. Note that only `rootUrl` is\nrequired, and it's unusual to configure any other options aside from\n`credentials`.\n\nFor each service, there are sync and async variants. The classes under\n`taskcluster` (e.g., `taskcluster.Queue`) operate synchronously. The classes\nunder `taskcluster.aio` (e.g., `taskcluster.aio.Queue`) are asynchronous.\n\n#### Authentication Options\n\nHere is a simple set-up of an Index client:\n\n```python\nimport taskcluster\nindex = taskcluster.Index({\n 'rootUrl': 'https://tc.example.com',\n 'credentials': {'clientId': 'id', 'accessToken': 'accessToken'},\n})\n```\n\nThe `rootUrl` option is required as it gives the Taskcluster deployment to\nwhich API requests should be sent. Credentials are only required if the\nrequest is to be authenticated -- many Taskcluster API methods do not require\nauthentication.\n\nIn most cases, the root URL and Taskcluster credentials should be provided in [standard environment variables](https://docs.taskcluster.net/docs/manual/design/env-vars). Use `taskcluster.optionsFromEnvironment()` to read these variables automatically:\n\n```python\nauth = taskcluster.Auth(taskcluster.optionsFromEnvironment())\n```\n\nNote that this function does not respect `TASKCLUSTER_PROXY_URL`. To use the Taskcluster Proxy from within a task:\n\n```python\nauth = taskcluster.Auth({'rootUrl': os.environ['TASKCLUSTER_PROXY_URL']})\n```\n\n#### Authorized Scopes\n\nIf you wish to perform requests on behalf of a third-party that has small set\nof scopes than you do. You can specify [which scopes your request should be\nallowed to\nuse](https://docs.taskcluster.net/docs/manual/design/apis/hawk/authorized-scopes),\nin the `authorizedScopes` option.\n\n```python\nopts = taskcluster.optionsFromEnvironment()\nopts['authorizedScopes'] = ['queue:create-task:highest:my-provisioner/my-worker-type']\nqueue = taskcluster.Queue(opts)\n```\n\n#### Other Options\n\nThe following additional options are accepted when constructing a client object:\n\n* `signedUrlExpiration` - default value for the `expiration` argument to `buildSignedUrl`\n* `maxRetries` - maximum number of times to retry a failed request\n\n### Calling API Methods\n\nAPI methods are available as methods on the corresponding client object. For\nsync clients, these are sync methods, and for async clients they are async\nmethods; the calling convention is the same in either case.\n\nThere are four calling conventions for methods:\n\n```python\nclient.method(v1, v1, payload)\nclient.method(payload, k1=v1, k2=v2)\nclient.method(payload=payload, query=query, params={k1: v1, k2: v2})\nclient.method(v1, v2, payload=payload, query=query)\n```\n\nHere, `v1` and `v2` are URL parameters (named `k1` and `k2`), `payload` is the\nrequest payload, and `query` is a dictionary of query arguments.\n\nFor example, in order to call an API method with query-string arguments:\n\n```python\nawait queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g',\n query={'continuationToken': previousResponse.get('continuationToken')})\n```\n\n\n### Generating URLs\n\nIt is often necessary to generate the URL for an API method without actually calling the method.\nTo do so, use `buildUrl` or, for an API method that requires authentication, `buildSignedUrl`.\n\n```python\nimport taskcluster\n\nindex = taskcluster.Index(taskcluster.optionsFromEnvironment())\nprint(index.buildUrl('findTask', 'builds.v1.latest'))\nsecrets = taskcluster.Secrets(taskcluster.optionsFromEnvironment())\nprint(secret.buildSignedUrl('get', 'my-secret'))\n```\n\nNote that signed URLs are time-limited; the expiration can be set with the `signedUrlExpiration` option to the client constructor, or with the `expiration` keyword arguement to `buildSignedUrl`, both given in seconds.\n\n### Generating Temporary Credentials\n\nIf you have non-temporary taskcluster credentials you can generate a set of\n[temporary credentials](https://docs.taskcluster.net/docs/manual/design/apis/hawk/temporary-credentials) as follows. Notice that the credentials cannot last more\nthan 31 days, and you can only revoke them by revoking the credentials that was\nused to issue them (this takes up to one hour).\n\nIt is not the responsibility of the caller to apply any clock drift adjustment\nto the start or expiry time - this is handled by the auth service directly.\n\n```python\nimport datetime\n\nstart = datetime.datetime.now()\nexpiry = start + datetime.timedelta(0,60)\nscopes = ['ScopeA', 'ScopeB']\nname = 'foo'\n\ncredentials = taskcluster.createTemporaryCredentials(\n # issuing clientId\n clientId,\n # issuing accessToken\n accessToken,\n # Validity of temporary credentials starts here, in timestamp\n start,\n # Expiration of temporary credentials, in timestamp\n expiry,\n # Scopes to grant the temporary credentials\n scopes,\n # credential name (optional)\n name\n)\n```\n\nYou cannot use temporary credentials to issue new temporary credentials. You\nmust have `auth:create-client:<name>` to create a named temporary credential,\nbut unnamed temporary credentials can be created regardless of your scopes.\n\n### Handling Timestamps\nMany taskcluster APIs require ISO 8601 time stamps offset into the future\nas way of providing expiration, deadlines, etc. These can be easily created\nusing `datetime.datetime.isoformat()`, however, it can be rather error prone\nand tedious to offset `datetime.datetime` objects into the future. Therefore\nthis library comes with two utility functions for this purposes.\n\n```python\ndateObject = taskcluster.fromNow(\"2 days 3 hours 1 minute\")\n # -> datetime.datetime(2017, 1, 21, 17, 8, 1, 607929)\ndateString = taskcluster.fromNowJSON(\"2 days 3 hours 1 minute\")\n # -> '2017-01-21T17:09:23.240178Z'\n```\n\nBy default it will offset the date time into the future, if the offset strings\nare prefixed minus (`-`) the date object will be offset into the past. This is\nuseful in some corner cases.\n\n```python\ndateObject = taskcluster.fromNow(\"- 1 year 2 months 3 weeks 5 seconds\");\n # -> datetime.datetime(2015, 10, 30, 18, 16, 50, 931161)\n```\n\nThe offset string is ignorant of whitespace and case insensitive. It may also\noptionally be prefixed plus `+` (if not prefixed minus), any `+` prefix will be\nignored. However, entries in the offset string must be given in order from\nhigh to low, ie. `2 years 1 day`. Additionally, various shorthands may be\nemployed, as illustrated below.\n\n```\n years, year, yr, y\n months, month, mo\n weeks, week, w\n days, day, d\n hours, hour, h\n minutes, minute, min\n seconds, second, sec, s\n```\n\nThe `fromNow` method may also be given a date to be relative to as a second\nargument. This is useful if offset the task expiration relative to the the task\ndeadline or doing something similar. This argument can also be passed as the\nkwarg `dateObj`\n\n```python\ndateObject1 = taskcluster.fromNow(\"2 days 3 hours\");\ndateObject2 = taskcluster.fromNow(\"1 year\", dateObject1);\ntaskcluster.fromNow(\"1 year\", dateObj=dateObject1);\n # -> datetime.datetime(2018, 1, 21, 17, 59, 0, 328934)\n```\n### Generating SlugIDs\n\nTo generate slugIds (Taskcluster's client-generated unique IDs), use\n`taskcluster.slugId()`, which will return a unique slugId on each call.\n\nIn some cases it is useful to be able to create a mapping from names to\nslugIds, with the ability to generate the same slugId multiple times.\nThe `taskcluster.stableSlugId()` function returns a callable that does\njust this.\n\n```python\ngen = taskcluster.stableSlugId()\nsometask = gen('sometask')\nassert gen('sometask') == sometask # same input generates same output\nassert gen('sometask') != gen('othertask')\n\ngen2 = taskcluster.stableSlugId()\nsometask2 = gen('sometask')\nassert sometask2 != sometask # but different slugId generators produce\n # different output\n```\n\n### Scope Analysis\n\nThe `scopeMatch(assumedScopes, requiredScopeSets)` function determines\nwhether one or more of a set of required scopes are satisfied by the assumed\nscopes, taking *-expansion into account. This is useful for making local\ndecisions on scope satisfaction, but note that `assumed_scopes` must be the\n*expanded* scopes, as this function cannot perform expansion.\n\nIt takes a list of a assumed scopes, and a list of required scope sets on\ndisjunctive normal form, and checks if any of the required scope sets are\nsatisfied.\n\nExample:\n\n```python\nrequiredScopeSets = [\n [\"scopeA\", \"scopeB\"],\n [\"scopeC:*\"]\n]\nassert scopesMatch(['scopeA', 'scopeB'], requiredScopeSets)\nassert scopesMatch(['scopeC:xyz'], requiredScopeSets)\nassert not scopesMatch(['scopeA'], requiredScopeSets)\nassert not scopesMatch(['scopeC'], requiredScopeSets)\n```\n\n### Pagination\n\nMany Taskcluster API methods are paginated. There are two ways to handle\npagination easily with the python client. The first is to implement pagination\nin your code:\n\n```python\nimport taskcluster\nqueue = taskcluster.Queue({'rootUrl': 'https://tc.example.com'})\ni = 0\ntasks = 0\noutcome = queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g')\nwhile outcome.get('continuationToken'):\n print('Response %d gave us %d more tasks' % (i, len(outcome['tasks'])))\n if outcome.get('continuationToken'):\n outcome = queue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g', query={'continuationToken': outcome.get('continuationToken')})\n i += 1\n tasks += len(outcome.get('tasks', []))\nprint('Task Group %s has %d tasks' % (outcome['taskGroupId'], tasks))\n```\n\nThere's also an experimental feature to support built in automatic pagination\nin the sync client. This feature allows passing a callback as the\n'paginationHandler' keyword-argument. This function will be passed the\nresponse body of the API method as its sole positional arugment.\n\nThis example of the built in pagination shows how a list of tasks could be\nbuilt and then counted:\n\n```python\nimport taskcluster\nqueue = taskcluster.Queue({'rootUrl': 'https://tc.example.com'})\n\nresponses = []\n\ndef handle_page(y):\n print(\"%d tasks fetched\" % len(y.get('tasks', [])))\n responses.append(y)\n\nqueue.listTaskGroup('JzTGxwxhQ76_Tt1dxkaG5g', paginationHandler=handle_page)\n\ntasks = 0\nfor response in responses:\n tasks += len(response.get('tasks', []))\n\nprint(\"%d requests fetch %d tasks\" % (len(responses), tasks))\n```\n\n### Pulse Events\n\nThis library can generate exchange patterns for Pulse messages based on the\nExchanges definitions provded by each service. This is done by instantiating a\n`<service>Events` class and calling a method with the name of the vent.\nOptions for the topic exchange methods can be in the form of either a single\ndictionary argument or keyword arguments. Only one form is allowed.\n\n```python\nfrom taskcluster import client\nqEvt = client.QueueEvents({rootUrl: 'https://tc.example.com'})\n# The following calls are equivalent\nprint(qEvt.taskCompleted({'taskId': 'atask'}))\nprint(qEvt.taskCompleted(taskId='atask'))\n```\n\nNote that the client library does *not* provide support for interfacing with a Pulse server.\n\n### Logging\n\nLogging is set up in `taskcluster/__init__.py`. If the special\n`DEBUG_TASKCLUSTER_CLIENT` environment variable is set, the `__init__.py`\nmodule will set the `logging` module's level for its logger to `logging.DEBUG`\nand if there are no existing handlers, add a `logging.StreamHandler()`\ninstance. This is meant to assist those who do not wish to bother figuring out\nhow to configure the python logging module but do want debug messages\n\n## Uploading and Downloading Objects\n\nThe Object service provides an API for reliable uploads and downloads of large objects.\nThis library provides convenience methods to implement the client portion of those APIs, providing well-tested, resilient upload and download functionality.\nThese methods will negotiate the appropriate method with the object service and perform the required steps to transfer the data.\n\nAll methods are available in both sync and async versions, with identical APIs except for the `async`/`await` keywords.\n\nIn either case, you will need to provide a configured `Object` instance with appropriate credentials for the operation.\n\nNOTE: There is an helper function to upload `s3` artifacts, `taskcluster.helper.upload_artifact`, but it is deprecated as it only supports the `s3` artifact type.\n\n### Uploads\n\nTo upload, use any of the following:\n\n* `await taskcluster.aio.upload.uploadFromBuf(projectId=.., name=.., contentType=.., contentLength=.., uploadId=.., expires=.., maxRetries=.., objectService=.., data=..)` - asynchronously upload data from a buffer full of bytes.\n* `await taskcluster.aio.upload.uploadFromFile(projectId=.., name=.., contentType=.., contentLength=.., uploadId=.., expires=.., maxRetries=.., objectService=.., file=..)` - asynchronously upload data from a standard Python file.\n Note that this is [probably what you want](https://github.com/python/asyncio/wiki/ThirdParty#filesystem), even in an async context.\n* `await taskcluster.aio.upload(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., readerFactory=..)` - asynchronously upload data from an async reader factory.\n* `taskcluster.upload.uploadFromBuf(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., data=..)` - upload data from a buffer full of bytes.\n* `taskcluster.upload.uploadFromFile(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., file=..)` - upload data from a standard Python file.\n* `taskcluster.upload(projectId=.., name=.., contentType=.., contentLength=.., expires=.., uploadId=.., maxRetries=.., objectService=.., readerFactory=..)` - upload data from a sync reader factory.\n\nA \"reader\" is an object with a `read(max_size=-1)` method which reads and returns a chunk of 1 .. `max_size` bytes, or returns an empty string at EOF, async for the async functions and sync for the remainder.\nA \"reader factory\" is an async callable which returns a fresh reader, ready to read the first byte of the object.\nWhen uploads are retried, the reader factory may be called more than once.\n\nThe `uploadId` parameter may be omitted, in which case a new slugId will be generated.\n\n### Downloads\n\nTo download, use any of the following:\n\n* `await taskcluster.aio.download.downloadToBuf(name=.., maxRetries=.., objectService=..)` - asynchronously download an object to an in-memory buffer, returning a tuple (buffer, content-type).\n If the file is larger than available memory, this will crash.\n* `await taskcluster.aio.download.downloadToFile(name=.., maxRetries=.., objectService=.., file=..)` - asynchronously download an object to a standard Python file, returning the content type.\n* `await taskcluster.aio.download.download(name=.., maxRetries=.., objectService=.., writerFactory=..)` - asynchronously download an object to an async writer factory, returning the content type.\n* `taskcluster.download.downloadToBuf(name=.., maxRetries=.., objectService=..)` - download an object to an in-memory buffer, returning a tuple (buffer, content-type).\n If the file is larger than available memory, this will crash.\n* `taskcluster.download.downloadToFile(name=.., maxRetries=.., objectService=.., file=..)` - download an object to a standard Python file, returning the content type.\n* `taskcluster.download.download(name=.., maxRetries=.., objectService=.., writerFactory=..)` - download an object to a sync writer factory, returning the content type.\n\nA \"writer\" is an object with a `write(data)` method which writes the given data, async for the async functions and sync for the remainder.\nA \"writer factory\" is a callable (again either async or sync) which returns a fresh writer, ready to write the first byte of the object.\nWhen uploads are retried, the writer factory may be called more than once.\n\n### Artifact Downloads\n\nArtifacts can be downloaded from the queue service with similar functions to those above.\nThese functions support all of the queue's storage types, raising an error for `error` artifacts.\nIn each case, if `runId` is omitted then the most recent run will be used.\n\n* `await taskcluster.aio.download.downloadArtifactToBuf(taskId=.., runId=.., name=.., maxRetries=.., queueService=..)` - asynchronously download an object to an in-memory buffer, returning a tuple (buffer, content-type).\n If the file is larger than available memory, this will crash.\n* `await taskcluster.aio.download.downloadArtifactToFile(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., file=..)` - asynchronously download an object to a standard Python file, returning the content type.\n* `await taskcluster.aio.download.downloadArtifact(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., writerFactory=..)` - asynchronously download an object to an async writer factory, returning the content type.\n* `taskcluster.download.downloadArtifactToBuf(taskId=.., runId=.., name=.., maxRetries=.., queueService=..)` - download an object to an in-memory buffer, returning a tuple (buffer, content-type).\n If the file is larger than available memory, this will crash.\n* `taskcluster.download.downloadArtifactToFile(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., file=..)` - download an object to a standard Python file, returning the content type.\n* `taskcluster.download.downloadArtifact(taskId=.., runId=.., name=.., maxRetries=.., queueService=.., writerFactory=..)` - download an object to a sync writer factory, returning the content type.\n\n## Integration Helpers\n\nThe Python Taskcluster client has a module `taskcluster.helper` with utilities which allows you to easily share authentication options across multiple services in your project.\n\nGenerally a project using this library will face different use cases and authentication options:\n\n* No authentication for a new contributor without Taskcluster access,\n* Specific client credentials through environment variables on a developer's computer,\n* Taskcluster Proxy when running inside a task.\n\n### Shared authentication\n\nThe class `taskcluster.helper.TaskclusterConfig` is made to be instantiated once in your project, usually in a top level module. That singleton is then accessed by different parts of your projects, whenever a Taskcluster service is needed.\n\nHere is a sample usage:\n\n1. in `project/__init__.py`, no call to Taskcluster is made at that point:\n\n```python\nfrom taskcluster.helper import Taskcluster config\n\ntc = TaskclusterConfig('https://community-tc.services.mozilla.com')\n```\n\n2. in `project/boot.py`, we authenticate on Taskcuster with provided credentials, or environment variables, or taskcluster proxy (in that order):\n\n```python\nfrom project import tc\n\ntc.auth(client_id='XXX', access_token='YYY')\n```\n\n3. at that point, you can load any service using the authenticated wrapper from anywhere in your code:\n\n```python\nfrom project import tc\n\ndef sync_usage():\n queue = tc.get_service('queue')\n queue.ping()\n\nasync def async_usage():\n hooks = tc.get_service('hooks', use_async=True) # Asynchronous service class\n await hooks.ping()\n```\n\nSupported environment variables are:\n- `TASKCLUSTER_ROOT_URL` to specify your Taskcluster instance base url. You can either use that variable or instanciate `TaskclusterConfig` with the base url.\n- `TASKCLUSTER_CLIENT_ID` & `TASKCLUSTER_ACCESS_TOKEN` to specify your client credentials instead of providing them to `TaskclusterConfig.auth`\n- `TASKCLUSTER_PROXY_URL` to specify the proxy address used to reach Taskcluster in a task. It defaults to `http://taskcluster` when not specified.\n\nFor more details on Taskcluster environment variables, [here is the documentation](https://docs.taskcluster.net/docs/manual/design/env-vars).\n\n### Loading secrets across multiple authentications\n\nAnother available utility is `taskcluster.helper.load_secrets` which allows you to retrieve a secret using an authenticated `taskcluster.Secrets` instance (using `TaskclusterConfig.get_service` or the synchronous class directly).\n\nThis utility loads a secret, but allows you to:\n1. share a secret across multiple projects, by using key prefixes inside the secret,\n2. check that some required keys are present in the secret,\n3. provide some default values,\n4. provide a local secret source instead of using the Taskcluster service (useful for local development or sharing _secrets_ with contributors)\n\nLet's say you have a secret on a Taskcluster instance named `project/foo/prod-config`, which is needed by a backend and some tasks. Here is its content:\n\n```yaml\ncommon:\n environment: production\n remote_log: https://log.xx.com/payload\n\nbackend:\n bugzilla_token: XXXX\n\ntask:\n backend_url: https://backend.foo.mozilla.com\n```\n\nIn your backend, you would do:\n\n```python\nfrom taskcluster import Secrets\nfrom taskcluster.helper import load_secrets\n\nprod_config = load_secrets(\n Secrets({...}),\n 'project/foo/prod-config',\n\n # We only need the common & backend parts\n prefixes=['common', 'backend'],\n\n # We absolutely need a bugzilla token to run\n required=['bugzilla_token'],\n\n # Let's provide some default value for the environment\n existing={\n 'environment': 'dev',\n }\n)\n # -> prod_config == {\n # \"environment\": \"production\"\n # \"remote_log\": \"https://log.xx.com/payload\",\n # \"bugzilla_token\": \"XXXX\",\n # }\n```\n\nIn your task, you could do the following using `TaskclusterConfig` mentionned above (the class has a shortcut to use an authenticated `Secrets` service automatically):\n\n```python\nfrom project import tc\n\nprod_config = tc.load_secrets(\n 'project/foo/prod-config',\n\n # We only need the common & bot parts\n prefixes=['common', 'bot'],\n\n # Let's provide some default value for the environment and backend_url\n existing={\n 'environment': 'dev',\n 'backend_url': 'http://localhost:8000',\n }\n)\n # -> prod_config == {\n # \"environment\": \"production\"\n # \"remote_log\": \"https://log.xx.com/payload\",\n # \"backend_url\": \"https://backend.foo.mozilla.com\",\n # }\n```\n\nTo provide local secrets value, you first need to load these values as a dictionary (usually by reading a local file in your format of choice : YAML, JSON, ...) and providing the dictionary to `load_secrets` by using the `local_secrets` parameter:\n\n```python\nimport os\nimport yaml\n\nfrom taskcluster import Secrets\nfrom taskcluster.helper import load_secrets\n\nlocal_path = 'path/to/file.yml'\n\nprod_config = load_secrets(\n Secrets({...}),\n 'project/foo/prod-config',\n\n # We support an optional local file to provide some configuration without reaching Taskcluster\n local_secrets=yaml.safe_load(open(local_path)) if os.path.exists(local_path) else None,\n)\n```\n\n## Compatibility\n\nThis library is co-versioned with Taskcluster itself.\nThat is, a client with version x.y.z contains API methods corresponding to Taskcluster version x.y.z.\nTaskcluster is careful to maintain API compatibility, and guarantees it within a major version.\nThat means that any client with version x.* will work against any Taskcluster services at version x.*, and is very likely to work for many other major versions of the Taskcluster services.\nAny incompatibilities are noted in the [Changelog](https://github.com/taskcluster/taskcluster/blob/main/CHANGELOG.md).\n",
"bugtrack_url": null,
"license": null,
"summary": "Python client for Taskcluster",
"version": "64.2.5",
"project_urls": {
"Homepage": "https://github.com/taskcluster/taskcluster"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a9f46c0ebff5f6c0aebbc6a2d061f336904c6c35e351da42db5ecf5debfbf7cf",
"md5": "3991d96c3a2f3cfa8a95d88a3e06cc43",
"sha256": "0972813cef47a6afca14445e945a978cfcb30dad401510a333b22c00803230d2"
},
"downloads": -1,
"filename": "taskcluster-64.2.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3991d96c3a2f3cfa8a95d88a3e06cc43",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 144579,
"upload_time": "2024-04-16T20:36:49",
"upload_time_iso_8601": "2024-04-16T20:36:49.901595Z",
"url": "https://files.pythonhosted.org/packages/a9/f4/6c0ebff5f6c0aebbc6a2d061f336904c6c35e351da42db5ecf5debfbf7cf/taskcluster-64.2.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "40271e5a81c563131bbf5c34867030df98aa22494fbd3ef52cf3251625370b96",
"md5": "24f5ba83876dc24d70945f935a6a0832",
"sha256": "cd419b0d2b3608b676bd07c415e2600a7bf0a316d92fc4d965bc652e937066a2"
},
"downloads": -1,
"filename": "taskcluster-64.2.5.tar.gz",
"has_sig": false,
"md5_digest": "24f5ba83876dc24d70945f935a6a0832",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 125547,
"upload_time": "2024-04-16T20:36:54",
"upload_time_iso_8601": "2024-04-16T20:36:54.919471Z",
"url": "https://files.pythonhosted.org/packages/40/27/1e5a81c563131bbf5c34867030df98aa22494fbd3ef52cf3251625370b96/taskcluster-64.2.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-16 20:36:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "taskcluster",
"github_project": "taskcluster",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "taskcluster"
}
JSON
