.. image:: https://github.com/con/tinuous/workflows/Test/badge.svg?branch=master
:target: https://github.com/con/tinuous/actions?workflow=Test
:alt: GitHub Actions Status
.. image:: https://travis-ci.com/con/tinuous.svg?branch=master
:target: https://travis-ci.com/con/tinuous
:alt: Travis CI Status
.. image:: https://ci.appveyor.com/api/projects/status/github/con/tinuous?branch=master&svg=true
:target: https://ci.appveyor.com/project/yarikoptic/tinuous/branch/master
:alt: Appveyor Status
.. image:: https://img.shields.io/pypi/pyversions/tinuous.svg
:target: https://pypi.org/project/tinuous/
.. image:: https://img.shields.io/conda/vn/conda-forge/tinuous.svg
:target: https://anaconda.org/conda-forge/tinuous
:alt: Conda Version
.. image:: https://img.shields.io/github/license/con/tinuous.svg
:target: https://opensource.org/licenses/MIT
:alt: MIT License
`GitHub <https://github.com/con/tinuous>`_
| `PyPI <https://pypi.org/project/tinuous/>`_
| `Anaconda <https://anaconda.org/conda-forge/tinuous>`_
| `Issues <https://github.com/con/tinuous/issues>`_
| `Changelog <https://github.com/con/tinuous/blob/master/CHANGELOG.md>`_
``tinuous`` is a command for downloading build logs and (for GitHub
only) artifacts & release assets for a GitHub repository from GitHub Actions,
Travis-CI.com, and/or Appveyor.
See <https://github.com/con/tinuous-inception> for an example setup that uses
``tinuous`` with GitHub Actions to fetch the CI logs for ``tinuous`` itself.
Installation
============
``tinuous`` requires Python 3.8 or higher. Just use `pip
<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install
``tinuous`` and its dependencies::
python3 -m pip install tinuous
``tinuous`` can also optionally integrate with DataLad_. To install DataLad
alongside ``tinuous``, specify the ``datalad`` extra::
python3 -m pip install "tinuous[datalad]"
``tinuous`` is also available for conda! To install, run::
conda install -c conda-forge tinuous
Usage
=====
::
tinuous [<global options>] <command> [<args> ...]
Global Options
--------------
-c FILE, --config FILE Read configuration from the given file [default
value: ``tinuous.yaml``]
-E FILE, --env FILE Load environment variables from the given
``.env`` file. By default, environment
variables are loaded from the first file named
"``.env``" found by searching from the current
directory upwards.
**Warning**: Care must be taken when this file
is located in a Git repository so as not to
publicly expose it: either list the file in
``.gitignore`` or, if using DataLad or
git-annex, configure git-annex to prohibit
public sharing of the file.
-l LEVEL, --log-level LEVEL Set the log level to the given value. Possible
values are "``CRITICAL``", "``ERROR``",
"``WARNING``", "``INFO``", "``DEBUG``" (all
case-insensitive) and their Python integer
equivalents. [default value: INFO]
``fetch`` Command
-----------------
::
tinuous [<global options>] fetch [<options>]
``tinuous fetch`` reads a configuration file telling it what repository to
retrieve logs & artifacts for, where to retrieve them from, and where to save
them, and then it carries those steps out.
Options
~~~~~~~
--sanitize-secrets Sanitize secrets from log files after
downloading
-S FILE, --state FILE Store program state (e.g., timestamps before
which all asset are known to have been fetched)
in the given file [default value:
``.tinuous.state.json``]
``sanitize`` Command
--------------------
::
tinuous [<global options>] sanitize <path> ...
Sanitize the given files, replacing all strings matching a secret regex with a
series of asterisks.
Configuration
-------------
The configuration file is a YAML file containing a mapping with the following
keys:
``repo``
*(required)* The GitHub repository to retrieve assets for, in the form ``OWNER/NAME``
``vars``
A mapping defining custom path template placeholders. Each key is the name
of a custom placeholder, without enclosing braces, and the value is the
string to substitute in its place. Custom values may contain standard path
template placeholders as well as other custom placeholders.
``ci``
*(required)* A mapping from the names of the CI systems from which to
retrieve assets to sub-mappings containing CI-specific configuration.
Including a given CI system is optional; assets will only be fetched from a
given system if it is listed in this mapping.
The CI systems and their sub-mappings are as follows:
``github``
Configuration for retrieving assets from GitHub Actions. Subfields:
``paths``
A mapping giving `template strings <Path Templates_>`_ for the
paths at which to save various types of assets. If this is empty
or not present, no assets are retrieved. Subfields:
``logs``
A template string that will be instantiated for each workflow
run to produce the path for the directory (relative to the
current working directory) under which the run's build logs
will be saved. If this is not specified, no logs will be
downloaded.
``artifacts``
A template string that will be instantiated for each workflow
run to produce the path for the directory (relative to the
current working directory) under which the run's artifacts will
be saved. If this is not specified, no artifacts will be
downloaded.
``releases``
A template string that will be instantiated for each
(non-draft, non-prerelease) GitHub release to produce the path
for the directory (relative to the current working directory)
under which the release's assets will be saved. If this is not
specified, no release assets will be downloaded.
``workflows``
A specification of the workflows for which to retrieve assets.
This can be either a list of workflow basenames, including the file
extension (e.g., ``test.yml``, not ``.github/workflows/test.yml``)
or a mapping containing the following fields:
``include``
A list of workflows to retrieve assets for, given as either
basenames or (when ``regex`` is true) `Python regular
expressions`_ to match against basenames. If ``include`` is
omitted, it defaults to including all workflows.
``exclude``
A list of workflows to not retrieve assets for, given as either
basenames or (when ``regex`` is true) `Python regular
expressions`_ to match against basenames. If ``exclude`` is
omitted, no workflows are excluded. Workflows that match both
``include`` and ``exclude`` are excluded.
``regex``
A boolean. If true (default false), the elements of the
``include`` and ``exclude`` fields are treated as `Python
regular expressions`_ that are matched (unanchored) against
workflow basenames; if false, they are used as exact names
When ``workflows`` is not specified, assets are retrieved for all
workflows in the repository.
``travis``
Configuration for retrieving logs from Travis-CI.com. Subfield:
``paths``
A mapping giving `template strings <Path Templates_>`_ for the
paths at which to save various types of assets. If this is empty
or not present, no assets are retrieved. Subfield:
``logs``
A template string that will be instantiated for each job of
each build to produce the path for the file (relative to the
current working directory) in which the job's logs will be
saved. If this is not specified, no logs will be downloaded.
``appveyor``
Configuration for retrieving logs from Appveyor. Subfields:
``paths``
A mapping giving `template strings <Path Templates_>`_ for the
paths at which to save various types of assets. If this is empty
or not present, no assets are retrieved. Subfield:
``logs``
A template string that will be instantiated for each job of
each build to produce the path for the file (relative to the
current working directory) in which the job's logs will be
saved. If this is not specified, no logs will be downloaded.
``accountName``
*(required)* The name of the Appveyor account to which the
repository belongs on Appveyor
``projectSlug``
The project slug for the repository on Appveyor; if not specified,
it is assumed that the slug is the same as the repository name
``since``
A timestamp (date, time, & timezone); only assets for builds started after
the given point in time will be retrieved. If not specified, the cutoff
set by ``max-days-back`` will be used.
As the script retrieves new build assets, it keeps track of their starting
points. Once the assets for all builds for the given CI system &
configuration have been fetched up to a certain point, the timestamp for
the latest such build is stored in the state file and used as the new
``since`` value for the respective CI system on subsequent runs. If the
``since`` setting in the configuration file is then updated to a newer
timestamp, the configuration will override the value in the state file, and
the next ``tinuous`` run will only retrieve assets after the new setting.
``max-days-back``
An integer specifying the maximum number of days back to look for builds;
defaults to 30.
If ``since`` is earlier than the date indicated by this value, and if the
timestamp for a given CI system saved in the state file is either missing
(i.e., if this is the first run of ``tinuous`` against the CI system) or
older than ``since``, then ``tinuous`` will use the explicitly-specified
``since`` value as the cutoff and ignore ``max-days-back``.
``until``
A timestamp (date, time, & timezone); only assets for builds started before
the given point in time will be retrieved
``types``
A list of build trigger event types; only assets for builds triggered by
one of the given events will be retrieved. If this is not specified,
assets will be retrieved for all recognized event types.
The recognized event types are:
``cron``
A build run on a schedule
``manual``
A build triggered manually by a human or through the CI system's API
``pr``
A build in response to activity on a pull request
``push``
A build in response to new commits
``secrets``
A mapping from names (used in log messages) to `Python regular
expressions`_ matching secrets to sanitize
``allow-secrets-regex``
Any strings that match a ``secrets`` regex and also match this regex will
not be sanitized. Note that ``allow-secrets-regex`` is tested against just
the substring that matched a ``secrets`` regex without any surrounding
text, and so lookahead and lookbehind will not work in this regex.
``datalad``
A sub-mapping describing integration of ``tinuous`` with DataLad_.
Subfields:
``enabled``
A boolean. If true (default false), DataLad must be installed, the
current directory will be converted into a DataLad dataset if it is not
one already, the assets will optionally be divided up into subdatasets,
and all new assets will be committed at the end of a run of ``tinuous
fetch``. ``path`` template strings may contain ``//`` separators
indicating the boundaries of subdatasets.
``cfg_proc``
Procedure to run on the dataset & subdatasets when creating them
.. _DataLad: https://www.datalad.org
.. _Python regular expressions: https://docs.python.org/3/library/re.html
#regular-expression-syntax
A sample config file:
.. code:: yaml
repo: datalad/datalad
vars:
path_prefix: '{year}//{month}//{day}/{ci}/{type}'
build_prefix: '{path_prefix}/{type_id}/{build_commit[:7]}'
ci:
github:
paths:
logs: '{build_prefix}/{wf_name}/{number}/logs/'
artifacts: '{build_prefix}/{wf_name}/{number}/artifacts/'
releases: '{path_prefix}/{release_tag}/'
workflows:
- test_crippled.yml
- test_extensions.yml
- test_macos.yml
travis:
paths:
logs: '{build_prefix}/{number}/{job}.txt'
appveyor:
paths:
logs: '{build_prefix}/{number}/{job}.txt'
accountName: mih
projectSlug: datalad
since: 2021-01-20T00:00:00Z
max-days-back: 14
types: [cron, manual, pr, push]
secrets:
github: '\bgh[a-z]_[A-Za-z0-9]{36,}\b'
docker-hub: '\b[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\b'
appveyor: '\b(v2\.)?[a-z0-9]{20}\b'
travis: '\b[a-zA-Z0-9]{22}\b'
aws: '\b[a-zA-Z0-9+/]{40}\b'
datalad:
enabled: true
cfg_proc: text2git
Path Templates
--------------
The path at which assets for a given workflow run, build job, or release are
saved is determined by instantiating the appropriate path template string given
in the configuration file for the corresponding CI system. A template string
is a filepath containing placeholders of the form ``{field}``, where the
available placeholders are:
====================== =======================================================
Placeholder Definition
====================== =======================================================
``{year}`` The four-digit year in which the build was started or
the release was published
``{month}`` The two-digit month in which the build was started or
the release was published
``{day}`` The two-digit day in which the build was started or the
release was published
``{hour}`` The two-digit hour at which the build was started or
the release was published
``{minute}`` The two-digit minute at which the build was started or
the release was published
``{second}`` The two-digit second at which the build was started or
the release was published
``{timestamp}`` The date & time at which the build was started or the
release was published. This is a Python datetime_
value; it can be formatted with a `strftime()`_ format
string by writing ``{timestamp:FORMAT}``, e.g.,
``{timestamp:%Y-%b-%d}`` will produce a string of the
form "2021-Jun-14". If written as just
``{timestamp}``, the date & time will be formatted in
ISO 8601 format.
``{timestamp_local}`` The date & time at which the build was started or the
release was published, in the local system timezone.
This is formatted in the same way as ``{timestamp}``.
``{ci}`` The name of the CI system (``github``, ``travis``, or
``appveyor``)
``{type}`` The event type that triggered the build (``cron``,
``manual``, ``pr``, or ``push``), or ``release`` for
GitHub releases
``{type_id}`` Further information on the triggering event; for
``cron`` and ``manual``, this is a timestamp for the
start of the build; for ``pr``, this is the number of
the associated pull request, or ``UNK`` if it cannot be
determined; for ``push``, this is the escaped [1]_ name
of the branch to which the push was made (or possibly
the tag that was pushed, if using Appveyor) [2]_
``{release_tag}`` *(``releases_path`` only)* The release tag
``{build_commit}`` The hash of the commit the build ran against or that
was tagged for the release. Note that, for PR builds
on Travis and Appveyor, this is the hash of an
autogenerated merge commit.
``{commit}`` The hash of the original commit that triggered the
build or that was tagged for the release. For pull
request builds, this is the head of the PR branch, or
``UNK`` if it cannot be determined. For other builds
(along with PR builds on GitHub Actions), this is
always the same as ``{build_commit}``.
``{number}`` The run number of the workflow run (GitHub) or the
build number (Travis and Appveyor) [2]_
``{status}`` The success status of the workflow run (GitHub) or job
(Travis and Appveyor); the exact strings used depend on
the CI system [2]_
``{common_status}`` The success status of the workflow run or job,
normalized into one of ``success``, ``failed``,
``errored``, or ``incomplete`` [2]_
``{wf_name}`` *(GitHub only)* The escaped [1]_ name of the workflow
[2]_
``{wf_file}`` *(GitHub only)* The basename of the workflow file
(including the file extension) [2]_
``{run_id}`` *(GitHub only)* The unique ID of the workflow run [2]_
``{job}`` *(Travis and Appveyor only)* The number of the job,
without the build number prefix (Travis) or the job ID
string (Appveyor) [2]_
``{job_index}`` *(Travis and Appveyor only)* The index of the job in
the list returned by the API, starting from 1 [2]_
``{job_env}`` *(Appveyor only)* The escaped [1]_ environment
variables specific to the job [2]_
``{job_env_hash}`` *(Appveyor only)* The SHA1 hash of ``{job_env}`` before
escaping [2]_
====================== =======================================================
.. _datetime: https://docs.python.org/3/library/datetime.html#datetime-objects
.. _strftime(): https://docs.python.org/3/library/datetime.html
#strftime-and-strptime-format-codes
.. [1] Escaping consists of percent-encoding the characters ``\/<>:|"?*%`` and
replacing each whitespace character with a space.
.. [2] These placeholders are only available for ``path`` and
``artifacts_path``, not ``releases_path``
A placeholder's value may be truncated to the first ``n`` characters by writing
``{placeholder[:n]}``, e.g., ``{commit[:7]}``.
All timestamps and timestamp components (other than ``{timestamp_local}``) are
in UTC.
Path templates may also contain custom placeholders defined in the top-level
``vars`` mapping of the configuration.
Authentication
--------------
Note that environment variables can be loaded from a ``.env`` file as an
alternative to setting them directly in the environment.
GitHub
~~~~~~
In order to retrieve assets from GitHub, a GitHub OAuth token must be specified
either via the ``GITHUB_TOKEN`` environment variable or as the value of the
``hub.oauthtoken`` Git config option.
Travis
~~~~~~
In order to retrieve logs from Travis, a Travis API access token must be either
specified via the ``TRAVIS_TOKEN`` environment variable or be retrievable by
running ``travis token --com --no-interactive``.
A Travis API access token can be acquired as follows:
- Install the `Travis command line client
<https://github.com/travis-ci/travis.rb>`_.
- Run ``travis login --com`` to authenticate.
- If your Travis account is linked to your GitHub account, you can
authenticate by running ``travis login --com --github-token
$GITHUB_TOKEN``.
- If the script will be run on the same machine that the above steps are
carried out on, you can stop here, and the script will retrieve the token
directly from the ``travis`` command.
- Run ``travis token --com`` to retrieve the API access token.
The Travis integration also requires a GitHub OAuth token in order to look up
information on pull requests that the Travis API does not report; this token
must be specified in the same way as for the GitHub integration.
Appveyor
~~~~~~~~
In order to retrieve logs from Appveyor, an Appveyor API key (for either all
accessible accounts or just the specific account associated with the
repository) must be specified via the ``APPVEYOR_TOKEN`` environment variable.
Such a key can be obtained at <https://ci.appveyor.com/api-keys>.
Cron Integration
================
If you want to set up scheduled runs of ``tinuous`` on a Linux server, one way
is as follows:
1. Create a new directory and ``cd`` into it.
2. Create a file named ``tinuous.yaml`` in this directory `as described above
<Configuration_>`_
3. Create a file named ``.env`` in this directory containing any needed
authentication tokens. Entries are of the form ``NAME=value``, e.g.::
GITHUB_TOKEN=ghp_abcdef0123456789
TRAVIS_TOKEN=asdfghjkl
APPVEYOR_TOKEN=v2.qwertyuiop
4. Create a Python virtualenv_ to provide an isolated environment to install
``tinuous`` into::
python3 -m venv venv
5. Install ``tinuous`` inside the virtualenv::
venv/bin/pip install tinuous
If you want to use DataLad with ``tinuous``, you need to install it as well,
even if it's already installed outside the virtualenv::
venv/bin/pip install datalad
6. Run ``tinuous`` to fetch your first logs and test your configuration::
venv/bin/tinuous fetch
7. Once you're satisfied with your ``tinuous`` config, set up scheduled runs by
creating a cronjob of the form::
0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock venv/bin/tinuous fetch
This job runs once a day at midnight; adjust the cron expression to taste.
We use ``chronic`` (from moreutils_) to suppress output unless the command
fails, thus preventing e-mails full of log messages for every run.
``flock`` is used to ensure that no more than one instance of ``tinuous`` is
running at a time.
8. If you want to commit your logs to a Git repository, first make sure that
``.env``, ``venv/``, and ``.tinuous.lock`` are included in the repository's
``.gitignore``. Consider setting up the repository with DataLad_; when the
DataLad integration is enabled, ``tinuous`` will automatically commit any
new logs at the end of a run.
If you're using a regular Git repository instead, you can commit any new
logs at the end of a run by adding the following script to your ``tinuous``
directory:
.. code:: bash
#!/bin/bash
set -ex
venv/bin/tinuous fetch
git add --all
if ! git diff --cached --quiet
then git commit -m "Ran tinuous"
# Uncomment if you want to push the commits to a remote repository:
#git push
fi
and changing your cronjob to::
0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock bash name-of-script.sh
9. If you ever need to upgrade ``tinuous``, run the following command inside
your ``tinuous`` directory::
venv/bin/pip install --upgrade tinuous
10. Enjoy your collection of logs, build artifacts, and/or release assets!
.. _virtualenv: https://packaging.python.org/guides/installing-using-pip-and
-virtual-environments/
.. _moreutils: https://joeyh.name/code/moreutils/
Raw data
{
"_id": null,
"home_page": "https://github.com/con/tinuous",
"name": "tinuous",
"maintainer": "John T. Wodder II",
"docs_url": null,
"requires_python": "~=3.8",
"maintainer_email": "tinuous@varonathe.org",
"keywords": "Appveyor,GitHub Actions,Travis CI,continuous integration",
"author": "Center for Open Neuroscience",
"author_email": "debian@onerussian.com",
"download_url": "https://files.pythonhosted.org/packages/9b/c0/de2c925c765eae01884854d77c61a142eab1dcf0dd63bbd679b21e5ca624/tinuous-0.6.1.tar.gz",
"platform": null,
"description": ".. image:: https://github.com/con/tinuous/workflows/Test/badge.svg?branch=master\n :target: https://github.com/con/tinuous/actions?workflow=Test\n :alt: GitHub Actions Status\n\n.. image:: https://travis-ci.com/con/tinuous.svg?branch=master\n :target: https://travis-ci.com/con/tinuous\n :alt: Travis CI Status\n\n.. image:: https://ci.appveyor.com/api/projects/status/github/con/tinuous?branch=master&svg=true\n :target: https://ci.appveyor.com/project/yarikoptic/tinuous/branch/master\n :alt: Appveyor Status\n\n.. image:: https://img.shields.io/pypi/pyversions/tinuous.svg\n :target: https://pypi.org/project/tinuous/\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/tinuous.svg\n :target: https://anaconda.org/conda-forge/tinuous\n :alt: Conda Version\n\n.. image:: https://img.shields.io/github/license/con/tinuous.svg\n :target: https://opensource.org/licenses/MIT\n :alt: MIT License\n\n`GitHub <https://github.com/con/tinuous>`_\n| `PyPI <https://pypi.org/project/tinuous/>`_\n| `Anaconda <https://anaconda.org/conda-forge/tinuous>`_\n| `Issues <https://github.com/con/tinuous/issues>`_\n| `Changelog <https://github.com/con/tinuous/blob/master/CHANGELOG.md>`_\n\n``tinuous`` is a command for downloading build logs and (for GitHub\nonly) artifacts & release assets for a GitHub repository from GitHub Actions,\nTravis-CI.com, and/or Appveyor.\n\nSee <https://github.com/con/tinuous-inception> for an example setup that uses\n``tinuous`` with GitHub Actions to fetch the CI logs for ``tinuous`` itself.\n\nInstallation\n============\n``tinuous`` requires Python 3.8 or higher. Just use `pip\n<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install\n``tinuous`` and its dependencies::\n\n python3 -m pip install tinuous\n\n``tinuous`` can also optionally integrate with DataLad_. To install DataLad\nalongside ``tinuous``, specify the ``datalad`` extra::\n\n python3 -m pip install \"tinuous[datalad]\"\n\n``tinuous`` is also available for conda! To install, run::\n\n conda install -c conda-forge tinuous\n\n\nUsage\n=====\n\n::\n\n tinuous [<global options>] <command> [<args> ...]\n\n\nGlobal Options\n--------------\n\n-c FILE, --config FILE Read configuration from the given file [default\n value: ``tinuous.yaml``]\n\n-E FILE, --env FILE Load environment variables from the given\n ``.env`` file. By default, environment\n variables are loaded from the first file named\n \"``.env``\" found by searching from the current\n directory upwards.\n\n **Warning**: Care must be taken when this file\n is located in a Git repository so as not to\n publicly expose it: either list the file in\n ``.gitignore`` or, if using DataLad or\n git-annex, configure git-annex to prohibit\n public sharing of the file.\n\n-l LEVEL, --log-level LEVEL Set the log level to the given value. Possible\n values are \"``CRITICAL``\", \"``ERROR``\",\n \"``WARNING``\", \"``INFO``\", \"``DEBUG``\" (all\n case-insensitive) and their Python integer\n equivalents. [default value: INFO]\n\n\n``fetch`` Command\n-----------------\n\n::\n\n tinuous [<global options>] fetch [<options>]\n\n``tinuous fetch`` reads a configuration file telling it what repository to\nretrieve logs & artifacts for, where to retrieve them from, and where to save\nthem, and then it carries those steps out.\n\nOptions\n~~~~~~~\n\n--sanitize-secrets Sanitize secrets from log files after\n downloading\n\n-S FILE, --state FILE Store program state (e.g., timestamps before\n which all asset are known to have been fetched)\n in the given file [default value:\n ``.tinuous.state.json``]\n\n``sanitize`` Command\n--------------------\n\n::\n\n tinuous [<global options>] sanitize <path> ...\n\nSanitize the given files, replacing all strings matching a secret regex with a\nseries of asterisks.\n\n\nConfiguration\n-------------\n\nThe configuration file is a YAML file containing a mapping with the following\nkeys:\n\n``repo``\n *(required)* The GitHub repository to retrieve assets for, in the form ``OWNER/NAME``\n\n``vars``\n A mapping defining custom path template placeholders. Each key is the name\n of a custom placeholder, without enclosing braces, and the value is the\n string to substitute in its place. Custom values may contain standard path\n template placeholders as well as other custom placeholders.\n\n``ci``\n *(required)* A mapping from the names of the CI systems from which to\n retrieve assets to sub-mappings containing CI-specific configuration.\n Including a given CI system is optional; assets will only be fetched from a\n given system if it is listed in this mapping.\n\n The CI systems and their sub-mappings are as follows:\n\n ``github``\n Configuration for retrieving assets from GitHub Actions. Subfields:\n\n ``paths``\n A mapping giving `template strings <Path Templates_>`_ for the\n paths at which to save various types of assets. If this is empty\n or not present, no assets are retrieved. Subfields:\n\n ``logs``\n A template string that will be instantiated for each workflow\n run to produce the path for the directory (relative to the\n current working directory) under which the run's build logs\n will be saved. If this is not specified, no logs will be\n downloaded.\n\n ``artifacts``\n A template string that will be instantiated for each workflow\n run to produce the path for the directory (relative to the\n current working directory) under which the run's artifacts will\n be saved. If this is not specified, no artifacts will be\n downloaded.\n\n ``releases``\n A template string that will be instantiated for each\n (non-draft, non-prerelease) GitHub release to produce the path\n for the directory (relative to the current working directory)\n under which the release's assets will be saved. If this is not\n specified, no release assets will be downloaded.\n\n ``workflows``\n A specification of the workflows for which to retrieve assets.\n This can be either a list of workflow basenames, including the file\n extension (e.g., ``test.yml``, not ``.github/workflows/test.yml``)\n or a mapping containing the following fields:\n\n ``include``\n A list of workflows to retrieve assets for, given as either\n basenames or (when ``regex`` is true) `Python regular\n expressions`_ to match against basenames. If ``include`` is\n omitted, it defaults to including all workflows.\n\n ``exclude``\n A list of workflows to not retrieve assets for, given as either\n basenames or (when ``regex`` is true) `Python regular\n expressions`_ to match against basenames. If ``exclude`` is\n omitted, no workflows are excluded. Workflows that match both\n ``include`` and ``exclude`` are excluded.\n\n ``regex``\n A boolean. If true (default false), the elements of the\n ``include`` and ``exclude`` fields are treated as `Python\n regular expressions`_ that are matched (unanchored) against\n workflow basenames; if false, they are used as exact names\n\n When ``workflows`` is not specified, assets are retrieved for all\n workflows in the repository.\n\n ``travis``\n Configuration for retrieving logs from Travis-CI.com. Subfield:\n\n ``paths``\n A mapping giving `template strings <Path Templates_>`_ for the\n paths at which to save various types of assets. If this is empty\n or not present, no assets are retrieved. Subfield:\n\n ``logs``\n A template string that will be instantiated for each job of\n each build to produce the path for the file (relative to the\n current working directory) in which the job's logs will be\n saved. If this is not specified, no logs will be downloaded.\n\n ``appveyor``\n Configuration for retrieving logs from Appveyor. Subfields:\n\n ``paths``\n A mapping giving `template strings <Path Templates_>`_ for the\n paths at which to save various types of assets. If this is empty\n or not present, no assets are retrieved. Subfield:\n\n ``logs``\n A template string that will be instantiated for each job of\n each build to produce the path for the file (relative to the\n current working directory) in which the job's logs will be\n saved. If this is not specified, no logs will be downloaded.\n\n ``accountName``\n *(required)* The name of the Appveyor account to which the\n repository belongs on Appveyor\n\n ``projectSlug``\n The project slug for the repository on Appveyor; if not specified,\n it is assumed that the slug is the same as the repository name\n\n``since``\n A timestamp (date, time, & timezone); only assets for builds started after\n the given point in time will be retrieved. If not specified, the cutoff\n set by ``max-days-back`` will be used.\n\n As the script retrieves new build assets, it keeps track of their starting\n points. Once the assets for all builds for the given CI system &\n configuration have been fetched up to a certain point, the timestamp for\n the latest such build is stored in the state file and used as the new\n ``since`` value for the respective CI system on subsequent runs. If the\n ``since`` setting in the configuration file is then updated to a newer\n timestamp, the configuration will override the value in the state file, and\n the next ``tinuous`` run will only retrieve assets after the new setting.\n\n``max-days-back``\n An integer specifying the maximum number of days back to look for builds;\n defaults to 30.\n\n If ``since`` is earlier than the date indicated by this value, and if the\n timestamp for a given CI system saved in the state file is either missing\n (i.e., if this is the first run of ``tinuous`` against the CI system) or\n older than ``since``, then ``tinuous`` will use the explicitly-specified\n ``since`` value as the cutoff and ignore ``max-days-back``.\n\n``until``\n A timestamp (date, time, & timezone); only assets for builds started before\n the given point in time will be retrieved\n\n``types``\n A list of build trigger event types; only assets for builds triggered by\n one of the given events will be retrieved. If this is not specified,\n assets will be retrieved for all recognized event types.\n\n The recognized event types are:\n\n ``cron``\n A build run on a schedule\n\n ``manual``\n A build triggered manually by a human or through the CI system's API\n\n ``pr``\n A build in response to activity on a pull request\n\n ``push``\n A build in response to new commits\n\n``secrets``\n A mapping from names (used in log messages) to `Python regular\n expressions`_ matching secrets to sanitize\n\n``allow-secrets-regex``\n Any strings that match a ``secrets`` regex and also match this regex will\n not be sanitized. Note that ``allow-secrets-regex`` is tested against just\n the substring that matched a ``secrets`` regex without any surrounding\n text, and so lookahead and lookbehind will not work in this regex.\n\n``datalad``\n A sub-mapping describing integration of ``tinuous`` with DataLad_.\n Subfields:\n\n ``enabled``\n A boolean. If true (default false), DataLad must be installed, the\n current directory will be converted into a DataLad dataset if it is not\n one already, the assets will optionally be divided up into subdatasets,\n and all new assets will be committed at the end of a run of ``tinuous\n fetch``. ``path`` template strings may contain ``//`` separators\n indicating the boundaries of subdatasets.\n\n ``cfg_proc``\n Procedure to run on the dataset & subdatasets when creating them\n\n .. _DataLad: https://www.datalad.org\n\n.. _Python regular expressions: https://docs.python.org/3/library/re.html\n #regular-expression-syntax\n\nA sample config file:\n\n.. code:: yaml\n\n repo: datalad/datalad\n vars:\n path_prefix: '{year}//{month}//{day}/{ci}/{type}'\n build_prefix: '{path_prefix}/{type_id}/{build_commit[:7]}'\n ci:\n github:\n paths:\n logs: '{build_prefix}/{wf_name}/{number}/logs/'\n artifacts: '{build_prefix}/{wf_name}/{number}/artifacts/'\n releases: '{path_prefix}/{release_tag}/'\n workflows:\n - test_crippled.yml\n - test_extensions.yml\n - test_macos.yml\n travis:\n paths:\n logs: '{build_prefix}/{number}/{job}.txt'\n appveyor:\n paths:\n logs: '{build_prefix}/{number}/{job}.txt'\n accountName: mih\n projectSlug: datalad\n since: 2021-01-20T00:00:00Z\n max-days-back: 14\n types: [cron, manual, pr, push]\n secrets:\n github: '\\bgh[a-z]_[A-Za-z0-9]{36,}\\b'\n docker-hub: '\\b[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\\b'\n appveyor: '\\b(v2\\.)?[a-z0-9]{20}\\b'\n travis: '\\b[a-zA-Z0-9]{22}\\b'\n aws: '\\b[a-zA-Z0-9+/]{40}\\b'\n datalad:\n enabled: true\n cfg_proc: text2git\n\n\nPath Templates\n--------------\n\nThe path at which assets for a given workflow run, build job, or release are\nsaved is determined by instantiating the appropriate path template string given\nin the configuration file for the corresponding CI system. A template string\nis a filepath containing placeholders of the form ``{field}``, where the\navailable placeholders are:\n\n====================== =======================================================\nPlaceholder Definition\n====================== =======================================================\n``{year}`` The four-digit year in which the build was started or\n the release was published\n``{month}`` The two-digit month in which the build was started or\n the release was published\n``{day}`` The two-digit day in which the build was started or the\n release was published\n``{hour}`` The two-digit hour at which the build was started or\n the release was published\n``{minute}`` The two-digit minute at which the build was started or\n the release was published\n``{second}`` The two-digit second at which the build was started or\n the release was published\n``{timestamp}`` The date & time at which the build was started or the\n release was published. This is a Python datetime_\n value; it can be formatted with a `strftime()`_ format\n string by writing ``{timestamp:FORMAT}``, e.g.,\n ``{timestamp:%Y-%b-%d}`` will produce a string of the\n form \"2021-Jun-14\". If written as just\n ``{timestamp}``, the date & time will be formatted in\n ISO 8601 format.\n``{timestamp_local}`` The date & time at which the build was started or the\n release was published, in the local system timezone.\n This is formatted in the same way as ``{timestamp}``.\n``{ci}`` The name of the CI system (``github``, ``travis``, or\n ``appveyor``)\n``{type}`` The event type that triggered the build (``cron``,\n ``manual``, ``pr``, or ``push``), or ``release`` for\n GitHub releases\n``{type_id}`` Further information on the triggering event; for\n ``cron`` and ``manual``, this is a timestamp for the\n start of the build; for ``pr``, this is the number of\n the associated pull request, or ``UNK`` if it cannot be\n determined; for ``push``, this is the escaped [1]_ name\n of the branch to which the push was made (or possibly\n the tag that was pushed, if using Appveyor) [2]_\n``{release_tag}`` *(``releases_path`` only)* The release tag\n``{build_commit}`` The hash of the commit the build ran against or that\n was tagged for the release. Note that, for PR builds\n on Travis and Appveyor, this is the hash of an\n autogenerated merge commit.\n``{commit}`` The hash of the original commit that triggered the\n build or that was tagged for the release. For pull\n request builds, this is the head of the PR branch, or\n ``UNK`` if it cannot be determined. For other builds\n (along with PR builds on GitHub Actions), this is\n always the same as ``{build_commit}``.\n``{number}`` The run number of the workflow run (GitHub) or the\n build number (Travis and Appveyor) [2]_\n``{status}`` The success status of the workflow run (GitHub) or job\n (Travis and Appveyor); the exact strings used depend on\n the CI system [2]_\n``{common_status}`` The success status of the workflow run or job,\n normalized into one of ``success``, ``failed``,\n ``errored``, or ``incomplete`` [2]_\n``{wf_name}`` *(GitHub only)* The escaped [1]_ name of the workflow\n [2]_\n``{wf_file}`` *(GitHub only)* The basename of the workflow file\n (including the file extension) [2]_\n``{run_id}`` *(GitHub only)* The unique ID of the workflow run [2]_\n``{job}`` *(Travis and Appveyor only)* The number of the job,\n without the build number prefix (Travis) or the job ID\n string (Appveyor) [2]_\n``{job_index}`` *(Travis and Appveyor only)* The index of the job in\n the list returned by the API, starting from 1 [2]_\n``{job_env}`` *(Appveyor only)* The escaped [1]_ environment\n variables specific to the job [2]_\n``{job_env_hash}`` *(Appveyor only)* The SHA1 hash of ``{job_env}`` before\n escaping [2]_\n====================== =======================================================\n\n.. _datetime: https://docs.python.org/3/library/datetime.html#datetime-objects\n.. _strftime(): https://docs.python.org/3/library/datetime.html\n #strftime-and-strptime-format-codes\n\n.. [1] Escaping consists of percent-encoding the characters ``\\/<>:|\"?*%`` and\n replacing each whitespace character with a space.\n\n.. [2] These placeholders are only available for ``path`` and\n ``artifacts_path``, not ``releases_path``\n\nA placeholder's value may be truncated to the first ``n`` characters by writing\n``{placeholder[:n]}``, e.g., ``{commit[:7]}``.\n\nAll timestamps and timestamp components (other than ``{timestamp_local}``) are\nin UTC.\n\nPath templates may also contain custom placeholders defined in the top-level\n``vars`` mapping of the configuration.\n\nAuthentication\n--------------\n\nNote that environment variables can be loaded from a ``.env`` file as an\nalternative to setting them directly in the environment.\n\nGitHub\n~~~~~~\n\nIn order to retrieve assets from GitHub, a GitHub OAuth token must be specified\neither via the ``GITHUB_TOKEN`` environment variable or as the value of the\n``hub.oauthtoken`` Git config option.\n\nTravis\n~~~~~~\n\nIn order to retrieve logs from Travis, a Travis API access token must be either\nspecified via the ``TRAVIS_TOKEN`` environment variable or be retrievable by\nrunning ``travis token --com --no-interactive``.\n\nA Travis API access token can be acquired as follows:\n\n- Install the `Travis command line client\n <https://github.com/travis-ci/travis.rb>`_.\n\n- Run ``travis login --com`` to authenticate.\n\n - If your Travis account is linked to your GitHub account, you can\n authenticate by running ``travis login --com --github-token\n $GITHUB_TOKEN``.\n\n- If the script will be run on the same machine that the above steps are\n carried out on, you can stop here, and the script will retrieve the token\n directly from the ``travis`` command.\n\n- Run ``travis token --com`` to retrieve the API access token.\n\nThe Travis integration also requires a GitHub OAuth token in order to look up\ninformation on pull requests that the Travis API does not report; this token\nmust be specified in the same way as for the GitHub integration.\n\nAppveyor\n~~~~~~~~\n\nIn order to retrieve logs from Appveyor, an Appveyor API key (for either all\naccessible accounts or just the specific account associated with the\nrepository) must be specified via the ``APPVEYOR_TOKEN`` environment variable.\nSuch a key can be obtained at <https://ci.appveyor.com/api-keys>.\n\n\nCron Integration\n================\n\nIf you want to set up scheduled runs of ``tinuous`` on a Linux server, one way\nis as follows:\n\n1. Create a new directory and ``cd`` into it.\n\n2. Create a file named ``tinuous.yaml`` in this directory `as described above\n <Configuration_>`_\n\n3. Create a file named ``.env`` in this directory containing any needed\n authentication tokens. Entries are of the form ``NAME=value``, e.g.::\n\n GITHUB_TOKEN=ghp_abcdef0123456789\n TRAVIS_TOKEN=asdfghjkl\n APPVEYOR_TOKEN=v2.qwertyuiop\n\n4. Create a Python virtualenv_ to provide an isolated environment to install\n ``tinuous`` into::\n\n python3 -m venv venv\n\n5. Install ``tinuous`` inside the virtualenv::\n\n venv/bin/pip install tinuous\n\n If you want to use DataLad with ``tinuous``, you need to install it as well,\n even if it's already installed outside the virtualenv::\n\n venv/bin/pip install datalad\n\n6. Run ``tinuous`` to fetch your first logs and test your configuration::\n\n venv/bin/tinuous fetch\n\n7. Once you're satisfied with your ``tinuous`` config, set up scheduled runs by\n creating a cronjob of the form::\n\n 0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock venv/bin/tinuous fetch\n\n This job runs once a day at midnight; adjust the cron expression to taste.\n We use ``chronic`` (from moreutils_) to suppress output unless the command\n fails, thus preventing e-mails full of log messages for every run.\n ``flock`` is used to ensure that no more than one instance of ``tinuous`` is\n running at a time.\n\n8. If you want to commit your logs to a Git repository, first make sure that\n ``.env``, ``venv/``, and ``.tinuous.lock`` are included in the repository's\n ``.gitignore``. Consider setting up the repository with DataLad_; when the\n DataLad integration is enabled, ``tinuous`` will automatically commit any\n new logs at the end of a run.\n\n If you're using a regular Git repository instead, you can commit any new\n logs at the end of a run by adding the following script to your ``tinuous``\n directory:\n\n .. code:: bash\n\n #!/bin/bash\n set -ex\n venv/bin/tinuous fetch\n git add --all\n if ! git diff --cached --quiet\n then git commit -m \"Ran tinuous\"\n # Uncomment if you want to push the commits to a remote repository:\n #git push\n fi\n\n and changing your cronjob to::\n\n 0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock bash name-of-script.sh\n\n9. If you ever need to upgrade ``tinuous``, run the following command inside\n your ``tinuous`` directory::\n\n venv/bin/pip install --upgrade tinuous\n\n10. Enjoy your collection of logs, build artifacts, and/or release assets!\n\n.. _virtualenv: https://packaging.python.org/guides/installing-using-pip-and\n -virtual-environments/\n\n.. _moreutils: https://joeyh.name/code/moreutils/\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Download build logs from GitHub Actions, Travis, and Appveyor",
"version": "0.6.1",
"split_keywords": [
"appveyor",
"github actions",
"travis ci",
"continuous integration"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f7d2fcaaedc3ed27b6f841177433fe50c808fd07895a42a1eb1ca2156b1b1aa8",
"md5": "cea5d736e9dc5464f7f67856f1f2e64d",
"sha256": "49640b089ce18d741b690936fb5c08efb2d3b3edc3bdf58fe80aee7706b0203e"
},
"downloads": -1,
"filename": "tinuous-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cea5d736e9dc5464f7f67856f1f2e64d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "~=3.8",
"size": 29703,
"upload_time": "2023-02-09T22:47:01",
"upload_time_iso_8601": "2023-02-09T22:47:01.454911Z",
"url": "https://files.pythonhosted.org/packages/f7/d2/fcaaedc3ed27b6f841177433fe50c808fd07895a42a1eb1ca2156b1b1aa8/tinuous-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9bc0de2c925c765eae01884854d77c61a142eab1dcf0dd63bbd679b21e5ca624",
"md5": "c6a180a93f0e8e314f9d3ce4a81a147d",
"sha256": "5c0ac2907cadff890338c271be31fa59cb291f02f9879e18a0fa43be62da447b"
},
"downloads": -1,
"filename": "tinuous-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "c6a180a93f0e8e314f9d3ce4a81a147d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "~=3.8",
"size": 55100,
"upload_time": "2023-02-09T22:47:03",
"upload_time_iso_8601": "2023-02-09T22:47:03.463853Z",
"url": "https://files.pythonhosted.org/packages/9b/c0/de2c925c765eae01884854d77c61a142eab1dcf0dd63bbd679b21e5ca624/tinuous-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-09 22:47:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "con",
"github_project": "tinuous",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"appveyor": true,
"circle": true,
"tox": true,
"lcname": "tinuous"
}
JSON
