=====================
psec (python_secrets)
=====================
.. image:: https://img.shields.io/pypi/v/python_secrets.svg
:target: https://pypi.python.org/pypi/python_secrets
.. image:: https://img.shields.io/travis/davedittrich/python_secrets.svg
:target: https://travis-ci.org/davedittrich/python_secrets
.. image:: https://readthedocs.org/projects/python-secrets/badge/?version=latest
:target: https://python-secrets.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
Python command line app for managing groups of secrets (passwords, API keys, etc) and
other project variables. Reduces security risks from things like weak default passwords,
secrets stored in files in the source code repository directory.
Version: 23.4.1
* Free software: `Apache 2.0 License <https://www.apache.org/licenses/LICENSE-2.0>`_
* Documentation: https://python_secrets.readthedocs.org.
Features
--------
* Uses the `openstack/cliff`_ command line framework for a robust and
full-featured CLI. It is easy to add new commands and features!
* Supports a "drop-in" model for defining variables in a modular manner
(something like the `python-update-dotdee`_ program), supporting simplified
bulk setting or generating values of variables as needed.
* Like `python-update-dotdee`_, `psec` produces a single master
``.json`` file to hold variables defined by the drop-in group
description files. That means you can use that file directly
to set variables to be used from within other programs like
Ansible (e.g. ``ansible-playbook playbook.yml -e @"$(psec secrets path)"``)
* Support multiple simultaneous sets of secrets (environments) for
flexibility and scalability in multi-environment deployments and to
support different use cases or different combinations of secrets.
* Supports changing the storage location of secrets and variables to
allow them to be stored on secure mobile media (such as self-encrypting
external SSD or Flash drives) or encrypted disk images mounted at
run-time to ensure the confidentiality of data at rest.
* List the groups of variables (and how many secrets in each group).
* Describe secrets by their variable name and type (e.g., ``password``,
``uuid4``, ``random_base64``). You can also include a descriptive
string to prompt the user for a value, a list of options to choose
from (or ``*`` for "any value the user enters"), and a list of
environment variables to export for other programs to use at
run time.
* Allows manual entry of values, setting non-secret variables from
a default value, or automatic generation of secrets according to
their type.
* Manually set ``string`` variables based on the output of simple
commands. This allows interfacing with external programs for
obtaining secrets, such as `Vault by Hashicorp`_.
* Generate unique values for variables, or use a single value per
type to simplify use of secrets in access control of services
while supporting a "break-glass" process to quickly regenerate
secrets when needed.
* Show the variables and their unredacted values (or redacted them
to maintain secrecy during demonstrations or in documentation).
* Export the variables (optionally with a specific prefix string)
to the environment and run a command that inherits them (e.g.,
to pass variables to `terraform`_ for provisioning cloud
instances).
* Output the variables and values in multiple different formats (CSV,
JSON, YAML) for use in shell scripts, etc. using ``cliff`` features.
* Send secrets to other users on demand using GPG encrypted email to
protect the secrets while in transit and while at rest in users'
email inboxes.
* Makes it easy to store temporary files (e.g., the output from
Jinja template rendering)
that may contain secrets *outside* of the source repo directory
in an environment-specific ``tmp/`` directory.
.. note::
Due to the use of the Python ``secrets`` module, which was introduced
in Python 3.6, only Python versions >= 3.6 can be used.
..
.. _limitations:
Limitations
-----------
* Secrets are stored in *unencrypted* form in the environments
directories. Permissions are set to limit access, but this is not an
"encrypt data at rest" solution like `Vault by Hashicorp`_.
* Does not handle secure distributed access for users on remote systems. You
must use something like `Vault by Hashicorp`_ or `libfuse/sshfs`_ for secure
(realtime) distributed access.
* Does not handle secure distribution of newly generated secrets out
to distributed systems that need them. You will need to use a program
like `Ansible`_ and related playbooks for pushing out and changing
secrets (or for retrieving backups). Look at the `D2 Ansible
playbooks`_ (https://github.com/davedittrich/ansible-dims-playbooks)
for example playbooks for doing these tasks.
* Does not clean up the environment-specific ``tmp/`` directories.
(You need to handle that in code, but at least they are less likely
to end up in a Git commit.)
Usage Concepts
--------------
There is a separate **Usage** chapter with individual command documentation.
The remainder of this section covers higher level usage concepts necessary to
best use the ``python_secrets`` package in your open source software project.
Directories and files
~~~~~~~~~~~~~~~~~~~~~
There are three file system concepts that are important to understand
regarding secrets storage:
#. The root *secrets base directory* for secrets storage;
#. The *environment* for organizing a set of secrets and
secret group descriptions;
#. The *secrets* file and *group descriptions*.
.. image:: https://asciinema.org/a/201503.png
:target: https://asciinema.org/a/201503?autoplay=1
:align: center
:alt: Environments
:width: 835px
..
Secrets Base Directory
^^^^^^^^^^^^^^^^^^^^^^
``psec`` expects to store all of files in a directory tree known as a
*secrets base directory*. Originally, this was intended to be located in the
current user's home directory. Unless you over-ride the name of this directory,
it defaults to ``.secrets`` on Linux and ``secrets`` on Windows.
The ability to locate this directory in a different file system path is
supported by command line options and an environment variable so you can store
files on an exported file share, in a common location for use by a group on a
workstation, or to move the contents to an encrypted disk or a different
partition with more disk space.
The first time you use ever use ``psec``, there will likely be no
directory:
.. code-block:: console
$ tree ~/.secrets
/Users/dittrich/.secrets [error opening dir]
0 directories, 0 files
..
.. note::
The secrets base directory may be created automatically for you the
first time you create an environment. For more information, see
``psec init --help``.
..
Environments
^^^^^^^^^^^^
Environments are sub-directories within the root secrets directory. You can
just create the directory structure without any files. You create
one environment per set of unique secrets that you need to manage. This could
be one for open source *Program A*, one for *Program B*, etc., or it could be
one for *development*, one for *testing*, one for *production*, etc. (or any
combination).
.. image:: https://asciinema.org/a/201505.png
:target: https://asciinema.org/a/201505?autoplay=1
:align: center
:alt: Groups, secrets, generating and setting
:width: 835px
..
The command ``environments create`` creates an environment. Since this
program is designed to support multiple environments, a name for the new
environment is required. The name of the environment can be provided
explicitly, or it can be inferred from the base name of the current working
directory:
.. code-block:: console
$ pwd
/Users/dittrich/git/python_secrets
$ psec environments create
environment directory /Users/dittrich/.secrets/python_secrets created
$ tree ~/.secrets
/Users/dittrich/.secrets
└── python_secrets
└── secrets.d
2 directories, 0 files
..
Let's say we want to create empty environments for the three deployments
(*development*, *testing*, and *production*). The names can be assigned
explicitly by (a) giving an argument on the command line, (b) using the ``-e`` or
``--environment`` command line flag, or (c) by setting the environment variable
``D2_ENVIRONMENT``:
.. code-block:: console
$ psec environments create development
environment directory /Users/dittrich/.secrets/development created
$ psec --environment testing environments create
environment directory /Users/dittrich/.secrets/testing created
$ D2_ENVIRONMENT=production psec environments create
environment directory /Users/dittrich/.secrets/production created
$ tree ~/.secrets
/Users/dittrich/.secrets
├── development
│ └── secrets.d
├── production
│ └── secrets.d
├── python_secrets
│ └── secrets.d
└── testing
└── secrets.d
8 directories, 0 files
..
If you want to create more than one environment at once, you will
have to specify all of the names on the command line as arguments:
.. code-block:: console
$ psec environments create development testing production
environment directory /Users/dittrich/.secrets/development created
environment directory /Users/dittrich/.secrets/testing created
environment directory /Users/dittrich/.secrets/production created
..
If you are using one source repository for building multiple deployments, of
course you can't rely on the basename of the directory for all deployments. The
default environment can be set, shown, or unset, using the ``environments
default`` command.
.. code-block:: console
$ psec environments default --help
usage: psec environments default [-h] [--unset-default] [environment]
Manage default environment via file in cwd
positional arguments:
environment
optional arguments:
-h, --help show this help message and exit
--unset-default Unset localized environment default
..
If no default is explicitly set, the default that would be
applied is returned:
.. code-block:: console
$ cd ~/git/python_secrets
$ psec environments default
default environment is "python_secrets"
..
You can get a list of all available environments at any time,
including which one would be the default used by sub-commands:
.. code-block:: console
$ psec environments list
+-------------+---------+
| Environment | Default |
+-------------+---------+
| development | No |
| testing | No |
| production | No |
+-------------+---------+
..
The following shows setting and unsetting the default:
.. code-block:: console
$ psec environments default testing
default environment set to "testing"
$ psec environments default
testing
$ psec environments list
+-------------+---------+
| Environment | Default |
+-------------+---------+
| development | No |
| testing | Yes |
| production | No |
+-------------+---------+
$ psec environments default --unset-default
default environment unset
..
The environment directories are useable for storing *all* secrets and
sensitive files (e.g., backups of certificates, databases, etc.) associated
with an environment.
For convenience, there is a command ``environments tree`` that produces
output similar to the Unix ``tree`` command:
.. code-block:: console
$ psec -e d2 environments tree
/Users/dittrich/.secrets/d2
├── backups
│ ├── black.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-06T23:36:58PDT.tgz
│ │ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ ├── green.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-06T23:45:49PDT.tgz
│ │ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ ├── purple.secretsmgmt.tk
│ │ ├── letsencrypt_2018-04-25T16:32:20PDT.tgz
│ │ ├── trident_2018-01-31T23:38:48PST.tar.bz2
│ │ └── trident_2018-02-04T20:05:33PST.tar.bz2
│ └── red.secretsmgmt.tk
│ ├── letsencrypt_2018-04-06T23:45:49PDT.tgz
│ └── letsencrypt_2018-04-25T16:32:20PDT.tgz
├── dittrich.asc
├── keys
│ └── opendkim
│ └── secretsmgmt.tk
│ ├── 201801.private
│ ├── 201801.txt
│ ├── 201802.private
│ └── 201802.txt
├── secrets.d
│ ├── ca.json
│ ├── consul.json
│ ├── jenkins.json
│ ├── rabbitmq.json
│ ├── trident.json
│ ├── vncserver.json
│ └── zookeper.json
├── secrets.json
└── vault_password.txt
..
To just see the directory structure and not files, add the ``--no-files`` option:
.. code-block:: console
$ psec -e d2 environments tree --no-files
/Users/dittrich/.secrets/d2
├── backups
│ ├── black.secretsmgmt.tk
│ ├── green.secretsmgmt.tk
│ ├── purple.secretsmgmt.tk
│ └── red.secretsmgmt.tk
├── keys
│ └── opendkim
│ └── secretsmgmt.tk
└── secrets.d
..
Secrets and group descriptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The environment directories just created are all empty. Secrets are stored in a
JSON file (``.json``) within the environment's directory, and group descriptions
are stored in a drop-in directory with the same base name, but with an
extention of ``.d`` instead of ``.json`` (following the Linux drop-in
configuration style directories used by programs like ``rsyslog``, ``dnsmasq``,
etc.)
The default secrets file name is ``secrets.json``, which means the default
descriptions directory would be named ``secrets.d``.
You can define environment variables to point to the secrets base directory
in which a set of different environments can be configured at one
time, to define the current environment, and to change the name
of the secrets file to something else.
.. code-block:: console
$ env | grep ^D2_
D2_SECRETS_BASEDIR=/Users/dittrich/.secrets
D2_ENVIRONMENT=do
..
Each environment is in turn rooted in a directory with the environment's
symbolic name (e.g., ``do`` for DigitalOcean in this example, and ``goSecure``
for the GitHub `davedittrich/goSecure`_ VPN project.)
.. code-block:: console
$ tree -L 1 ~/.secrets
/Users/dittrich/.secrets
├── do
└── goSecure
3 directories, 0 files
..
Each set of secrets for a given service or purpose is described in its own
file.
.. code-block:: console
.
├── secrets.d
│ ├── ca.json
│ ├── consul.json
│ ├── jenkins.json
│ ├── rabbitmq.json
│ ├── trident.json
│ ├── vncserver.json
│ └── zookeper.json
└── secrets.json
..
You can see one of the descriptions files from the template
in this repository using ``cat tests/secrets.d/myapp.json``:
.. code-block:: json
[
{
"Variable": "myapp_pi_password",
"Type": "password",
"Prompt": "Password for myapp 'pi' user account",
"Export": "DEMO_pi_password"
},
{
"Variable": "myapp_app_password",
"Type": "password",
"Prompt": "Password for myapp web app",
"Export": "DEMO_app_password"
},
{
"Variable": "myapp_client_psk",
"Type": "string",
"Prompt": "Pre-shared key for myapp client WiFi AP",
"Options": "*",
"Export": "DEMO_client_psk"
},
{
"Variable": "myapp_client_ssid",
"Type": "string",
"Prompt": "SSID for myapp client WiFi AP",
"Options": "myapp_ssid,*",
"Export": "DEMO_client_ssid"
},
{
"Variable": "myapp_ondemand_wifi",
"Type": "boolean",
"Prompt": "'Connect on demand' when connected to wifi",
"Options": "true,false",
"Export": "DEMO_ondemand_wifi"
},
{
"Variable": "myapp_optional_setting",
"Type": "boolean",
"Prompt": "Optionally do something",
"Options": "false,true",
"Export": "DEMO_options_setting"
}
]
..
The ``psec`` program uses the `openstack/cliff`_ command line
interface framework, which supports multiple output formats. The default
format the ``table`` format, which makes for nice clean output. (Other
formats will be described later.)
The groups can be listed using the ``groups list`` command:
.. code-block:: console
$ psec groups list
+---------+-------+
| Group | Items |
+---------+-------+
| jenkins | 1 |
| myapp | 4 |
| trident | 2 |
+---------+-------+
..
The variables in one or more groups can be shown with
the ``groups show`` command:
.. code-block:: console
$ psec groups show trident myapp
+---------+-----------------------+
| Group | Variable |
+---------+-----------------------+
| trident | trident_sysadmin_pass |
| trident | trident_db_pass |
| myapp | myapp_app_password |
| myapp | myapp_client_psk |
| myapp | myapp_client_ssid |
| myapp | myapp_ondemand_wifi |
| myapp | myapp_pi_password |
+---------+-----------------------+
..
When integrating a new open source tool or project, you can create
a new group and clone its secrets descriptions. This does not copy
any values, just the descriptions, allowing the current environment
to manage its own values.
.. code-block:: console
$ psec groups create newgroup --clone-from ~/git/goSecure/secrets/secrets.d/gosecure.json
created new group "newgroup"
$ psec groups list 2>/dev/null
+----------+-------+
| Group | Items |
+----------+-------+
| jenkins | 1 |
| myapp | 5 |
| newgroup | 12 |
| trident | 2 |
+----------+-------+
..
Showing Secrets
~~~~~~~~~~~~~~~
To examine the secrets, use the ``secrets show`` command:
.. code-block:: console
$ psec secrets show
+------------------------+----------+----------+------------------------+
| Variable | Type | Value | Export |
+------------------------+----------+----------+------------------------+
| jenkins_admin_password | password | REDACTED | jenkins_admin_password |
| myapp_app_password | password | REDACTED | DEMO_app_password |
| myapp_client_psk | string | REDACTED | DEMO_client_ssid |
| myapp_client_ssid | string | REDACTED | DEMO_client_ssid |
| myapp_ondemand_wifi | boolean | REDACTED | DEMO_ondemand_wifi |
| myapp_pi_password | password | REDACTED | DEMO_pi_password |
| trident_db_pass | password | REDACTED | trident_db_pass |
| trident_sysadmin_pass | password | REDACTED | trident_sysadmin_pass |
+------------------------+----------+----------+------------------------+
..
By default, the values of secrets are redacted when output. To show
the values in clear text in the terminal output, add the ``--no-redact`` flag:
.. code-block:: console
$ psec secrets show --no-redact
+------------------------+----------+------------------------------+------------------------+
| Variable | Type | Value | Export |
+------------------------+----------+------------------------------+------------------------+
| jenkins_admin_password | password | fetch.outsider.awning.maroon | jenkins_admin_password |
| myapp_app_password | password | fetch.outsider.awning.maroon | DEMO_app_password |
| myapp_client_psk | string | PSK | DEMO_client_psk |
| myapp_client_ssid | string | SSID | DEMO_client_ssid |
| myapp_ondemand_wifi | boolean | true | DEMO_ondemand_wifi |
| myapp_pi_password | password | fetch.outsider.awning.maroon | DEMO_pi_password |
| trident_db_pass | password | fetch.outsider.awning.maroon | trident_db_pass |
| trident_sysadmin_pass | password | fetch.outsider.awning.maroon | trident_sysadmin_pass |
+------------------------+----------+------------------------------+------------------------+
..
If you don't care about redaction and want to turn it off and save
the dozen keystrokes it takes to type `` --no-redact``, you can export
the environment variable ``D2_NO_REDACT`` set to (case-insensitive)
"true", "1", or "yes". Anything else leaves the default the same.
We'll do this now for later examples.
.. code-block:: console
$ export D2_NO_REDACT=true
..
The default is also to show all secrets. If you only want to process a
subset of secrets, you have two ways to do this.
#. Specify the variables you want to show on the command line as arguments:
.. code-block:: console
$ psec secrets show rabbitmq_default_user_pass rabbitmq_admin_user_pass
+----------------------------+----------+--------------------------------------+
| Variable | Type | Value |
+----------------------------+----------+--------------------------------------+
| rabbitmq_default_user_pass | password | handheld.angrily.letdown.frisk |
| rabbitmq_admin_user_pass | password | handheld.angrily.letdown.frisk |
+----------------------------+----------+--------------------------------------+
..
#. Use the ``--group`` flag and specify the group(s) you want to show
as command line arguments:
.. code-block:: console
$ psec secrets show --group jenkins trident
+----------------------------+----------+--------------------------------------+
| Variable | Type | Value |
+----------------------------+----------+--------------------------------------+
| jenkins_admin_password | password | handheld.angrily.letdown.frisk |
| trident_db_pass | password | handheld.angrily.letdown.frisk |
| trident_sysadmin_pass | password | handheld.angrily.letdown.frisk |
+----------------------------+----------+--------------------------------------+
..
Describing Secrets and Secret Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To describe the secrets in the select environment, use the
``secrets describe`` command:
.. code-block:: console
$ psec secrets describe
+----------------------------+----------+--------------------------------------------+
| Variable | Type | Prompt |
+----------------------------+----------+--------------------------------------------+
| google_oauth_client_id | string | Google OAuth2 client id |
| google_oauth_client_secret | string | Google OAuth2 client secret |
| google_oauth_refresh_token | string | Google OAuth2 refresh token |
| google_oauth_username | None | google_oauth_username |
| jenkins_admin_password | password | Password for Jenkins "admin" account |
| myapp_app_password | password | Password for myapp web app |
| myapp_client_psk | string | Pre-shared key for myapp client WiFi AP |
| myapp_client_ssid | string | SSID for myapp client WiFi AP |
| myapp_ondemand_wifi | boolean | "Connect on demand" when connected to wifi |
| myapp_pi_password | password | Password for myapp "pi" user account |
| trident_db_pass | password | Password for Trident postgres database |
| trident_sysadmin_pass | password | Password for Trident sysadmin account |
+----------------------------+----------+--------------------------------------------+
$ psec secrets describe --group trident
+-----------------------+----------+----------------------------------------+
| Variable | Type | Prompt |
+-----------------------+----------+----------------------------------------+
| trident_db_pass | password | Password for Trident postgres database |
| trident_sysadmin_pass | password | Password for Trident sysadmin account |
+-----------------------+----------+----------------------------------------+
..
To get a description of the available secret types, add the ``--types`` flag.
.. code-block:: console
$ psec secrets describe --types
+------------------+----------------------------------+
| Type | Description |
+------------------+----------------------------------+
| password | Simple (xkcd) password string |
| string | Simple string |
| boolean | Boolean ("true"/"false") |
| crypt_6 | crypt() SHA512 ("$6$") |
| token_hex | Hexadecimal token |
| token_urlsafe | URL-safe token |
| sha256_digest | DIGEST-SHA256 (user:pass) digest |
| uuid4 | UUID4 token |
| random_base64 | Random BASE64 token |
+------------------+----------------------------------+
..
.. note::
The type ``string`` is for secrets that are managed by another entity that you
must obtain and use to access some remote service (e.g., the pre-shared key for
someone's WiFi network, or an API key for accessing a cloud service provider's
platform). All other types are structured secret types that you generate for
configuring services.
..
Generating and Setting variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Secrets are generated using the ``secrets generate`` command
and are set manually using the ``secrets set`` command.
.. code-block:: console
$ psec help secrets generate
usage: psec secrets generate [-h] [-U] [args [args ...]]
Generate values for secrets
positional arguments:
args
optional arguments:
-h, --help show this help message and exit
-U, --unique Generate unique values for each type of secret (default:
False)
..
.. code-block:: console
$ psec secrets set --help
usage: psec secrets set [-h] [--undefined] [args [args ...]]
Set values manually for secrets
positional arguments:
args
optional arguments:
-h, --help show this help message and exit
--undefined Set values for undefined variables (default: False)
..
To regenerate all of the non-string secrets at once, using the same value for
each type of secret to simplify things, use the ``secrets generate`` command:
.. code-block:: console
$ psec secrets generate
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | gargle.earlobe.eggplant.kissable |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |
| rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
..
You can set one or more variables manually using ``secrets set`` and
specifying the variable and value in the form ``variable=value``:
.. code-block:: console
$ psec secrets set trident_db_pass="rural coffee purple sedan"
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | rural coffee purple sedan |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |
| rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
..
.. caution::
Note in the example above that the command argument is
``trident_db_pass="rural coffee purple sedan"`` and not
``trident_db_pass='rural coffee purple sedan'``.
When using the ``variable=value`` form of the ``secrets set``
command with a value that contains spaces, you **must** quote the value with
the double-quote character (``"``) as opposed to the single-quote
(apostrophe, or ``'``) character. The Bash shell (and possibly other
shells) will not properly parse the command line and the resulting
``sys.argv`` argument vector will be incorrectly set as seen here:
.. code-block:: console
_sys.argv[1:] = {list} <class 'list'>: ['--debug', 'secrets', 'set', 'trident_db_password=rural coffee purple sedan']
0 = {str} '--debug'
1 = {str} 'secrets'
2 = {str} 'set'
3 = {str} 'trident_db_password=rural coffee purple sedan'
__len__ = {int} 4
_sys.argv[1:] = {list} <class 'list'>: ['--debug', 'secrets', 'set', "trident_db_password='rural", 'coffee', 'purple', "sedan'"]
0 = {str} '--debug'
1 = {str} 'secrets'
2 = {str} 'set'
3 = {str} 'trident_db_password=\\'rural'
4 = {str} 'coffee'
5 = {str} 'purple'
6 = {str} 'sedan\\''
__len__ = {int} 7
..
Or you can generate one or more variables in a similar manner by adding
them to the command line as arguments to ``secrets generate``:
.. code-block:: console
$ psec secrets generate rabbitmq_default_user_pass rabbitmq_admin_user_pass
$ psec secrets show --column Variable --column Value
+----------------------------+----------------------------------------------+
| Variable | Value |
+----------------------------+----------------------------------------------+
| trident_db_pass | rural.coffee.purple.sedan |
| ca_rootca_password | gargle.earlobe.eggplant.kissable |
| consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
| jenkins_admin_password | gargle.earlobe.eggplant.kissable |
| rabbitmq_default_user_pass | embezzle.xerox.excess.skydiver |
| rabbitmq_admin_user_pass | embezzle.xerox.excess.skydiver |
| trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |
| vncserver_password | gargle.earlobe.eggplant.kissable |
| zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |
+----------------------------+----------------------------------------------+
..
A set of secrets for an open source project can be bootstrapped using the
following steps:
#. Create a template secrets environment directory that contains just
the secrets definitions. This example uses the template found
in the `davedittrich/goSecure`_ repository
(directory https://github.com/davedittrich/goSecure/tree/master/secrets).
#. Use this template to clone a secrets environment, which will initially
be empty:
.. code-block:: console
$ psec environments create test --clone-from ~/git/goSecure/secrets
new password variable "gosecure_app_password" is unset
new string variable "gosecure_client_ssid" is unset
new string variable "gosecure_client_ssid" is unset
new string variable "gosecure_client_psk" is unset
new password variable "gosecure_pi_password" is unset
new string variable "gosecure_pi_pubkey" is unset
environment directory /Users/dittrich/.secrets/test created
..
.. note::
The warnings about undefined new variables are presented on the standard
error file handle (a.k.a., ``&2``). You get rid of them on the console by
redirecting ``stderr`` to ``/dev/null`` or a file:
.. code-block:: console
$ psec environments create test --clone-from ~/git/goSecure/secrets 2>/dev/null
environment directory /Users/dittrich/.secrets/test created
..
.. code-block:: console
$ psec -e test secrets show --no-redact --fit-width
+-----------------------+----------+-------+
| Variable | Type | Value |
+-----------------------+----------+-------+
| gosecure_app_password | password | None |
| gosecure_client_ssid | string | None |
| gosecure_client_psk | string | None |
| gosecure_pi_password | password | None |
| gosecure_pi_pubkey | string | None |
+-----------------------+----------+-------+
..
#. First, generate all secrets whose type is not ``string``:
.. code-block:: console
$ psec -e test secrets generate
new password variable "gosecure_app_password" is unset
new string variable "gosecure_client_ssid" is unset
new string variable "gosecure_client_ssid" is unset
new string variable "gosecure_client_psk" is unset
new password variable "gosecure_pi_password" is unset
new string variable "gosecure_pi_pubkey" is unset
$ psec -e test secrets show --no-redact --fit-width
+-----------------------+----------+------------------------------+
| Variable | Type | Value |
+-----------------------+----------+------------------------------+
| gosecure_app_password | password | brunt.outclass.alike.turbine |
| gosecure_client_psk | string | None |
| gosecure_client_ssid | string | None |
| gosecure_pi_password | password | brunt.outclass.alike.turbine |
| gosecure_pi_pubkey | string | None |
+-----------------------+----------+------------------------------+
..
#. Finally, manually set the remaining ``string`` type variables:
.. code-block:: console
$ psec -e test secrets set --undefined
new string variable "gosecure_client_psk" is unset
new string variable "gosecure_client_ssid" is unset
new string variable "gosecure_pi_pubkey" is unset
Pre-shared key for goSecure client WiFi AP? [None]: atjhK5AlsQMw3Zh
SSID for goSecure client WiFi AP? [None]: YourWiFiSSID
SSH public key for accessing "pi" account? [None]: @~/.ssh/new_rsa.pub
$ psec -e test secrets show --no-redact --fit-width
+-----------------------+----------+------------------------------------------------------------------------------------------+
| Variable | Type | Value |
+-----------------------+----------+------------------------------------------------------------------------------------------+
| gosecure_app_password | password | brunt.outclass.alike.turbine |
| gosecure_client_psk | string | atjhK5AlsQMw3Zh
| gosecure_client_ssid | string | YourWiFiSSID |
| gosecure_pi_password | password | brunt.outclass.alike.turbine |
| gosecure_pi_pubkey | string | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU |
| | | uQ2/NVikfOfPo5mt9YTQyqRbeBzKlNgbHnsxh0AZatjhK5AlsQMw3ZhZUcLYZbt7szuQy8ineN0potlCJoVaMSOb |
| | | 9htf9gAPvzwxUnHxg35jPCzAXYAi3Erc6y338+CL0XxQvCogXOA+MwH7wZGgdT3WpupLG/7HAr/3KJEQQk1FlS2m |
| | | Rd+WuewnLbKkqBP21N+48ccq6XhEhAmlzzr9SENw5DMmrvMAYIYkoTwUeD3Qx4YebjFkCxZw+w7AafEFn0Kz6vCX |
| | | 4mp/6ZF/Ko+o04HM2sVr6wtCu2dB dittrich@localhost |
+-----------------------+----------+------------------------------------------------------------------------------------------+
..
.. note::
If you don't want to see the warnings about new variables that are not
defined, simply add the ``-q`` flag.
.. code-block:: console
$ psec -q secrets generate
$ psec -q secrets set --undefined
Pre-shared key for goSecure client WiFi AP? [None]:
..
..
You are now ready to compile your software, or build your project!
There is also a mechanism to run simple commands (i.e., basic arguments with
no special inline command substitution or variable expansion features of
shells like ``bash``) and use the resulting output as the value.
For this example, let's assume an environment that requires a CIDR
notation address for ingres access control (e.g., when using Amazon
Web Services to allow control of instances from your remote laptop).
.. code-block:: console
$ psec -e xgt secrets set aws_cidr_allowed=""
$ psec -e secrets show --no-redact aws_cidr_allowed
+------------------+--------+-------+
| Variable | Type | Value |
+------------------+--------+-------+
| aws_cidr_allowed | string | |
+------------------+--------+-------+
..
The ``psec`` program has a utility feature that will return
the current routable IP source address as an IP address, or using CIDR
notation. The variable can be set in one of two ways:
#. Via (non-interactive) inline command subtitution from the terminal shell:
.. code-block:: console
$ psec -e xgt secrets set aws_cidr_allowed="$(psec utils myip --cidr)"
..
#. Interactively when prompted using simple command line form:
.. code-block:: console
$ psec -e xgt secrets set aws_cidr_allowed
aws_cidr_allowed? []: !psec utils myip --cidr
..
The variable now contains the output of the specified program:
.. code-block:: console
$ psec secrets show --no-redact aws_cidr_allowed
+------------------+--------+------------------+
| Variable | Type | Value |
+------------------+--------+------------------+
| aws_cidr_allowed | string | 93.184.216.34/32 |
+------------------+--------+------------------+
..
.. note::
If you work from behind a static NAT firewall, this IP address will
likely not change very often (if at all). If you are using a mobile device
that is assigned differing DHCP addresses depending on location, the IP address
may change fairly regularly and the initial AWS Security Group setting will
begin to block access to your cloud instances. Programs like ``terraform``
can refresh their state, allowing you to simply reset the variable used to
create the Security Group and re-apply the plan to regenerate the AWS
Security Group and re-enable your remote access.
..
Sharing secrets
~~~~~~~~~~~~~~~
The ``psec`` program has a mechanism for sharing secrets with
others using GPG encrypted email messages for securing secrets in transit
and at rest in users' inboxes. Email is sent using Google's OAuth2
authenticated SMTP services.
.. note::
The Electronic Frontier Foundation (EFF) has a `Surveillance Self-Defense
Guide`_ that includes guides on `How to Use PGP for Linux`_ and other operating
systems. Follow their instructions if you are new to PGP/GPG.
..
The command is ``secrets send``.
.. code-block:: console
$ psec secrets send --help
usage: psec secrets send [-h] [-T] [--test-smtp] [-H SMTP_HOST]
[-U SMTP_USERNAME] [-F SMTP_SENDER] [-S SMTP_SUBJECT]
[args [args ...]]
Send secrets using GPG encrypted email. Arguments are USERNAME@EMAIL.ADDRESS
and/or VARIABLE references.
positional arguments:
args
optional arguments:
-h, --help show this help message and exit
-T, --refresh-token Refresh Google API Oauth2 token and exit (default:
False)
--test-smtp Test Oauth2 SMTP authentication and exit (default:
False)
-H SMTP_HOST, --smtp-host SMTP_HOST
SMTP host (default: localhost)
-U SMTP_USERNAME, --smtp-username SMTP_USERNAME
SMTP authentication username (default: None)
-F SMTP_SENDER, --from SMTP_SENDER
Sender address (default: 'noreply@nowhere')
-S SMTP_SUBJECT, --subject SMTP_SUBJECT
Subject line (default: 'For Your Information')
..
Any arguments (``args``) that contain the ``@`` symbol are assumed to be email
addresses while the rest are assumed to be the names of secrets variables
to be sent.
All recipients must have GPG public keys in your keyring. An exception is thrown
if no GPG key is associated with the recipient(s) email addresses.
.. code-block:: console
$ psec secrets send dittrich@u.washington.edu myapp_app_password
Setting homedir to '/Users/dittrich/.gnupg'
Initialised settings:
binary: /usr/local/bin/gpg
binary version: 1.4.11\ncfg:pubkey:1;2;3;16;17\ncfg:cipher:2;3;4;7;8;9;10;11;12;13\ncfg:ciphername:3DES;CAST5;BLOWFISH;AES;AES192;AES256;TWOFISH;CAMELLIA128;CAMELLIA192;CAMELLIA256\ncfg:digest:1;2;3;8;9;10;11\ncfg:digestname:MD5;SHA1;RIPEMD160;SHA256;SHA384;SHA512;SHA224\ncfg:compress:0;1;2;3\n'
homedir: /Users/dittrich/.gnupg
ignore_homedir_permissions: False
keyring: /Users/dittrich/.gnupg/pubring.gpg
secring: /Users/dittrich/.gnupg/secring.gpg
default_preference_list: SHA512 SHA384 SHA256 AES256 CAMELLIA256 TWOFISH AES192 ZLIB ZIP Uncompressed
keyserver: hkp://wwwkeys.pgp.net
options: None
verbose: False
use_agent: False
Creating the trustdb is only available with GnuPG>=2.x
sent encrypted secrets to dittrich@u.washington.edu
..
Use ``-q`` to produce no extraneous output.
.. code-block:: console
$ psec -q secrets send dittrich@u.washington.edu myapp_app_password
..
The resulting email looks like this:
.. code-block:: console
Message-ID: <5bac64ce.1c69fb81.b136e.45ae@mx.google.com>
Date: Wed, 26 Sep 2018 22:04:14 -0700 (PDT)
From: dave.dittrich@gmail.com
X-Google-Original-From: noreply@nowhere
Content-Type: multipart/related; boundary="===============6413073026511107073=="
MIME-Version: 1.0
Subject: For Your Information
To: dittrich@u.washington.edu
This is a multi-part message in MIME format.
--===============6413073026511107073==
Content-Type: multipart/alternative; boundary="===============2830935289665347054=="
MIME-Version: 1.0
--===============2830935289665347054==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tCgpoUUlXQStSZlhnK3dLTGJlRUFnZlFNcjZYb0lT
cS9BaTlMbEVpZTFTejd5ckEzUmN4SWdjb01XTUNSM3JBaXBHCjF0TTJoZkpxRGJZOThSOEVST01F
aVltSzR2aVJ4ZjgrSU54NU54SUJPbFh1T1JQTy82NElUKzdrVSt5aDZGV00KNU1MK0Jkb21sQzNF
eC9pd3hwbTJ1R2FPczFpcU9DaDIxbTd5RnJWYkNVSW5NN1ZiMTEwck41aXNOZ3BFdndrQgpaZHhp
alJqazdtYVl1eFNkc2c3Y2RVQ29uSmdBR214QU0vZkFzOTREcHNrYkwzMFpqZE1iRHlMbUk4NWp2
QU45CjU3KzAxLzM1MEMyN1hrbEUxdEZudWNlRkRqZ04zeEd4K2Zud0pqdkFpNUpaVHltanRkQi9r
dUZUMlJTTmJJTlAKMWRZdHp4WGxNeVd0SVphNDVYcHdNenZ1TkFTbEJtbENjQXk4YlluSEJmeFRy
SGdJSUlCMlZNY1N6dmdjR3BtVApkYzZqaDVOeEV1bWljOWdXMmplSnFqRHRtdW9Ib3dxZldZb2xX
bGlXUTMrNDNzeVkrdHFlMGgvWEwzS2ZxSTMrClZzWWdyQmpGd0hnem1INEthMWxucXdUZkMzZTJ3
cUI4Uk5hcllqcXAzbHFQOVBhMHdzSVVWMHVYN2dhL01kVWcKdHNRSktPWWJRTnlXVTFLZEZWNHl4
Ynp1TWVlQ3ltMmxMbXJwVks5T3hCV04vbCtXMjRsWmhkck9TcGFJQnpNdgpnc1p3VWVuVzBXR054
bklwUGhoSWRuVE40ZlNscE5JVDhMcmJYeUhoY2ZVS2lsUDNpeEVPRS9Lc25QUFJNTURFCk9SY0xT
Z3FMMTB4b0toMnNzZTNxNG5RaHZkZW5IVVVxVjJ0WW1UVmRCNVl3cTN1MFdtY3BGSGU2NnBZeTBB
VSsKdzRjb2JVM2crQWtJMHBNQnllRzZYaWV4VzF1UzRLVVVnaFlhWVlYQ2dnazJZNEpZT05QSDJJ
NlIydmxuNjFsVApZdm1tR0NNamw3cC9pTnE2RWJpbndoMnNsbkpLMHd3S1BIbVBPUjJvRjdWREN0
dE9idHA0cEZUWTNHalByc0dRCkNDT3dYR2hCSFVQRnY2c3R4NEdtUi9GUWpBRWxxaEpjQWtTbDFz
WWhsUFRhSmEyVGgyNG81L1lPUmxRaHhhRUgKUEFrNFgzcGVCMk9UVjRNR2RCOD0KPTc0aXEKLS0t
LS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=
--===============2830935289665347054==
Content-Type: text/html; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
VGhlIGZvbGxvd2luZyBzZWNyZXQgaXMgYmVpbmcgc2hhcmVkIHdpdGggeW91OgoKbXlhcHBfYXBw
X3Bhc3N3b3JkPWJydW50IG91dGNsYXNzIGFsaWtlIHR1cmJpbmU=
--===============2830935289665347054==--
--===============6413073026511107073==--
..
Decrypted, it looks like this:
.. code-block:: console
Date: Wed, 26 Sep 2018 22:04:14 -0700 (PDT)
From: dave.dittrich@gmail.com
Subject: For Your Information
To: dittrich@u.washington.edu
The following secret is being shared with you:
myapp_app_password=brunt.outclass.alike.turbine
--
Sent using psec version 23.4.1
https://pypi.org/project/python-secrets/
https://github.com/davedittrich/python_secrets
..
A group of secrets required for Google's `OAuth 2.0 Mechanism`_ is provided
and must be set according to Google's instructions. See also:
+ https://github.com/google/gmail-oauth2-tools/wiki/OAuth2DotPyRunThrough
+ http://blog.macuyiko.com/post/2016/how-to-send-html-mails-with-oauth2-and-gmail-in-python.html
+ https://developers.google.com/api-client-library/python/guide/aaa_oauth
+ https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py
+ https://developers.google.com/identity/protocols/OAuth2
.. code-block:: console
$ psec groups show oauth
+-------+----------------------------+
| Group | Variable |
+-------+----------------------------+
| oauth | google_oauth_client_id |
| oauth | google_oauth_client_secret |
| oauth | google_oauth_refresh_token |
+-------+----------------------------+
..
Processing templates
~~~~~~~~~~~~~~~~~~~~
.. image:: https://asciinema.org/a/201507.png
:target: https://asciinema.org/a/201507?autoplay=1
:align: center
:alt: Rendering templates outside the source repo directory
:width: 835px
..
Outputting structured information for use in other scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once secrets are created and stored, they will eventually need to be accessed
in order to use them in program execution. This can be done by passing the
``.json`` secrets file itself to a program, or by outputting the variables in
other formats like CSV, JSON, or as environment type variables.
Passing the secrets file by path
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One way to do this is to take advantage of command line options like
`Ansible`_'s ``--extra-vars`` and passing it a path to the ``.json`` secrets
file. (See `Passing Variables On The Command Line`_). Here is how to do
it.
Let's assume we want to use ``consul_key`` variable to configure Consul
using Ansible. Here is the variable as stored:
.. code-block:: console
$ psec secrets show consul_key
+------------+-----------+----------------------------------------------+
| Variable | Type | Value |
+------------+-----------+----------------------------------------------+
| consul_key | token_hex | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |
+------------+-----------+----------------------------------------------+
..
Using Ansible's ``debug`` module, we can verify that this variable is not
set by any previously loaded Ansible inventory:
.. code-block:: console
$ ansible -i localhost, -m debug -a 'var=consul_key' localhost
localhost | SUCCESS => {
"consul_key": "VARIABLE IS NOT DEFINED!"
}
..
In order for Ansible to set the ``consul_key`` variable outside of any
pre-defined inventory files, we need to pass a file path to the
``--extra-vars`` option. The path can be obtained using the
``psec secrets path`` command:
.. code-block:: console
$ psec secrets path
/Users/dittrich/.secrets/python_secrets/secrets.json
..
It is possible to run this command in an in-line command expansion operation in
Bash. Ansible expects the file path passed to ``-extra-vars`` to start with an
``@`` character, so the command line to use would look like this:
.. code-block:: console
$ ansible -i localhost, -e @"$(psec secrets path)" -m debug -a 'var=consul_key' localhost
localhost | SUCCESS => {
"consul_key": "HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY="
}
..
Ansible now has the value and can use it in templating configuration files, or
so forth.
Other programs like Hashicorp `terraform`_ look for environment variables that
begin with ``TF_VAR_`` and use them to set ``terraform`` variables for use
in modules. To prove we are running in a sub-shell, we will first change the
shell prompt.
.. code-block:: console
$ PS1="test> "
test> psec -e test --export-env-vars --env-var-prefix="TEST_" run bash
$ env | grep '^TEST_'
TEST_gosecure_pi_pubkey=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU [...]
TEST_gosecure_client_psk=atjhK5AlsQMw3Zh
TEST_gosecure_client_ssid=YourWiFiSSID
TEST_gosecure_pi_password=brunt.outclass.alike.turbine
TEST_gosecure_app_password=brunt.outclass.alike.turbine
$ exit
test>
..
.. image:: https://asciinema.org/a/201510.png
:target: https://asciinema.org/a/201510?autoplay=1
:align: center
:alt: Exporting secrets via the environment
:width: 835px
..
Operational Security
----------------------
As noted in the Limitations section above, secrets are stored in plaintext
plaintext form (i.e., they are *not* encrypted) in files. Those files are in
turn stored in a directory in the file system, subject to Linux file
ownership and permission access controls.
The default location for storing these files is in an *environment directory*
in a subdirectory of the user's home directory whose name starts with a period
character (a.k.a., a *dot*). Files (or directories) whose name starts with a
period are known as *dot files*, or *hidden files* because the `ls` command
does not show it unless you use the `-a` flag.
The secrets environment directories can also be used to store other files
besides secrets. One such use case is storing JSON Web Tokens (JWTs) used as
bearer tokens by protocols like Google's `OAuth 2.0 Mechanism`_ for securing
access to web services and APIs. While this improves security in terms of
remote access, is not not without its own risks (including the JWT file being
stored in the file system for an indefinite period of time).
* `JSON Web Tokens (JWT) are Dangerous for User Sessions—Here’s a
Solution`_, by Raja Rao, June 24, 2021
* `Stop Using JSON Web Tokens For Authentication. Use Stateful Sessions
Instead`_, by Francisco Sainz, April 4, 2022
* `What’s the Secure Way to Store JWT?`_, by Yang Liu, July 23, 2020
Besides JWTs, other use cases for storing sensitive files within `psec`
environments include backups of database contents, Let's Encrypt certificates,
SSH keys, or other secrets necessary for ensuring cloud instances can be
destroyed and recreated without losing state or requiring regeneration
(and redistribution or revalidation) of secrets.
The output of `init --help` mentions this risk and offers a way to mitigate
some of the risk by locating the secrets storage base directory within a
directory that is stored on an encrypted USB-connected disk device or encrypted
disk image, or a removable device or remote file system, that is only mounted
when needed and unmounted as soon as possible. This ensures sensitive data that
are not being actively used are left encrypted in storage. The
`D2_SECRETS_BASEDIR` environment variable or `-d` option allow you to specify
the directory to use.
The `psec` CLI has a secure deletion mechanism that over-writes file contents
prior to deletion, helping to reduce leaving remnants of secrets in unallocated
file system storage, similar to the way the Linux `shred` command works.
Python Script Security
----------------------
Last, but certainly not least, take the time to read up on `Python Security`_
and understand the types and sources of security vulnerabilities related to
Python programs. Keep these ideas in mind when using and/or modifying this
program.
As part of testing, the `Bandit`_ security validation program is used.
(See `Getting started with Bandit`_).
.. _Bandit: https://pypi.org/project/bandit/
.. _Getting started with Bandit: https://developer.rackspace.com/blog/getting-started-with-bandit/
In situations where Bandit warnings can safely be ignored, the ``# nosec``
comment appears on source code lines. Comments as to why these can be
safely ignored are included in the code. (Please feel free to issue pull
requests if you disagree.)
One runtime security mechanism employed by ``psec`` is control of the process'
``umask``. This is important when running programs that create files, which
will inherit their permissions per the process ``umask``. The ``umask`` will be
inherited by every new child process and can be set in the user's ``.bashrc``
(or other shell initialization) file.
The ``psec run`` command can be used to run programs as child processes,
optionally exporting environment variables as well, so controlling the
``umask`` results in improved file permission security regardless of
whether the user knows to set their process ``umask``.
You can see the effect in these two examples.
First, by setting the ``umask`` to ``0`` you see the very permissive file
permissions (as well as getting a warning from ``psec`` about finding a file
with lax permissions):
.. code-block:: console
$ psec --umask 0o000 run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo
1+0 records in
1+0 records out
512 bytes copied, 0.000019 s, 2.7 MB/s
$ ls -l $(psec environments path --tmpdir)/foo
[!] file /Users/dittrich/.secrets/python_secrets/tmp/foo is mode 0o100666
-rw-rw-rw- 1 dittrich staff 512 Sep 8 13:05 /Users/dittrich/.secrets/python_secrets/tmp/foo
$ rm $(psec environments path --tmpdir)/foo
..
Now when using the default ``--umask`` value, the file permissions are restricted
(and thus no more warning):
.. code-block:: console
$ psec run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo
1+0 records in
1+0 records out
512 bytes copied, 0.000243 s, 2.1 MB/s
$ ls -l $(psec environments path --tmpdir)/foo
-rw------- 1 dittrich staff 512 Sep 8 13:04 /Users/dittrich/.secrets/python_secrets/tmp/foo
$ rm $(psec environments path --tmpdir)/foo
..
Bugs, Enhancements, and Future Work
-----------------------------------
Feature requests (and of course bug reports) are highly encouraged. You can
do that by `opening an issue`_ on GitHub. Better yet, make a `pull
request`_ with your own fix or feature. (Check there to see if one
may already exist.)
If you want to help, there are some things that are on the "to do"
list. These are tracked on this repository's GitHub `Projects`_ page.
General or more elaborate potential enhancements are listed here:
* Increase test coverage (test driven development is a Good Thing(TM))
* The Mantl project (GitHub `mantl/mantl`_) employs a `security-setup`_ script
that takes care of setting secrets (and non-secret related variables) in a
monolithic manner. It has specific command line options, specific secret
generation functions, and specific data structures for each of the component
subsystems used by `mantl/mantl`_. This method is not modular or extensible, and
the `security-setup`_ script is not generalized such that it can be used by
any other project. These limitations are primary motivators for writing
``python_secrets``, which could eventually replace ``security-setup``.
At this point, the Mantl ``security.yml`` file can be read in and
values can be manually set, as seen here:
.. code-block:: console
$ psec -d ~/git/mantl --secrets-file security.yml secrets show -f yaml
secrets descriptions directory not found
- Value: admin:password
Variable: chronos_http_credentials
- Value: chronos
Variable: chronos_principal
- Value: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx
Variable: chronos_secret
- Value: 88821cbe-c004-4cff-9f91-2bc36cd347dc
Variable: consul_acl_agent_token
- Value: f9acbe14-28d3-4d06-a1c9-c617da5ebb4e
Variable: consul_acl_mantl_api_token
- Value: de54ae85-8226-4146-959f-8926b0b8ee55
Variable: consul_acl_marathon_token
- Value: dfc9b244-5140-41ad-b93a-ac5c2451fb95
Variable: consul_acl_master_token
- Value: e149b50f-cb5c-4efe-be96-26a52efdc715
Variable: consul_acl_secure_token
- Value: 719f2328-6446-4647-adf6-310013bac636
Variable: consul_acl_vault_token
- Value: Z0niD1jeiTkx7xaoewJm2A==
Variable: consul_gossip_key
- Value: true
Variable: do_chronos_auth
- Value: true
Variable: do_chronos_iptables
- Value: true
Variable: do_chronos_ssl
- Value: true
Variable: do_consul_auth
- Value: true
Variable: do_consul_ssl
- Value: true
Variable: do_mantl_api_auth
- Value: true
Variable: do_mantlui_auth
- Value: true
Variable: do_mantlui_ssl
- Value: true
Variable: do_marathon_auth
- Value: true
Variable: do_marathon_iptables
- Value: true
Variable: do_marathon_ssl
- Value: true
Variable: do_mesos_auth
- Value: true
Variable: do_mesos_follower_auth
- Value: true
Variable: do_mesos_framework_auth
- Value: true
Variable: do_mesos_iptables
- Value: true
Variable: do_mesos_ssl
- Value: false
Variable: do_private_docker_registry
- Value: mantl-api
Variable: mantl_api_principal
- Value: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+
Variable: mantl_api_secret
- Value: admin:password
Variable: marathon_http_credentials
- Value: marathon
Variable: marathon_principal
- Value: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX
Variable: marathon_secret
- Value:
- principal: marathon
secret: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX
- principal: chronos
secret: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx
- principal: mantl-api
secret: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+
Variable: mesos_credentials
- Value: follower
Variable: mesos_follower_principal
- Value: Q53uAa2mNM0UNe2RUjrX6k7QvK6ojjH1gHXYLcm3Lmfr
Variable: mesos_follower_secret
- Value: password
Variable: nginx_admin_password
- Value: true
Variable: security_enabled
- Value: chronos
Variable: zk_chronos_user
- Value: JWPO11z4lU5qeilZ
Variable: zk_chronos_user_secret
- Value: hsr+R6YQBAOXoY84a8ne8bU0opg=
Variable: zk_chronos_user_secret_digest
- Value: marathon
Variable: zk_marathon_user
- Value: UBh77ok2svQAqWox
Variable: zk_marathon_user_secret
- Value: mo2mQGXcsc21zB4wYD18jn+Csks=
Variable: zk_marathon_user_secret_digest
- Value: mesos
Variable: zk_mesos_user
- Value: L3t9FEMsXehqeBvl
Variable: zk_mesos_user_secret
- Value: bHYvGteRBxou4jqJ8XWAYmOmzxs=
Variable: zk_mesos_user_secret_digest
- Value: super
Variable: zk_super_user
- Value: 2DyL/n/GLi3Q0pa75z9OjODGZKC1RCaEiKNV1ZXo1Wpk
Variable: zk_super_user_secret
$ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password
secrets descriptions directory not found
"nginx_admin_password","password"
$ psec -d ~/git/mantl --secrets-file security.yml secrets set nginx_admin_password=newpassword
secrets descriptions directory not found
$ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password
secrets descriptions directory not found
"nginx_admin_password","newpassword"
..
There are a few things that can be done to use ``psec`` as a replacement
for the ``security-setup`` script. These include:
* Produce secrets descriptions in a ``security.d`` directory.
* Remove the variables that are not secrets requiring regeneration for rotation
or "break-glass" procedures (e.g., like ``chronos_principal``, which is a
userID value, and ``do_mesos_auth``, which is a boolean flag).
* Break down more complex data structures (specifically, the ``mesos_credentials``
list of dictionaries with keys ``principal`` and ``secret``). These could
instead be discrete variables like ``marathon_secret`` (which appears to
be the secret associated with the invariant "variable" ``marathon_principal``).
.. note::
Alternatively, these kind of variables could be supported by defining a type ``invariant``
or ``string`` and prompting the user to provide a new value (using any current value
as the default).
..
Credits
-------
Tools used in rendering this package:
* Cookiecutter_
* `cookiecutter-pypackage`_
Development of this program was supported in part under an Open Source
Development Grant from the Comcast Innovation Fund.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
.. _openstack/cliff: https://github.com/openstack/cliff
.. _python-update-dotdee: https://pypi.org/project/update-dotdee/
.. _terraform: https://www.terraform.io/
.. _Vault by Hashicorp: https://www.vaultproject.io/
.. _mantl/mantl: https://github.com/mantl/mantl
.. _security-setup: http://docs.mantl.io/en/latest/security/security_setup.html
.. _Ansible: https://docs.ansible.com/
.. _libfuse/sshfs: https://github.com/libfuse/sshfs
.. _D2 Ansible Playbooks: https://github.com/davedittrich/ansible-dims-playbooks
.. _Passing variables on the Command Line: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#passing-variables-on-the-command-line
.. _OAuth 2.0 Mechanism: https://developers.google.com/gmail/imap/xoauth2-protocol.
.. _davedittrich/goSecure: https://github.com/davedittrich/goSecure
.. _Surveillance Self-Defense Guide: https://ssd.eff.org/en
.. _opening an issue: https://github.com/davedittrich/python_secrets/issues
.. _pull request: https://github.com/davedittrich/python_secrets/pulls
.. _Projects: https://github.com/davedittrich/python_secrets/projects/1
.. _How to Use PGP for Linux: https://ssd.eff.org/en/module/how-use-pgp-linux
.. _Python Security: https://python-security.readthedocs.io/index.html
.. _JSON Web Tokens (JWT) are Dangerous for User Sessions—Here’s a Solution: https://redis.com/blog/json-web-tokens-jwt-are-dangerous-for-user-sessions/
.. _Stop Using JSON Web Tokens For Authentication. Use Stateful Sessions Instead: https://betterprogramming.pub/stop-using-json-web-tokens-for-authentication-use-stateful-sessions-instead-c0a803931a5d
.. _What’s the Secure Way to Store JWT?: https://medium.com/swlh/whats-the-secure-way-to-store-jwt-dd362f5b7914
.. _changelog:
History
-------
.. Follow: https://keepachangelog.com/en/1.0.0/
..
.. Guiding Principles
.. ------------------
.. Changelogs are for humans, not machines.
.. There should be an entry for every single version.
.. The same types of changes should be grouped.
.. Versions and sections should be linkable.
.. The latest version comes first.
.. The release date of each version is displayed.
.. Mention whether you follow Semantic Versioning.
..
.. Types of changes
.. ----------------
.. Added for new features.
.. Changed for changes in existing functionality.
.. Deprecated for soon-to-be removed features.
.. Removed for now removed features.
.. Fixed for any bug fixes.
.. Security in case of vulnerabilities.
23.4.1 (2023-04-19)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Added `secrets find` command.
- Added support for new variable type `boolean`.
Changed
^^^^^^^
- Updated GitHub Actions workflows (default to Python 3.9.16).
- Drop Python 3.7, 3.8, add Python 3.11 (default to 3.10) for `tox`.
- Fixed downstream dependency and `pip` installation problems.
- Resolved new `pep8` and `bandit` findings.
22.6.1 (2022-06-21)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Added `--ignore-missing` option to continue when settings variables.
- Added 'Operational Security' section to README.
22.6.0 (2022-06-10)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add `about` command to expose selected settings for situational awareness.
- Add `pytest` code coverage reporting.
- Add BATS runtime tests related to changes.
Changed
^^^^^^^
- Fix caching bug with non-unique secret generation.
- Fix bugs with setting/deleting secrets.
- Improve secrets basedir initialization logic.
- Expand use of `pathlib.Path`.
- Improvements to source code, test, and vscode launch configuration quality.
22.5.1 (2022-05-25)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Switch to using factory pattern for secrets generation.
- General code quality and test improvements.
- Improve `secrets get` command logic and help.
- Fix `utils yaml-to-json` subcommand and tests.
- Resolve setuptools warnings.
- Separate utility functions from `utils` subcommands.
Removed
^^^^^^^
- Retire `consul_key` secret type in favor of `token_base64`.
- Retire insecure secrets types (e.g., use of SHA1).
22.5.0 (2022-05-11)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Test support for Python 3.10.
- Add better logging controls.
Changed
^^^^^^^
- Generalize Google OAuth2 email functionality.
- Improve use and testing of exceptions.
22.1.0 (2022-01-22)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add `init` command and `--init` flag to initialize secrets base directory.
- Ensure overridden values via flags are exported to process environment
for subprocesses to use.
- Add missing tests for features added in a previous release.
- Add and start using application-specific exception classes.
Changed
^^^^^^^
- Move functions and variables to `utils` to improve reuse ability.
- Use `get_` prefix more consistently for getter method/function names.
- Over-ride cliff formatter class globally in app parser setup.
- Use `pathlib.Path` for paths for cleaner code.
- Fix bugs in `environments delete` command.
- Fix bugs in `--from-options` feature of `secrets get` and `secrets set`.
- Improvements to source code, test, and vscode launch configuration quality.
21.11.0 (2021-11-22)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add `Help` attribute to descriptions for URL to more information.
Changed
^^^^^^^
- General code quality, documentation, and testing enhancements
- Move `tmpdir` path creation to `secrets_environment.SecretsEnvironment()`.
- Move `umask()` function and variables to `utils`.
Removed
^^^^^^^
- Drop Python 3.6 support due to it being EOL.
21.9.1 (2021-09-15)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Added `secrets tree` subcommand.
Changed
^^^^^^^
- Fixed bugs with `environments path --tmpdir` subcommand and
`run` subcommand with `--elapsed` option when no environment exists.
- Changed license file name.
- Improved documentation.
21.9.0 (2021-09-07)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Increased test coverage to address bugs (below) being fixed.
Changed
^^^^^^^
- Fixed bugs in `Makefile` and `tox.ini` file.
- Fixed bug setting undefined variables.
- Switched from `numpy` to Python `secrets` module for random bytes.
- Increased key size from 16 to 32 bits for `consul_key`, `token_hex` and `token_urlsafe`.
21.8.0 (2021-08-12)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Fixed bug in setup.py+setup.cfg
21.7.0 (2021-07-30)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Secrets descriptions for demoing HypriotOS Flash mods Medium article
Changed
^^^^^^^
- Improve `secrets set --from-options`
- General code quality, documentation, and testing enhancements
21.6.0 (2021-06-23)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Ability to set and generate secrets from defaults options
- Ability to create an alias for an existing environment
- Allow retroactive mirroring of new secrets
Changed
^^^^^^^
- Switched from `pbr` to `setuptools_scm` for version numbering
- Switched to more secure random number generation
21.2.0 (20201-02-23)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Improve GitHub Actions workflows
- Overall documentation and code enhancements
- Improve handling of wildcards in options list
Changed
^^^^^^^
- Fix bugs with handling empty lists, cloning environments, BATS tests
- Increase password complexity a bit more
- Fix ReadTheDocs
20.11.0 (2020-11-17)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add `secrets create` and `secrets delete` commands
Changed
^^^^^^^
- Normalize all logger and exception output text
- Refactoring code for better modulatiry
- Normalize `group create` and `group delete` code
- Normalize `secrets show` and `secrets describe` code
- Fix bug that left variables missing after cloning
- Add Python 3.9 to testing matrix
- Switch from .yml to .json format for secrets
- Expand IP address support in `utils` subcommand
20.8.1 (2020-08-11)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Fixes to v20.8.0
20.8.0 (2020-08-11)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add GitHub workflow to publish to test.pypi.org
- Add `secrets backup` and `secrets restore` logic
- Open web browser to documentation for help
Changed
^^^^^^^
- Go back to date-based version numbering
- General CI/CD workflow updates
- Improve directory handling in `environments path`
20.2.15 (2012-02-15)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Added Python 3.8 support to test matrix
Changed
^^^^^^^
- Fix bug in `environments default`
- Put elapsed time (and BELL) on stdout
- Fix bug in `environments tree`
- Allow setting vars using diff names+environment
19.12.0 (2019-12-16)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add and document new `boolean` data type
- Add `groups delete` command
Changed
^^^^^^^
- Improve default environment handling
- Improve tox+BATS testing
- Address security issue per "Your xkcd passwords are pwned" article
- General code quality and test improvements
- Add protection from over-writing existing env vars
- Add `Options` attribute
19.11.1 (2019-11-29)
~~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Enhancements to better support Windows 10
- Allow cloning group descriptions from environment
- Fix tty/no-tty handling with `environments delete`
- Expose terraform command on `-v`
- Validate variable exists in environment
- Fix broken `environments tree` code
19.10.1 (2019-10-20)
~~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Move BATS unit tests into tox testing
- Avoid attempting interactive things when no tty
- Improve file and directory permissions logic
19.10.0 (2019-10-14)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Working SSH key and configuration management
- Use `bullet` for interactive list selection
- Elapsed timer feature
- Parsing of terraform output to extract SSH public keys
- `umask` control for better new file permission settings
- Support configuring terraform `tfstate` backend
- Allow setting secrets by copying from another environment
Changed
^^^^^^^
- Numerous bug fixes
- Refine testing
- Option to only show undefined variables
- Sort environments when listing
19.9.0 (2019-09-05)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add `environments delete` subcommand
- Allow cloning environment from an existing one
Changed
^^^^^^^
19.8.3 (2019-08-28)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Dynamically get version number
- General testing enhancements
- General code quality enhancements
- Ensure more secure file permissions
19.8.2 (2019-08-23)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- General code quality enhancements
19.8.0 (2019-08-22)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- IP address determination
- Allow cloning new group in an empty environment
- Make `python -m psec` work
- JSON output method
- Environment aliasing feature
Changed
^^^^^^^
- General code quality and testing enhancements
- Be more explicit about default environment
- Tighten permissions on cloned environments/groups
- Add insecure permissions checking
19.5.1 (2019-05-08)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
Add `HISTORY.rst` file
19.4.5 (2019-05-08)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add command `ssh config` to manage SSH configuration snippet
for use by `update-dotdee` to generate ~/.ssh/config file
- Add command `ssh known-hosts add` and `ssh known-hosts remove`
to manage system known_hosts file(s)
Changed
^^^^^^^
- Generalized exception to fix --version bug
- Clean up temporary docs/psec_help.txt file
19.4.4 (2019-04-21)
~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
- Fix Bats dependencies/tests
- Fix broken documentation (wt?)
- Fix messed up release tagging
19.4.0 (2019-04-19)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Python 3.7 coverage for Travis CI
Changed
^^^^^^^
- Complete --help output (epilog text) in all commands
- Install a script 'psec' to complement console_script entry point
- Clarify arguments in --help output
Deprecated
^^^^^^^^^^
- The 'python_secrets' command is now just 'psec'
19.3.1 (2019-04-06)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add ``environments rename`` command
- Add ``utils set-aws-credentials`` command to mirror AWS CLI credentials
- Use ``autoprogram_cliff`` for self-documentation
- Add ``cliff.sphinxext`` for documentation
Changed
^^^^^^^
- Refactored ``SecretsEnvironment()`` so ``autoprogram_cliff`` works
18.11.0 (2018-11-09)
~~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Add "--type" option to "secrets describe"
- Improve visibility into default environment
- Add screencasts to documenation
- Add RST checks to ensure PyPi documentation works
- Add feedback about minimum Python version
- Add ``--json`` output to ``environments path``
- Add reference to proof-of-concept using goSecure fork
Changed
^^^^^^^
- The "secrets describe" command now describes variables and types
- Allow ``secrets set`` to set any type (not just ``string``)
18.9.0 (2018-09-27)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Switched to calendar version numbering
- Finish GPG encrypted email delivery of secrets
- ``groups create`` command
- Improve error handling consistency when no environment exists
0.16.0 (2018-09-12)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Use attribute maps instead of lookup loops
- Add Prompt attribute in descriptions for better UX when setting variables
- Note new undefined variables when adding groups or ``environments create --clone-from``
- When exporting vars, also export PYTHON_SECRETS_ENVIRONMENT w/environment name
- Add reference to Python Security coding information
- ``environments tree`` command
- ``environments path`` command with features supporting Ansible Lookup Plugin
- ``secrets get`` command
- ``groups path`` command
- ``environments default`` command
0.14.0 (2018-08-30)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- Option to export secrets as environment variables (with optional prefix)
- Can now set secrets (any specified or all undefined) via command line
- ``utils myip`` command returns routable IP address (with CIDR option)
- ``run`` command allows running commands with exported environment variables
Changed
^^^^^^^
- Renamed ``template`` comamnd to ``utils tfoutput``
Removed
^^^^^^^
- Dropped support for Python 3.4, 3.5, since ``secrets`` module only in Python >= 3.6
0.10.0 (2018-08-23)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
- New ``string`` type for manually set secrets
- ``secrets path`` command provides path to secrets ``.yml`` file
- ``template`` command (Jinja templating)
- Default environment to basename of cwd
- Clone environment from skeleton directory in repo
0.9.1 (2018-08-19)
~~~~~~~~~~~~~~~~~~
Added
^^^^^
- ``secrets describe`` command
- ``environments create`` command
- ``environments list`` command
- Expand secrets types and generation methods
- Add initial feature for sending secrets via email using Google OAuth2 SMTP
Removed
^^^^^^^
- Drop Python 2.7 support (at least for now...)
Security
^^^^^^^^
- Add ``six`` for securing ``input`` call
0.8.0 (2018-05-11)
~~~~~~~~~~~~~~~~~~
(TBD)
0.4.0 (2018-05-01)
~~~~~~~~~~~~~~~~~~
(TBD)
0.3.6 (2018-04-29)
~~~~~~~~~~~~~~~~~~
(TBD)
0.3.0 (2018-04-27)
~~~~~~~~~~~~~~~~~~
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/davedittrich/python_secrets",
"name": "python-secrets",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Dave Dittrich",
"author_email": "dave.dittrich@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/30/61/4001a0f35881bdbb2753a14420cf805baa977883da0050ec0cda3abe1a5f/python_secrets-23.4.1.tar.gz",
"platform": null,
"description": "=====================\npsec (python_secrets)\n=====================\n\n.. image:: https://img.shields.io/pypi/v/python_secrets.svg\n :target: https://pypi.python.org/pypi/python_secrets\n\n.. image:: https://img.shields.io/travis/davedittrich/python_secrets.svg\n :target: https://travis-ci.org/davedittrich/python_secrets\n\n.. image:: https://readthedocs.org/projects/python-secrets/badge/?version=latest\n :target: https://python-secrets.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n\nPython command line app for managing groups of secrets (passwords, API keys, etc) and\nother project variables. Reduces security risks from things like weak default passwords,\nsecrets stored in files in the source code repository directory.\n\nVersion: 23.4.1\n\n* Free software: `Apache 2.0 License <https://www.apache.org/licenses/LICENSE-2.0>`_\n* Documentation: https://python_secrets.readthedocs.org.\n\nFeatures\n--------\n\n* Uses the `openstack/cliff`_ command line framework for a robust and\n full-featured CLI. It is easy to add new commands and features!\n\n* Supports a \"drop-in\" model for defining variables in a modular manner\n (something like the `python-update-dotdee`_ program), supporting simplified\n bulk setting or generating values of variables as needed.\n\n* Like `python-update-dotdee`_, `psec` produces a single master\n ``.json`` file to hold variables defined by the drop-in group\n description files. That means you can use that file directly\n to set variables to be used from within other programs like\n Ansible (e.g. ``ansible-playbook playbook.yml -e @\"$(psec secrets path)\"``)\n\n* Support multiple simultaneous sets of secrets (environments) for\n flexibility and scalability in multi-environment deployments and to\n support different use cases or different combinations of secrets.\n\n* Supports changing the storage location of secrets and variables to\n allow them to be stored on secure mobile media (such as self-encrypting\n external SSD or Flash drives) or encrypted disk images mounted at\n run-time to ensure the confidentiality of data at rest.\n\n* List the groups of variables (and how many secrets in each group).\n\n* Describe secrets by their variable name and type (e.g., ``password``,\n ``uuid4``, ``random_base64``). You can also include a descriptive\n string to prompt the user for a value, a list of options to choose\n from (or ``*`` for \"any value the user enters\"), and a list of\n environment variables to export for other programs to use at\n run time.\n\n* Allows manual entry of values, setting non-secret variables from\n a default value, or automatic generation of secrets according to\n their type.\n\n* Manually set ``string`` variables based on the output of simple\n commands. This allows interfacing with external programs for\n obtaining secrets, such as `Vault by Hashicorp`_.\n\n* Generate unique values for variables, or use a single value per\n type to simplify use of secrets in access control of services\n while supporting a \"break-glass\" process to quickly regenerate\n secrets when needed.\n\n* Show the variables and their unredacted values (or redacted them\n to maintain secrecy during demonstrations or in documentation).\n\n* Export the variables (optionally with a specific prefix string)\n to the environment and run a command that inherits them (e.g.,\n to pass variables to `terraform`_ for provisioning cloud\n instances).\n\n* Output the variables and values in multiple different formats (CSV,\n JSON, YAML) for use in shell scripts, etc. using ``cliff`` features.\n\n* Send secrets to other users on demand using GPG encrypted email to\n protect the secrets while in transit and while at rest in users'\n email inboxes.\n\n* Makes it easy to store temporary files (e.g., the output from\n Jinja template rendering)\n that may contain secrets *outside* of the source repo directory\n in an environment-specific ``tmp/`` directory.\n\n.. note::\n\n Due to the use of the Python ``secrets`` module, which was introduced\n in Python 3.6, only Python versions >= 3.6 can be used.\n\n..\n\n.. _limitations:\n\nLimitations\n-----------\n\n* Secrets are stored in *unencrypted* form in the environments\n directories. Permissions are set to limit access, but this is not an\n \"encrypt data at rest\" solution like `Vault by Hashicorp`_.\n\n* Does not handle secure distributed access for users on remote systems. You\n must use something like `Vault by Hashicorp`_ or `libfuse/sshfs`_ for secure\n (realtime) distributed access.\n\n* Does not handle secure distribution of newly generated secrets out\n to distributed systems that need them. You will need to use a program\n like `Ansible`_ and related playbooks for pushing out and changing\n secrets (or for retrieving backups). Look at the `D2 Ansible\n playbooks`_ (https://github.com/davedittrich/ansible-dims-playbooks)\n for example playbooks for doing these tasks.\n\n* Does not clean up the environment-specific ``tmp/`` directories.\n (You need to handle that in code, but at least they are less likely\n to end up in a Git commit.)\n\nUsage Concepts\n--------------\n\nThere is a separate **Usage** chapter with individual command documentation.\nThe remainder of this section covers higher level usage concepts necessary to\nbest use the ``python_secrets`` package in your open source software project.\n\n\nDirectories and files\n~~~~~~~~~~~~~~~~~~~~~\n\nThere are three file system concepts that are important to understand\nregarding secrets storage:\n\n#. The root *secrets base directory* for secrets storage;\n#. The *environment* for organizing a set of secrets and\n secret group descriptions;\n#. The *secrets* file and *group descriptions*.\n\n\n.. image:: https://asciinema.org/a/201503.png\n :target: https://asciinema.org/a/201503?autoplay=1\n :align: center\n :alt: Environments\n :width: 835px\n\n..\n\n\nSecrets Base Directory\n^^^^^^^^^^^^^^^^^^^^^^\n\n``psec`` expects to store all of files in a directory tree known as a\n*secrets base directory*. Originally, this was intended to be located in the\ncurrent user's home directory. Unless you over-ride the name of this directory,\nit defaults to ``.secrets`` on Linux and ``secrets`` on Windows.\n\nThe ability to locate this directory in a different file system path is\nsupported by command line options and an environment variable so you can store\nfiles on an exported file share, in a common location for use by a group on a\nworkstation, or to move the contents to an encrypted disk or a different\npartition with more disk space.\n\nThe first time you use ever use ``psec``, there will likely be no\ndirectory:\n\n.. code-block:: console\n\n $ tree ~/.secrets\n /Users/dittrich/.secrets [error opening dir]\n\n 0 directories, 0 files\n\n..\n\n.. note::\n\n The secrets base directory may be created automatically for you the\n first time you create an environment. For more information, see\n ``psec init --help``.\n\n..\n\nEnvironments\n^^^^^^^^^^^^\n\nEnvironments are sub-directories within the root secrets directory. You can\njust create the directory structure without any files. You create\none environment per set of unique secrets that you need to manage. This could\nbe one for open source *Program A*, one for *Program B*, etc., or it could be\none for *development*, one for *testing*, one for *production*, etc. (or any\ncombination).\n\n.. image:: https://asciinema.org/a/201505.png\n :target: https://asciinema.org/a/201505?autoplay=1\n :align: center\n :alt: Groups, secrets, generating and setting\n :width: 835px\n\n..\n\nThe command ``environments create`` creates an environment. Since this\nprogram is designed to support multiple environments, a name for the new\nenvironment is required. The name of the environment can be provided\nexplicitly, or it can be inferred from the base name of the current working\ndirectory:\n\n.. code-block:: console\n\n $ pwd\n /Users/dittrich/git/python_secrets\n $ psec environments create\n environment directory /Users/dittrich/.secrets/python_secrets created\n $ tree ~/.secrets\n /Users/dittrich/.secrets\n \u2514\u2500\u2500 python_secrets\n \u2514\u2500\u2500 secrets.d\n\n 2 directories, 0 files\n\n..\n\nLet's say we want to create empty environments for the three deployments\n(*development*, *testing*, and *production*). The names can be assigned\nexplicitly by (a) giving an argument on the command line, (b) using the ``-e`` or\n``--environment`` command line flag, or (c) by setting the environment variable\n``D2_ENVIRONMENT``:\n\n.. code-block:: console\n\n $ psec environments create development\n environment directory /Users/dittrich/.secrets/development created\n\n $ psec --environment testing environments create\n environment directory /Users/dittrich/.secrets/testing created\n\n $ D2_ENVIRONMENT=production psec environments create\n environment directory /Users/dittrich/.secrets/production created\n\n $ tree ~/.secrets\n /Users/dittrich/.secrets\n \u251c\u2500\u2500 development\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 secrets.d\n \u251c\u2500\u2500 production\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 secrets.d\n \u251c\u2500\u2500 python_secrets\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 secrets.d\n \u2514\u2500\u2500 testing\n \u2514\u2500\u2500 secrets.d\n\n 8 directories, 0 files\n\n..\n\nIf you want to create more than one environment at once, you will\nhave to specify all of the names on the command line as arguments:\n\n.. code-block:: console\n\n $ psec environments create development testing production\n environment directory /Users/dittrich/.secrets/development created\n environment directory /Users/dittrich/.secrets/testing created\n environment directory /Users/dittrich/.secrets/production created\n\n..\n\nIf you are using one source repository for building multiple deployments, of\ncourse you can't rely on the basename of the directory for all deployments. The\ndefault environment can be set, shown, or unset, using the ``environments\ndefault`` command.\n\n.. code-block:: console\n\n $ psec environments default --help\n usage: psec environments default [-h] [--unset-default] [environment]\n\n Manage default environment via file in cwd\n\n positional arguments:\n environment\n\n optional arguments:\n -h, --help show this help message and exit\n --unset-default Unset localized environment default\n\n..\n\nIf no default is explicitly set, the default that would be\napplied is returned:\n\n.. code-block:: console\n\n $ cd ~/git/python_secrets\n $ psec environments default\n default environment is \"python_secrets\"\n\n..\n\nYou can get a list of all available environments at any time,\nincluding which one would be the default used by sub-commands:\n\n.. code-block:: console\n\n $ psec environments list\n +-------------+---------+\n | Environment | Default |\n +-------------+---------+\n | development | No |\n | testing | No |\n | production | No |\n +-------------+---------+\n\n..\n\nThe following shows setting and unsetting the default:\n\n.. code-block:: console\n\n $ psec environments default testing\n default environment set to \"testing\"\n $ psec environments default\n testing\n $ psec environments list\n +-------------+---------+\n | Environment | Default |\n +-------------+---------+\n | development | No |\n | testing | Yes |\n | production | No |\n +-------------+---------+\n $ psec environments default --unset-default\n default environment unset\n\n..\n\nThe environment directories are useable for storing *all* secrets and\nsensitive files (e.g., backups of certificates, databases, etc.) associated\nwith an environment.\n\nFor convenience, there is a command ``environments tree`` that produces\noutput similar to the Unix ``tree`` command:\n\n.. code-block:: console\n\n $ psec -e d2 environments tree\n /Users/dittrich/.secrets/d2\n \u251c\u2500\u2500 backups\n \u2502 \u251c\u2500\u2500 black.secretsmgmt.tk\n \u2502 \u2502 \u251c\u2500\u2500 letsencrypt_2018-04-06T23:36:58PDT.tgz\n \u2502 \u2502 \u2514\u2500\u2500 letsencrypt_2018-04-25T16:32:20PDT.tgz\n \u2502 \u251c\u2500\u2500 green.secretsmgmt.tk\n \u2502 \u2502 \u251c\u2500\u2500 letsencrypt_2018-04-06T23:45:49PDT.tgz\n \u2502 \u2502 \u2514\u2500\u2500 letsencrypt_2018-04-25T16:32:20PDT.tgz\n \u2502 \u251c\u2500\u2500 purple.secretsmgmt.tk\n \u2502 \u2502 \u251c\u2500\u2500 letsencrypt_2018-04-25T16:32:20PDT.tgz\n \u2502 \u2502 \u251c\u2500\u2500 trident_2018-01-31T23:38:48PST.tar.bz2\n \u2502 \u2502 \u2514\u2500\u2500 trident_2018-02-04T20:05:33PST.tar.bz2\n \u2502 \u2514\u2500\u2500 red.secretsmgmt.tk\n \u2502 \u251c\u2500\u2500 letsencrypt_2018-04-06T23:45:49PDT.tgz\n \u2502 \u2514\u2500\u2500 letsencrypt_2018-04-25T16:32:20PDT.tgz\n \u251c\u2500\u2500 dittrich.asc\n \u251c\u2500\u2500 keys\n \u2502 \u2514\u2500\u2500 opendkim\n \u2502 \u2514\u2500\u2500 secretsmgmt.tk\n \u2502 \u251c\u2500\u2500 201801.private\n \u2502 \u251c\u2500\u2500 201801.txt\n \u2502 \u251c\u2500\u2500 201802.private\n \u2502 \u2514\u2500\u2500 201802.txt\n \u251c\u2500\u2500 secrets.d\n \u2502 \u251c\u2500\u2500 ca.json\n \u2502 \u251c\u2500\u2500 consul.json\n \u2502 \u251c\u2500\u2500 jenkins.json\n \u2502 \u251c\u2500\u2500 rabbitmq.json\n \u2502 \u251c\u2500\u2500 trident.json\n \u2502 \u251c\u2500\u2500 vncserver.json\n \u2502 \u2514\u2500\u2500 zookeper.json\n \u251c\u2500\u2500 secrets.json\n \u2514\u2500\u2500 vault_password.txt\n\n..\n\nTo just see the directory structure and not files, add the ``--no-files`` option:\n\n.. code-block:: console\n\n $ psec -e d2 environments tree --no-files\n /Users/dittrich/.secrets/d2\n \u251c\u2500\u2500 backups\n \u2502 \u251c\u2500\u2500 black.secretsmgmt.tk\n \u2502 \u251c\u2500\u2500 green.secretsmgmt.tk\n \u2502 \u251c\u2500\u2500 purple.secretsmgmt.tk\n \u2502 \u2514\u2500\u2500 red.secretsmgmt.tk\n \u251c\u2500\u2500 keys\n \u2502 \u2514\u2500\u2500 opendkim\n \u2502 \u2514\u2500\u2500 secretsmgmt.tk\n \u2514\u2500\u2500 secrets.d\n\n..\n\nSecrets and group descriptions\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe environment directories just created are all empty. Secrets are stored in a\nJSON file (``.json``) within the environment's directory, and group descriptions\nare stored in a drop-in directory with the same base name, but with an\nextention of ``.d`` instead of ``.json`` (following the Linux drop-in\nconfiguration style directories used by programs like ``rsyslog``, ``dnsmasq``,\netc.)\n\nThe default secrets file name is ``secrets.json``, which means the default\ndescriptions directory would be named ``secrets.d``.\n\nYou can define environment variables to point to the secrets base directory\nin which a set of different environments can be configured at one\ntime, to define the current environment, and to change the name\nof the secrets file to something else.\n\n.. code-block:: console\n\n $ env | grep ^D2_\n D2_SECRETS_BASEDIR=/Users/dittrich/.secrets\n D2_ENVIRONMENT=do\n\n..\n\nEach environment is in turn rooted in a directory with the environment's\nsymbolic name (e.g., ``do`` for DigitalOcean in this example, and ``goSecure``\nfor the GitHub `davedittrich/goSecure`_ VPN project.)\n\n.. code-block:: console\n\n $ tree -L 1 ~/.secrets\n /Users/dittrich/.secrets\n \u251c\u2500\u2500 do\n \u2514\u2500\u2500 goSecure\n\n 3 directories, 0 files\n\n..\n\n\nEach set of secrets for a given service or purpose is described in its own\nfile.\n\n.. code-block:: console\n\n .\n \u251c\u2500\u2500 secrets.d\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 ca.json\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 consul.json\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 jenkins.json\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 rabbitmq.json\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 trident.json\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 vncserver.json\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 zookeper.json\n \u2514\u2500\u2500 secrets.json\n\n..\n\nYou can see one of the descriptions files from the template\nin this repository using ``cat tests/secrets.d/myapp.json``:\n\n.. code-block:: json\n\n [\n {\n \"Variable\": \"myapp_pi_password\",\n \"Type\": \"password\",\n \"Prompt\": \"Password for myapp 'pi' user account\",\n \"Export\": \"DEMO_pi_password\"\n },\n {\n \"Variable\": \"myapp_app_password\",\n \"Type\": \"password\",\n \"Prompt\": \"Password for myapp web app\",\n \"Export\": \"DEMO_app_password\"\n },\n {\n \"Variable\": \"myapp_client_psk\",\n \"Type\": \"string\",\n \"Prompt\": \"Pre-shared key for myapp client WiFi AP\",\n \"Options\": \"*\",\n \"Export\": \"DEMO_client_psk\"\n },\n {\n \"Variable\": \"myapp_client_ssid\",\n \"Type\": \"string\",\n \"Prompt\": \"SSID for myapp client WiFi AP\",\n \"Options\": \"myapp_ssid,*\",\n \"Export\": \"DEMO_client_ssid\"\n },\n {\n \"Variable\": \"myapp_ondemand_wifi\",\n \"Type\": \"boolean\",\n \"Prompt\": \"'Connect on demand' when connected to wifi\",\n \"Options\": \"true,false\",\n \"Export\": \"DEMO_ondemand_wifi\"\n },\n {\n \"Variable\": \"myapp_optional_setting\",\n \"Type\": \"boolean\",\n \"Prompt\": \"Optionally do something\",\n \"Options\": \"false,true\",\n \"Export\": \"DEMO_options_setting\"\n }\n ]\n\n..\n\nThe ``psec`` program uses the `openstack/cliff`_ command line\ninterface framework, which supports multiple output formats. The default\nformat the ``table`` format, which makes for nice clean output. (Other\nformats will be described later.)\n\nThe groups can be listed using the ``groups list`` command:\n\n.. code-block:: console\n\n $ psec groups list\n +---------+-------+\n | Group | Items |\n +---------+-------+\n | jenkins | 1 |\n | myapp | 4 |\n | trident | 2 |\n +---------+-------+\n\n..\n\nThe variables in one or more groups can be shown with\nthe ``groups show`` command:\n\n.. code-block:: console\n\n $ psec groups show trident myapp\n +---------+-----------------------+\n | Group | Variable |\n +---------+-----------------------+\n | trident | trident_sysadmin_pass |\n | trident | trident_db_pass |\n | myapp | myapp_app_password |\n | myapp | myapp_client_psk |\n | myapp | myapp_client_ssid |\n | myapp | myapp_ondemand_wifi |\n | myapp | myapp_pi_password |\n +---------+-----------------------+\n\n..\n\nWhen integrating a new open source tool or project, you can create\na new group and clone its secrets descriptions. This does not copy\nany values, just the descriptions, allowing the current environment\nto manage its own values.\n\n.. code-block:: console\n\n $ psec groups create newgroup --clone-from ~/git/goSecure/secrets/secrets.d/gosecure.json\n created new group \"newgroup\"\n $ psec groups list 2>/dev/null\n +----------+-------+\n | Group | Items |\n +----------+-------+\n | jenkins | 1 |\n | myapp | 5 |\n | newgroup | 12 |\n | trident | 2 |\n +----------+-------+\n\n..\n\n\nShowing Secrets\n~~~~~~~~~~~~~~~\n\nTo examine the secrets, use the ``secrets show`` command:\n\n.. code-block:: console\n\n $ psec secrets show\n +------------------------+----------+----------+------------------------+\n | Variable | Type | Value | Export |\n +------------------------+----------+----------+------------------------+\n | jenkins_admin_password | password | REDACTED | jenkins_admin_password |\n | myapp_app_password | password | REDACTED | DEMO_app_password |\n | myapp_client_psk | string | REDACTED | DEMO_client_ssid |\n | myapp_client_ssid | string | REDACTED | DEMO_client_ssid |\n | myapp_ondemand_wifi | boolean | REDACTED | DEMO_ondemand_wifi |\n | myapp_pi_password | password | REDACTED | DEMO_pi_password |\n | trident_db_pass | password | REDACTED | trident_db_pass |\n | trident_sysadmin_pass | password | REDACTED | trident_sysadmin_pass |\n +------------------------+----------+----------+------------------------+\n\n..\n\nBy default, the values of secrets are redacted when output. To show\nthe values in clear text in the terminal output, add the ``--no-redact`` flag:\n\n.. code-block:: console\n\n $ psec secrets show --no-redact\n +------------------------+----------+------------------------------+------------------------+\n | Variable | Type | Value | Export |\n +------------------------+----------+------------------------------+------------------------+\n | jenkins_admin_password | password | fetch.outsider.awning.maroon | jenkins_admin_password |\n | myapp_app_password | password | fetch.outsider.awning.maroon | DEMO_app_password |\n | myapp_client_psk | string | PSK | DEMO_client_psk |\n | myapp_client_ssid | string | SSID | DEMO_client_ssid |\n | myapp_ondemand_wifi | boolean | true | DEMO_ondemand_wifi |\n | myapp_pi_password | password | fetch.outsider.awning.maroon | DEMO_pi_password |\n | trident_db_pass | password | fetch.outsider.awning.maroon | trident_db_pass |\n | trident_sysadmin_pass | password | fetch.outsider.awning.maroon | trident_sysadmin_pass |\n +------------------------+----------+------------------------------+------------------------+\n\n..\n\nIf you don't care about redaction and want to turn it off and save\nthe dozen keystrokes it takes to type `` --no-redact``, you can export\nthe environment variable ``D2_NO_REDACT`` set to (case-insensitive)\n\"true\", \"1\", or \"yes\". Anything else leaves the default the same.\nWe'll do this now for later examples.\n\n.. code-block:: console\n\n $ export D2_NO_REDACT=true\n\n..\n\nThe default is also to show all secrets. If you only want to process a\nsubset of secrets, you have two ways to do this.\n\n#. Specify the variables you want to show on the command line as arguments:\n\n .. code-block:: console\n\n $ psec secrets show rabbitmq_default_user_pass rabbitmq_admin_user_pass\n +----------------------------+----------+--------------------------------------+\n | Variable | Type | Value |\n +----------------------------+----------+--------------------------------------+\n | rabbitmq_default_user_pass | password | handheld.angrily.letdown.frisk |\n | rabbitmq_admin_user_pass | password | handheld.angrily.letdown.frisk |\n +----------------------------+----------+--------------------------------------+\n\n ..\n\n#. Use the ``--group`` flag and specify the group(s) you want to show\n as command line arguments:\n\n .. code-block:: console\n\n $ psec secrets show --group jenkins trident\n +----------------------------+----------+--------------------------------------+\n | Variable | Type | Value |\n +----------------------------+----------+--------------------------------------+\n | jenkins_admin_password | password | handheld.angrily.letdown.frisk |\n | trident_db_pass | password | handheld.angrily.letdown.frisk |\n | trident_sysadmin_pass | password | handheld.angrily.letdown.frisk |\n +----------------------------+----------+--------------------------------------+\n\n ..\n\n\nDescribing Secrets and Secret Types\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo describe the secrets in the select environment, use the\n``secrets describe`` command:\n\n.. code-block:: console\n\n $ psec secrets describe\n +----------------------------+----------+--------------------------------------------+\n | Variable | Type | Prompt |\n +----------------------------+----------+--------------------------------------------+\n | google_oauth_client_id | string | Google OAuth2 client id |\n | google_oauth_client_secret | string | Google OAuth2 client secret |\n | google_oauth_refresh_token | string | Google OAuth2 refresh token |\n | google_oauth_username | None | google_oauth_username |\n | jenkins_admin_password | password | Password for Jenkins \"admin\" account |\n | myapp_app_password | password | Password for myapp web app |\n | myapp_client_psk | string | Pre-shared key for myapp client WiFi AP |\n | myapp_client_ssid | string | SSID for myapp client WiFi AP |\n | myapp_ondemand_wifi | boolean | \"Connect on demand\" when connected to wifi |\n | myapp_pi_password | password | Password for myapp \"pi\" user account |\n | trident_db_pass | password | Password for Trident postgres database |\n | trident_sysadmin_pass | password | Password for Trident sysadmin account |\n +----------------------------+----------+--------------------------------------------+\n $ psec secrets describe --group trident\n +-----------------------+----------+----------------------------------------+\n | Variable | Type | Prompt |\n +-----------------------+----------+----------------------------------------+\n | trident_db_pass | password | Password for Trident postgres database |\n | trident_sysadmin_pass | password | Password for Trident sysadmin account |\n +-----------------------+----------+----------------------------------------+\n\n..\n\nTo get a description of the available secret types, add the ``--types`` flag.\n\n.. code-block:: console\n\n $ psec secrets describe --types\n +------------------+----------------------------------+\n | Type | Description |\n +------------------+----------------------------------+\n | password | Simple (xkcd) password string |\n | string | Simple string |\n | boolean | Boolean (\"true\"/\"false\") |\n | crypt_6 | crypt() SHA512 (\"$6$\") |\n | token_hex | Hexadecimal token |\n | token_urlsafe | URL-safe token |\n | sha256_digest | DIGEST-SHA256 (user:pass) digest |\n | uuid4 | UUID4 token |\n | random_base64 | Random BASE64 token |\n +------------------+----------------------------------+\n\n..\n\n.. note::\n\n The type ``string`` is for secrets that are managed by another entity that you\n must obtain and use to access some remote service (e.g., the pre-shared key for\n someone's WiFi network, or an API key for accessing a cloud service provider's\n platform). All other types are structured secret types that you generate for\n configuring services.\n\n..\n\nGenerating and Setting variables\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSecrets are generated using the ``secrets generate`` command\nand are set manually using the ``secrets set`` command.\n\n.. code-block:: console\n\n $ psec help secrets generate\n usage: psec secrets generate [-h] [-U] [args [args ...]]\n\n Generate values for secrets\n\n positional arguments:\n args\n\n optional arguments:\n -h, --help show this help message and exit\n -U, --unique Generate unique values for each type of secret (default:\n False)\n\n ..\n\n.. code-block:: console\n\n $ psec secrets set --help\n usage: psec secrets set [-h] [--undefined] [args [args ...]]\n\n Set values manually for secrets\n\n positional arguments:\n args\n\n optional arguments:\n -h, --help show this help message and exit\n --undefined Set values for undefined variables (default: False)\n\n..\n\nTo regenerate all of the non-string secrets at once, using the same value for\neach type of secret to simplify things, use the ``secrets generate`` command:\n\n.. code-block:: console\n\n $ psec secrets generate\n $ psec secrets show --column Variable --column Value\n +----------------------------+----------------------------------------------+\n | Variable | Value |\n +----------------------------+----------------------------------------------+\n | trident_db_pass | gargle.earlobe.eggplant.kissable |\n | ca_rootca_password | gargle.earlobe.eggplant.kissable |\n | consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |\n | jenkins_admin_password | gargle.earlobe.eggplant.kissable |\n | rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |\n | rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |\n | trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |\n | vncserver_password | gargle.earlobe.eggplant.kissable |\n | zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |\n +----------------------------+----------------------------------------------+\n\n..\n\nYou can set one or more variables manually using ``secrets set`` and\nspecifying the variable and value in the form ``variable=value``:\n\n.. code-block:: console\n\n $ psec secrets set trident_db_pass=\"rural coffee purple sedan\"\n $ psec secrets show --column Variable --column Value\n +----------------------------+----------------------------------------------+\n | Variable | Value |\n +----------------------------+----------------------------------------------+\n | trident_db_pass | rural coffee purple sedan |\n | ca_rootca_password | gargle.earlobe.eggplant.kissable |\n | consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |\n | jenkins_admin_password | gargle.earlobe.eggplant.kissable |\n | rabbitmq_default_user_pass | gargle.earlobe.eggplant.kissable |\n | rabbitmq_admin_user_pass | gargle.earlobe.eggplant.kissable |\n | trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |\n | vncserver_password | gargle.earlobe.eggplant.kissable |\n | zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |\n +----------------------------+----------------------------------------------+\n\n..\n\n.. caution::\n\n Note in the example above that the command argument is\n ``trident_db_pass=\"rural coffee purple sedan\"`` and not\n ``trident_db_pass='rural coffee purple sedan'``.\n When using the ``variable=value`` form of the ``secrets set``\n command with a value that contains spaces, you **must** quote the value with\n the double-quote character (``\"``) as opposed to the single-quote\n (apostrophe, or ``'``) character. The Bash shell (and possibly other\n shells) will not properly parse the command line and the resulting\n ``sys.argv`` argument vector will be incorrectly set as seen here:\n\n .. code-block:: console\n\n _sys.argv[1:] = {list} <class 'list'>: ['--debug', 'secrets', 'set', 'trident_db_password=rural coffee purple sedan']\n 0 = {str} '--debug'\n 1 = {str} 'secrets'\n 2 = {str} 'set'\n 3 = {str} 'trident_db_password=rural coffee purple sedan'\n __len__ = {int} 4\n\n\n _sys.argv[1:] = {list} <class 'list'>: ['--debug', 'secrets', 'set', \"trident_db_password='rural\", 'coffee', 'purple', \"sedan'\"]\n 0 = {str} '--debug'\n 1 = {str} 'secrets'\n 2 = {str} 'set'\n 3 = {str} 'trident_db_password=\\\\'rural'\n 4 = {str} 'coffee'\n 5 = {str} 'purple'\n 6 = {str} 'sedan\\\\''\n __len__ = {int} 7\n\n..\n\nOr you can generate one or more variables in a similar manner by adding\nthem to the command line as arguments to ``secrets generate``:\n\n.. code-block:: console\n\n $ psec secrets generate rabbitmq_default_user_pass rabbitmq_admin_user_pass\n $ psec secrets show --column Variable --column Value\n +----------------------------+----------------------------------------------+\n | Variable | Value |\n +----------------------------+----------------------------------------------+\n | trident_db_pass | rural.coffee.purple.sedan |\n | ca_rootca_password | gargle.earlobe.eggplant.kissable |\n | consul_key | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |\n | jenkins_admin_password | gargle.earlobe.eggplant.kissable |\n | rabbitmq_default_user_pass | embezzle.xerox.excess.skydiver |\n | rabbitmq_admin_user_pass | embezzle.xerox.excess.skydiver |\n | trident_sysadmin_pass | gargle.earlobe.eggplant.kissable |\n | vncserver_password | gargle.earlobe.eggplant.kissable |\n | zookeeper_uuid4 | 769a77ad-b06f-4018-857e-23f970c777c2 |\n +----------------------------+----------------------------------------------+\n\n..\n\n\nA set of secrets for an open source project can be bootstrapped using the\nfollowing steps:\n\n#. Create a template secrets environment directory that contains just\n the secrets definitions. This example uses the template found\n in the `davedittrich/goSecure`_ repository\n (directory https://github.com/davedittrich/goSecure/tree/master/secrets).\n\n#. Use this template to clone a secrets environment, which will initially\n be empty:\n\n .. code-block:: console\n\n $ psec environments create test --clone-from ~/git/goSecure/secrets\n new password variable \"gosecure_app_password\" is unset\n new string variable \"gosecure_client_ssid\" is unset\n new string variable \"gosecure_client_ssid\" is unset\n new string variable \"gosecure_client_psk\" is unset\n new password variable \"gosecure_pi_password\" is unset\n new string variable \"gosecure_pi_pubkey\" is unset\n environment directory /Users/dittrich/.secrets/test created\n\n ..\n\n .. note::\n\n The warnings about undefined new variables are presented on the standard\n error file handle (a.k.a., ``&2``). You get rid of them on the console by\n redirecting ``stderr`` to ``/dev/null`` or a file:\n\n .. code-block:: console\n\n $ psec environments create test --clone-from ~/git/goSecure/secrets 2>/dev/null\n environment directory /Users/dittrich/.secrets/test created\n\n ..\n\n .. code-block:: console\n\n $ psec -e test secrets show --no-redact --fit-width\n +-----------------------+----------+-------+\n | Variable | Type | Value |\n +-----------------------+----------+-------+\n | gosecure_app_password | password | None |\n | gosecure_client_ssid | string | None |\n | gosecure_client_psk | string | None |\n | gosecure_pi_password | password | None |\n | gosecure_pi_pubkey | string | None |\n +-----------------------+----------+-------+\n\n ..\n\n#. First, generate all secrets whose type is not ``string``:\n\n .. code-block:: console\n\n $ psec -e test secrets generate\n new password variable \"gosecure_app_password\" is unset\n new string variable \"gosecure_client_ssid\" is unset\n new string variable \"gosecure_client_ssid\" is unset\n new string variable \"gosecure_client_psk\" is unset\n new password variable \"gosecure_pi_password\" is unset\n new string variable \"gosecure_pi_pubkey\" is unset\n\n $ psec -e test secrets show --no-redact --fit-width\n +-----------------------+----------+------------------------------+\n | Variable | Type | Value |\n +-----------------------+----------+------------------------------+\n | gosecure_app_password | password | brunt.outclass.alike.turbine |\n | gosecure_client_psk | string | None |\n | gosecure_client_ssid | string | None |\n | gosecure_pi_password | password | brunt.outclass.alike.turbine |\n | gosecure_pi_pubkey | string | None |\n +-----------------------+----------+------------------------------+\n\n ..\n\n#. Finally, manually set the remaining ``string`` type variables:\n\n .. code-block:: console\n\n $ psec -e test secrets set --undefined\n new string variable \"gosecure_client_psk\" is unset\n new string variable \"gosecure_client_ssid\" is unset\n new string variable \"gosecure_pi_pubkey\" is unset\n Pre-shared key for goSecure client WiFi AP? [None]: atjhK5AlsQMw3Zh\n SSID for goSecure client WiFi AP? [None]: YourWiFiSSID\n SSH public key for accessing \"pi\" account? [None]: @~/.ssh/new_rsa.pub\n\n $ psec -e test secrets show --no-redact --fit-width\n +-----------------------+----------+------------------------------------------------------------------------------------------+\n | Variable | Type | Value |\n +-----------------------+----------+------------------------------------------------------------------------------------------+\n | gosecure_app_password | password | brunt.outclass.alike.turbine |\n | gosecure_client_psk | string | atjhK5AlsQMw3Zh\n | gosecure_client_ssid | string | YourWiFiSSID |\n | gosecure_pi_password | password | brunt.outclass.alike.turbine |\n | gosecure_pi_pubkey | string | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU |\n | | | uQ2/NVikfOfPo5mt9YTQyqRbeBzKlNgbHnsxh0AZatjhK5AlsQMw3ZhZUcLYZbt7szuQy8ineN0potlCJoVaMSOb |\n | | | 9htf9gAPvzwxUnHxg35jPCzAXYAi3Erc6y338+CL0XxQvCogXOA+MwH7wZGgdT3WpupLG/7HAr/3KJEQQk1FlS2m |\n | | | Rd+WuewnLbKkqBP21N+48ccq6XhEhAmlzzr9SENw5DMmrvMAYIYkoTwUeD3Qx4YebjFkCxZw+w7AafEFn0Kz6vCX |\n | | | 4mp/6ZF/Ko+o04HM2sVr6wtCu2dB dittrich@localhost |\n +-----------------------+----------+------------------------------------------------------------------------------------------+\n\n ..\n\n.. note::\n\n If you don't want to see the warnings about new variables that are not\n defined, simply add the ``-q`` flag.\n\n .. code-block:: console\n\n $ psec -q secrets generate\n $ psec -q secrets set --undefined\n Pre-shared key for goSecure client WiFi AP? [None]:\n\n ..\n\n..\n\nYou are now ready to compile your software, or build your project!\n\nThere is also a mechanism to run simple commands (i.e., basic arguments with\nno special inline command substitution or variable expansion features of\nshells like ``bash``) and use the resulting output as the value.\n\nFor this example, let's assume an environment that requires a CIDR\nnotation address for ingres access control (e.g., when using Amazon\nWeb Services to allow control of instances from your remote laptop).\n\n.. code-block:: console\n\n $ psec -e xgt secrets set aws_cidr_allowed=\"\"\n $ psec -e secrets show --no-redact aws_cidr_allowed\n +------------------+--------+-------+\n | Variable | Type | Value |\n +------------------+--------+-------+\n | aws_cidr_allowed | string | |\n +------------------+--------+-------+\n\n..\n\nThe ``psec`` program has a utility feature that will return\nthe current routable IP source address as an IP address, or using CIDR\nnotation. The variable can be set in one of two ways:\n\n#. Via (non-interactive) inline command subtitution from the terminal shell:\n\n .. code-block:: console\n\n $ psec -e xgt secrets set aws_cidr_allowed=\"$(psec utils myip --cidr)\"\n\n ..\n\n#. Interactively when prompted using simple command line form:\n\n .. code-block:: console\n\n $ psec -e xgt secrets set aws_cidr_allowed\n aws_cidr_allowed? []: !psec utils myip --cidr\n\n ..\n\n\nThe variable now contains the output of the specified program:\n\n.. code-block:: console\n\n $ psec secrets show --no-redact aws_cidr_allowed\n +------------------+--------+------------------+\n | Variable | Type | Value |\n +------------------+--------+------------------+\n | aws_cidr_allowed | string | 93.184.216.34/32 |\n +------------------+--------+------------------+\n\n..\n\n.. note::\n\n If you work from behind a static NAT firewall, this IP address will\n likely not change very often (if at all). If you are using a mobile device\n that is assigned differing DHCP addresses depending on location, the IP address\n may change fairly regularly and the initial AWS Security Group setting will\n begin to block access to your cloud instances. Programs like ``terraform``\n can refresh their state, allowing you to simply reset the variable used to\n create the Security Group and re-apply the plan to regenerate the AWS\n Security Group and re-enable your remote access.\n\n..\n\n\nSharing secrets\n~~~~~~~~~~~~~~~\n\nThe ``psec`` program has a mechanism for sharing secrets with\nothers using GPG encrypted email messages for securing secrets in transit\nand at rest in users' inboxes. Email is sent using Google's OAuth2\nauthenticated SMTP services.\n\n.. note::\n\n The Electronic Frontier Foundation (EFF) has a `Surveillance Self-Defense\n Guide`_ that includes guides on `How to Use PGP for Linux`_ and other operating\n systems. Follow their instructions if you are new to PGP/GPG.\n\n..\n\nThe command is ``secrets send``.\n\n.. code-block:: console\n\n $ psec secrets send --help\n usage: psec secrets send [-h] [-T] [--test-smtp] [-H SMTP_HOST]\n [-U SMTP_USERNAME] [-F SMTP_SENDER] [-S SMTP_SUBJECT]\n [args [args ...]]\n\n Send secrets using GPG encrypted email. Arguments are USERNAME@EMAIL.ADDRESS\n and/or VARIABLE references.\n\n positional arguments:\n args\n\n optional arguments:\n -h, --help show this help message and exit\n -T, --refresh-token Refresh Google API Oauth2 token and exit (default:\n False)\n --test-smtp Test Oauth2 SMTP authentication and exit (default:\n False)\n -H SMTP_HOST, --smtp-host SMTP_HOST\n SMTP host (default: localhost)\n -U SMTP_USERNAME, --smtp-username SMTP_USERNAME\n SMTP authentication username (default: None)\n -F SMTP_SENDER, --from SMTP_SENDER\n Sender address (default: 'noreply@nowhere')\n -S SMTP_SUBJECT, --subject SMTP_SUBJECT\n Subject line (default: 'For Your Information')\n\n..\n\nAny arguments (``args``) that contain the ``@`` symbol are assumed to be email\naddresses while the rest are assumed to be the names of secrets variables\nto be sent.\n\nAll recipients must have GPG public keys in your keyring. An exception is thrown\nif no GPG key is associated with the recipient(s) email addresses.\n\n.. code-block:: console\n\n $ psec secrets send dittrich@u.washington.edu myapp_app_password\n Setting homedir to '/Users/dittrich/.gnupg'\n\n Initialised settings:\n binary: /usr/local/bin/gpg\n binary version: 1.4.11\\ncfg:pubkey:1;2;3;16;17\\ncfg:cipher:2;3;4;7;8;9;10;11;12;13\\ncfg:ciphername:3DES;CAST5;BLOWFISH;AES;AES192;AES256;TWOFISH;CAMELLIA128;CAMELLIA192;CAMELLIA256\\ncfg:digest:1;2;3;8;9;10;11\\ncfg:digestname:MD5;SHA1;RIPEMD160;SHA256;SHA384;SHA512;SHA224\\ncfg:compress:0;1;2;3\\n'\n homedir: /Users/dittrich/.gnupg\n ignore_homedir_permissions: False\n keyring: /Users/dittrich/.gnupg/pubring.gpg\n secring: /Users/dittrich/.gnupg/secring.gpg\n default_preference_list: SHA512 SHA384 SHA256 AES256 CAMELLIA256 TWOFISH AES192 ZLIB ZIP Uncompressed\n keyserver: hkp://wwwkeys.pgp.net\n options: None\n verbose: False\n use_agent: False\n\n Creating the trustdb is only available with GnuPG>=2.x\n sent encrypted secrets to dittrich@u.washington.edu\n\n..\n\nUse ``-q`` to produce no extraneous output.\n\n.. code-block:: console\n\n $ psec -q secrets send dittrich@u.washington.edu myapp_app_password\n\n..\n\nThe resulting email looks like this:\n\n.. code-block:: console\n\n Message-ID: <5bac64ce.1c69fb81.b136e.45ae@mx.google.com>\n Date: Wed, 26 Sep 2018 22:04:14 -0700 (PDT)\n From: dave.dittrich@gmail.com\n X-Google-Original-From: noreply@nowhere\n Content-Type: multipart/related; boundary=\"===============6413073026511107073==\"\n MIME-Version: 1.0\n Subject: For Your Information\n To: dittrich@u.washington.edu\n\n This is a multi-part message in MIME format.\n --===============6413073026511107073==\n Content-Type: multipart/alternative; boundary=\"===============2830935289665347054==\"\n MIME-Version: 1.0\n\n --===============2830935289665347054==\n Content-Type: text/plain; charset=\"utf-8\"\n MIME-Version: 1.0\n Content-Transfer-Encoding: base64\n\n LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tCgpoUUlXQStSZlhnK3dLTGJlRUFnZlFNcjZYb0lT\n cS9BaTlMbEVpZTFTejd5ckEzUmN4SWdjb01XTUNSM3JBaXBHCjF0TTJoZkpxRGJZOThSOEVST01F\n aVltSzR2aVJ4ZjgrSU54NU54SUJPbFh1T1JQTy82NElUKzdrVSt5aDZGV00KNU1MK0Jkb21sQzNF\n eC9pd3hwbTJ1R2FPczFpcU9DaDIxbTd5RnJWYkNVSW5NN1ZiMTEwck41aXNOZ3BFdndrQgpaZHhp\n alJqazdtYVl1eFNkc2c3Y2RVQ29uSmdBR214QU0vZkFzOTREcHNrYkwzMFpqZE1iRHlMbUk4NWp2\n QU45CjU3KzAxLzM1MEMyN1hrbEUxdEZudWNlRkRqZ04zeEd4K2Zud0pqdkFpNUpaVHltanRkQi9r\n dUZUMlJTTmJJTlAKMWRZdHp4WGxNeVd0SVphNDVYcHdNenZ1TkFTbEJtbENjQXk4YlluSEJmeFRy\n SGdJSUlCMlZNY1N6dmdjR3BtVApkYzZqaDVOeEV1bWljOWdXMmplSnFqRHRtdW9Ib3dxZldZb2xX\n bGlXUTMrNDNzeVkrdHFlMGgvWEwzS2ZxSTMrClZzWWdyQmpGd0hnem1INEthMWxucXdUZkMzZTJ3\n cUI4Uk5hcllqcXAzbHFQOVBhMHdzSVVWMHVYN2dhL01kVWcKdHNRSktPWWJRTnlXVTFLZEZWNHl4\n Ynp1TWVlQ3ltMmxMbXJwVks5T3hCV04vbCtXMjRsWmhkck9TcGFJQnpNdgpnc1p3VWVuVzBXR054\n bklwUGhoSWRuVE40ZlNscE5JVDhMcmJYeUhoY2ZVS2lsUDNpeEVPRS9Lc25QUFJNTURFCk9SY0xT\n Z3FMMTB4b0toMnNzZTNxNG5RaHZkZW5IVVVxVjJ0WW1UVmRCNVl3cTN1MFdtY3BGSGU2NnBZeTBB\n VSsKdzRjb2JVM2crQWtJMHBNQnllRzZYaWV4VzF1UzRLVVVnaFlhWVlYQ2dnazJZNEpZT05QSDJJ\n NlIydmxuNjFsVApZdm1tR0NNamw3cC9pTnE2RWJpbndoMnNsbkpLMHd3S1BIbVBPUjJvRjdWREN0\n dE9idHA0cEZUWTNHalByc0dRCkNDT3dYR2hCSFVQRnY2c3R4NEdtUi9GUWpBRWxxaEpjQWtTbDFz\n WWhsUFRhSmEyVGgyNG81L1lPUmxRaHhhRUgKUEFrNFgzcGVCMk9UVjRNR2RCOD0KPTc0aXEKLS0t\n LS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=\n\n --===============2830935289665347054==\n Content-Type: text/html; charset=\"utf-8\"\n MIME-Version: 1.0\n Content-Transfer-Encoding: base64\n\n VGhlIGZvbGxvd2luZyBzZWNyZXQgaXMgYmVpbmcgc2hhcmVkIHdpdGggeW91OgoKbXlhcHBfYXBw\n X3Bhc3N3b3JkPWJydW50IG91dGNsYXNzIGFsaWtlIHR1cmJpbmU=\n\n --===============2830935289665347054==--\n\n --===============6413073026511107073==--\n\n..\n\nDecrypted, it looks like this:\n\n.. code-block:: console\n\n Date: Wed, 26 Sep 2018 22:04:14 -0700 (PDT)\n From: dave.dittrich@gmail.com\n Subject: For Your Information\n To: dittrich@u.washington.edu\n\n The following secret is being shared with you:\n\n myapp_app_password=brunt.outclass.alike.turbine\n\n --\n Sent using psec version 23.4.1\n https://pypi.org/project/python-secrets/\n https://github.com/davedittrich/python_secrets\n\n..\n\nA group of secrets required for Google's `OAuth 2.0 Mechanism`_ is provided\nand must be set according to Google's instructions. See also:\n\n+ https://github.com/google/gmail-oauth2-tools/wiki/OAuth2DotPyRunThrough\n\n+ http://blog.macuyiko.com/post/2016/how-to-send-html-mails-with-oauth2-and-gmail-in-python.html\n\n+ https://developers.google.com/api-client-library/python/guide/aaa_oauth\n\n+ https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py\n\n+ https://developers.google.com/identity/protocols/OAuth2\n\n\n.. code-block:: console\n\n $ psec groups show oauth\n +-------+----------------------------+\n | Group | Variable |\n +-------+----------------------------+\n | oauth | google_oauth_client_id |\n | oauth | google_oauth_client_secret |\n | oauth | google_oauth_refresh_token |\n +-------+----------------------------+\n\n..\n\n\nProcessing templates\n~~~~~~~~~~~~~~~~~~~~\n\n.. image:: https://asciinema.org/a/201507.png\n :target: https://asciinema.org/a/201507?autoplay=1\n :align: center\n :alt: Rendering templates outside the source repo directory\n :width: 835px\n\n..\n\n\nOutputting structured information for use in other scripts\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOnce secrets are created and stored, they will eventually need to be accessed\nin order to use them in program execution. This can be done by passing the\n``.json`` secrets file itself to a program, or by outputting the variables in\nother formats like CSV, JSON, or as environment type variables.\n\nPassing the secrets file by path\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nOne way to do this is to take advantage of command line options like\n`Ansible`_'s ``--extra-vars`` and passing it a path to the ``.json`` secrets\nfile. (See `Passing Variables On The Command Line`_). Here is how to do\nit.\n\nLet's assume we want to use ``consul_key`` variable to configure Consul\nusing Ansible. Here is the variable as stored:\n\n.. code-block:: console\n\n $ psec secrets show consul_key\n +------------+-----------+----------------------------------------------+\n | Variable | Type | Value |\n +------------+-----------+----------------------------------------------+\n | consul_key | token_hex | HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY= |\n +------------+-----------+----------------------------------------------+\n\n..\n\nUsing Ansible's ``debug`` module, we can verify that this variable is not\nset by any previously loaded Ansible inventory:\n\n.. code-block:: console\n\n $ ansible -i localhost, -m debug -a 'var=consul_key' localhost\n localhost | SUCCESS => {\n \"consul_key\": \"VARIABLE IS NOT DEFINED!\"\n }\n\n..\n\nIn order for Ansible to set the ``consul_key`` variable outside of any\npre-defined inventory files, we need to pass a file path to the\n``--extra-vars`` option. The path can be obtained using the\n``psec secrets path`` command:\n\n.. code-block:: console\n\n $ psec secrets path\n /Users/dittrich/.secrets/python_secrets/secrets.json\n\n..\n\nIt is possible to run this command in an in-line command expansion operation in\nBash. Ansible expects the file path passed to ``-extra-vars`` to start with an\n``@`` character, so the command line to use would look like this:\n\n.. code-block:: console\n\n $ ansible -i localhost, -e @\"$(psec secrets path)\" -m debug -a 'var=consul_key' localhost\n localhost | SUCCESS => {\n \"consul_key\": \"HEvUAItLFZ0+GjxfwTxLDKq5Fbt86UtXrInzpf71GGY=\"\n }\n\n..\n\nAnsible now has the value and can use it in templating configuration files, or\nso forth.\n\nOther programs like Hashicorp `terraform`_ look for environment variables that\nbegin with ``TF_VAR_`` and use them to set ``terraform`` variables for use\nin modules. To prove we are running in a sub-shell, we will first change the\nshell prompt.\n\n.. code-block:: console\n\n $ PS1=\"test> \"\n test> psec -e test --export-env-vars --env-var-prefix=\"TEST_\" run bash\n $ env | grep '^TEST_'\n TEST_gosecure_pi_pubkey=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+qUIucrPvRkTmY0tgxr9ac/VtBUHhYHfOdDVpU99AcryLMWiU [...]\n TEST_gosecure_client_psk=atjhK5AlsQMw3Zh\n TEST_gosecure_client_ssid=YourWiFiSSID\n TEST_gosecure_pi_password=brunt.outclass.alike.turbine\n TEST_gosecure_app_password=brunt.outclass.alike.turbine\n $ exit\n test>\n\n..\n\n.. image:: https://asciinema.org/a/201510.png\n :target: https://asciinema.org/a/201510?autoplay=1\n :align: center\n :alt: Exporting secrets via the environment\n :width: 835px\n\n..\n\n\n\nOperational Security\n----------------------\n\nAs noted in the Limitations section above, secrets are stored in plaintext\nplaintext form (i.e., they are *not* encrypted) in files. Those files are in\nturn stored in a directory in the file system, subject to Linux file\nownership and permission access controls.\n\nThe default location for storing these files is in an *environment directory*\nin a subdirectory of the user's home directory whose name starts with a period\ncharacter (a.k.a., a *dot*). Files (or directories) whose name starts with a\nperiod are known as *dot files*, or *hidden files* because the `ls` command\ndoes not show it unless you use the `-a` flag.\n\nThe secrets environment directories can also be used to store other files\nbesides secrets. One such use case is storing JSON Web Tokens (JWTs) used as\nbearer tokens by protocols like Google's `OAuth 2.0 Mechanism`_ for securing\naccess to web services and APIs. While this improves security in terms of\nremote access, is not not without its own risks (including the JWT file being\nstored in the file system for an indefinite period of time).\n\n* `JSON Web Tokens (JWT) are Dangerous for User Sessions\u2014Here\u2019s a\n Solution`_, by Raja Rao, June 24, 2021\n\n* `Stop Using JSON Web Tokens For Authentication. Use Stateful Sessions\n Instead`_, by Francisco Sainz, April 4, 2022\n\n* `What\u2019s the Secure Way to Store JWT?`_, by Yang Liu, July 23, 2020\n\nBesides JWTs, other use cases for storing sensitive files within `psec`\nenvironments include backups of database contents, Let's Encrypt certificates,\nSSH keys, or other secrets necessary for ensuring cloud instances can be\ndestroyed and recreated without losing state or requiring regeneration\n(and redistribution or revalidation) of secrets.\n\nThe output of `init --help` mentions this risk and offers a way to mitigate\nsome of the risk by locating the secrets storage base directory within a\ndirectory that is stored on an encrypted USB-connected disk device or encrypted\ndisk image, or a removable device or remote file system, that is only mounted\nwhen needed and unmounted as soon as possible. This ensures sensitive data that\nare not being actively used are left encrypted in storage. The\n`D2_SECRETS_BASEDIR` environment variable or `-d` option allow you to specify\nthe directory to use.\n\nThe `psec` CLI has a secure deletion mechanism that over-writes file contents\nprior to deletion, helping to reduce leaving remnants of secrets in unallocated\nfile system storage, similar to the way the Linux `shred` command works.\n\n\n\nPython Script Security\n----------------------\n\nLast, but certainly not least, take the time to read up on `Python Security`_\nand understand the types and sources of security vulnerabilities related to\nPython programs. Keep these ideas in mind when using and/or modifying this\nprogram.\n\nAs part of testing, the `Bandit`_ security validation program is used.\n(See `Getting started with Bandit`_).\n\n.. _Bandit: https://pypi.org/project/bandit/\n.. _Getting started with Bandit: https://developer.rackspace.com/blog/getting-started-with-bandit/\n\nIn situations where Bandit warnings can safely be ignored, the ``# nosec``\ncomment appears on source code lines. Comments as to why these can be\nsafely ignored are included in the code. (Please feel free to issue pull\nrequests if you disagree.)\n\nOne runtime security mechanism employed by ``psec`` is control of the process'\n``umask``. This is important when running programs that create files, which\nwill inherit their permissions per the process ``umask``. The ``umask`` will be\ninherited by every new child process and can be set in the user's ``.bashrc``\n(or other shell initialization) file.\n\nThe ``psec run`` command can be used to run programs as child processes,\noptionally exporting environment variables as well, so controlling the\n``umask`` results in improved file permission security regardless of\nwhether the user knows to set their process ``umask``.\n\nYou can see the effect in these two examples.\n\nFirst, by setting the ``umask`` to ``0`` you see the very permissive file\npermissions (as well as getting a warning from ``psec`` about finding a file\nwith lax permissions):\n\n.. code-block:: console\n\n $ psec --umask 0o000 run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo\n 1+0 records in\n 1+0 records out\n 512 bytes copied, 0.000019 s, 2.7 MB/s\n $ ls -l $(psec environments path --tmpdir)/foo\n [!] file /Users/dittrich/.secrets/python_secrets/tmp/foo is mode 0o100666\n -rw-rw-rw- 1 dittrich staff 512 Sep 8 13:05 /Users/dittrich/.secrets/python_secrets/tmp/foo\n $ rm $(psec environments path --tmpdir)/foo\n\n..\n\nNow when using the default ``--umask`` value, the file permissions are restricted\n(and thus no more warning):\n\n.. code-block:: console\n\n $ psec run -- dd if=/dev/random count=1 of=$(psec environments path --tmpdir)/foo\n 1+0 records in\n 1+0 records out\n 512 bytes copied, 0.000243 s, 2.1 MB/s\n $ ls -l $(psec environments path --tmpdir)/foo\n -rw------- 1 dittrich staff 512 Sep 8 13:04 /Users/dittrich/.secrets/python_secrets/tmp/foo\n $ rm $(psec environments path --tmpdir)/foo\n\n..\n\nBugs, Enhancements, and Future Work\n-----------------------------------\n\nFeature requests (and of course bug reports) are highly encouraged. You can\ndo that by `opening an issue`_ on GitHub. Better yet, make a `pull\nrequest`_ with your own fix or feature. (Check there to see if one\nmay already exist.)\n\nIf you want to help, there are some things that are on the \"to do\"\nlist. These are tracked on this repository's GitHub `Projects`_ page.\n\nGeneral or more elaborate potential enhancements are listed here:\n\n* Increase test coverage (test driven development is a Good Thing(TM))\n\n* The Mantl project (GitHub `mantl/mantl`_) employs a `security-setup`_ script\n that takes care of setting secrets (and non-secret related variables) in a\n monolithic manner. It has specific command line options, specific secret\n generation functions, and specific data structures for each of the component\n subsystems used by `mantl/mantl`_. This method is not modular or extensible, and\n the `security-setup`_ script is not generalized such that it can be used by\n any other project. These limitations are primary motivators for writing\n ``python_secrets``, which could eventually replace ``security-setup``.\n\n At this point, the Mantl ``security.yml`` file can be read in and\n values can be manually set, as seen here:\n\n .. code-block:: console\n\n $ psec -d ~/git/mantl --secrets-file security.yml secrets show -f yaml\n secrets descriptions directory not found\n - Value: admin:password\n Variable: chronos_http_credentials\n - Value: chronos\n Variable: chronos_principal\n - Value: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx\n Variable: chronos_secret\n - Value: 88821cbe-c004-4cff-9f91-2bc36cd347dc\n Variable: consul_acl_agent_token\n - Value: f9acbe14-28d3-4d06-a1c9-c617da5ebb4e\n Variable: consul_acl_mantl_api_token\n - Value: de54ae85-8226-4146-959f-8926b0b8ee55\n Variable: consul_acl_marathon_token\n - Value: dfc9b244-5140-41ad-b93a-ac5c2451fb95\n Variable: consul_acl_master_token\n - Value: e149b50f-cb5c-4efe-be96-26a52efdc715\n Variable: consul_acl_secure_token\n - Value: 719f2328-6446-4647-adf6-310013bac636\n Variable: consul_acl_vault_token\n - Value: Z0niD1jeiTkx7xaoewJm2A==\n Variable: consul_gossip_key\n - Value: true\n Variable: do_chronos_auth\n - Value: true\n Variable: do_chronos_iptables\n - Value: true\n Variable: do_chronos_ssl\n - Value: true\n Variable: do_consul_auth\n - Value: true\n Variable: do_consul_ssl\n - Value: true\n Variable: do_mantl_api_auth\n - Value: true\n Variable: do_mantlui_auth\n - Value: true\n Variable: do_mantlui_ssl\n - Value: true\n Variable: do_marathon_auth\n - Value: true\n Variable: do_marathon_iptables\n - Value: true\n Variable: do_marathon_ssl\n - Value: true\n Variable: do_mesos_auth\n - Value: true\n Variable: do_mesos_follower_auth\n - Value: true\n Variable: do_mesos_framework_auth\n - Value: true\n Variable: do_mesos_iptables\n - Value: true\n Variable: do_mesos_ssl\n - Value: false\n Variable: do_private_docker_registry\n - Value: mantl-api\n Variable: mantl_api_principal\n - Value: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+\n Variable: mantl_api_secret\n - Value: admin:password\n Variable: marathon_http_credentials\n - Value: marathon\n Variable: marathon_principal\n - Value: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX\n Variable: marathon_secret\n - Value:\n - principal: marathon\n secret: +Y5bvIsWliFvcWgbXGWa8kwT6Qf3etogQJe+cK+IV2hX\n - principal: chronos\n secret: S0JMz5z8oxQGQXMyZjwE0ZCmu4zeJV4oWDUrdc25MBLx\n - principal: mantl-api\n secret: Se4R9nRy8WTAgmU9diJyIPwLYsBU+V1yBxTQumiOriK+\n Variable: mesos_credentials\n - Value: follower\n Variable: mesos_follower_principal\n - Value: Q53uAa2mNM0UNe2RUjrX6k7QvK6ojjH1gHXYLcm3Lmfr\n Variable: mesos_follower_secret\n - Value: password\n Variable: nginx_admin_password\n - Value: true\n Variable: security_enabled\n - Value: chronos\n Variable: zk_chronos_user\n - Value: JWPO11z4lU5qeilZ\n Variable: zk_chronos_user_secret\n - Value: hsr+R6YQBAOXoY84a8ne8bU0opg=\n Variable: zk_chronos_user_secret_digest\n - Value: marathon\n Variable: zk_marathon_user\n - Value: UBh77ok2svQAqWox\n Variable: zk_marathon_user_secret\n - Value: mo2mQGXcsc21zB4wYD18jn+Csks=\n Variable: zk_marathon_user_secret_digest\n - Value: mesos\n Variable: zk_mesos_user\n - Value: L3t9FEMsXehqeBvl\n Variable: zk_mesos_user_secret\n - Value: bHYvGteRBxou4jqJ8XWAYmOmzxs=\n Variable: zk_mesos_user_secret_digest\n - Value: super\n Variable: zk_super_user\n - Value: 2DyL/n/GLi3Q0pa75z9OjODGZKC1RCaEiKNV1ZXo1Wpk\n Variable: zk_super_user_secret\n $ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password\n secrets descriptions directory not found\n \"nginx_admin_password\",\"password\"\n $ psec -d ~/git/mantl --secrets-file security.yml secrets set nginx_admin_password=newpassword\n secrets descriptions directory not found\n $ psec -d ~/git/mantl --secrets-file security.yml secrets show -f csv | grep nginx_admin_password\n secrets descriptions directory not found\n \"nginx_admin_password\",\"newpassword\"\n\n ..\n\n There are a few things that can be done to use ``psec`` as a replacement\n for the ``security-setup`` script. These include:\n\n * Produce secrets descriptions in a ``security.d`` directory.\n * Remove the variables that are not secrets requiring regeneration for rotation\n or \"break-glass\" procedures (e.g., like ``chronos_principal``, which is a\n userID value, and ``do_mesos_auth``, which is a boolean flag).\n * Break down more complex data structures (specifically, the ``mesos_credentials``\n list of dictionaries with keys ``principal`` and ``secret``). These could\n instead be discrete variables like ``marathon_secret`` (which appears to\n be the secret associated with the invariant \"variable\" ``marathon_principal``).\n\n .. note::\n\n Alternatively, these kind of variables could be supported by defining a type ``invariant``\n or ``string`` and prompting the user to provide a new value (using any current value\n as the default).\n\n ..\n\nCredits\n-------\n\nTools used in rendering this package:\n\n* Cookiecutter_\n* `cookiecutter-pypackage`_\n\nDevelopment of this program was supported in part under an Open Source\nDevelopment Grant from the Comcast Innovation Fund.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n.. _openstack/cliff: https://github.com/openstack/cliff\n.. _python-update-dotdee: https://pypi.org/project/update-dotdee/\n.. _terraform: https://www.terraform.io/\n.. _Vault by Hashicorp: https://www.vaultproject.io/\n.. _mantl/mantl: https://github.com/mantl/mantl\n.. _security-setup: http://docs.mantl.io/en/latest/security/security_setup.html\n.. _Ansible: https://docs.ansible.com/\n.. _libfuse/sshfs: https://github.com/libfuse/sshfs\n.. _D2 Ansible Playbooks: https://github.com/davedittrich/ansible-dims-playbooks\n.. _Passing variables on the Command Line: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#passing-variables-on-the-command-line\n.. _OAuth 2.0 Mechanism: https://developers.google.com/gmail/imap/xoauth2-protocol.\n.. _davedittrich/goSecure: https://github.com/davedittrich/goSecure\n.. _Surveillance Self-Defense Guide: https://ssd.eff.org/en\n.. _opening an issue: https://github.com/davedittrich/python_secrets/issues\n.. _pull request: https://github.com/davedittrich/python_secrets/pulls\n.. _Projects: https://github.com/davedittrich/python_secrets/projects/1\n.. _How to Use PGP for Linux: https://ssd.eff.org/en/module/how-use-pgp-linux\n.. _Python Security: https://python-security.readthedocs.io/index.html\n.. _JSON Web Tokens (JWT) are Dangerous for User Sessions\u2014Here\u2019s a Solution: https://redis.com/blog/json-web-tokens-jwt-are-dangerous-for-user-sessions/\n.. _Stop Using JSON Web Tokens For Authentication. Use Stateful Sessions Instead: https://betterprogramming.pub/stop-using-json-web-tokens-for-authentication-use-stateful-sessions-instead-c0a803931a5d\n.. _What\u2019s the Secure Way to Store JWT?: https://medium.com/swlh/whats-the-secure-way-to-store-jwt-dd362f5b7914\n\n\n.. _changelog:\n\nHistory\n-------\n\n.. Follow: https://keepachangelog.com/en/1.0.0/\n..\n.. Guiding Principles\n.. ------------------\n.. Changelogs are for humans, not machines.\n.. There should be an entry for every single version.\n.. The same types of changes should be grouped.\n.. Versions and sections should be linkable.\n.. The latest version comes first.\n.. The release date of each version is displayed.\n.. Mention whether you follow Semantic Versioning.\n..\n.. Types of changes\n.. ----------------\n.. Added for new features.\n.. Changed for changes in existing functionality.\n.. Deprecated for soon-to-be removed features.\n.. Removed for now removed features.\n.. Fixed for any bug fixes.\n.. Security in case of vulnerabilities.\n\n23.4.1 (2023-04-19)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Added `secrets find` command.\n- Added support for new variable type `boolean`.\n\nChanged\n^^^^^^^\n\n- Updated GitHub Actions workflows (default to Python 3.9.16).\n- Drop Python 3.7, 3.8, add Python 3.11 (default to 3.10) for `tox`.\n- Fixed downstream dependency and `pip` installation problems.\n- Resolved new `pep8` and `bandit` findings.\n\n22.6.1 (2022-06-21)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Added `--ignore-missing` option to continue when settings variables.\n- Added 'Operational Security' section to README.\n\n22.6.0 (2022-06-10)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add `about` command to expose selected settings for situational awareness.\n- Add `pytest` code coverage reporting.\n- Add BATS runtime tests related to changes.\n\nChanged\n^^^^^^^\n\n- Fix caching bug with non-unique secret generation.\n- Fix bugs with setting/deleting secrets.\n- Improve secrets basedir initialization logic.\n- Expand use of `pathlib.Path`.\n- Improvements to source code, test, and vscode launch configuration quality.\n\n\n22.5.1 (2022-05-25)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Switch to using factory pattern for secrets generation.\n- General code quality and test improvements.\n- Improve `secrets get` command logic and help.\n- Fix `utils yaml-to-json` subcommand and tests.\n- Resolve setuptools warnings.\n- Separate utility functions from `utils` subcommands.\n\nRemoved\n^^^^^^^\n\n- Retire `consul_key` secret type in favor of `token_base64`.\n- Retire insecure secrets types (e.g., use of SHA1).\n\n22.5.0 (2022-05-11)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Test support for Python 3.10.\n- Add better logging controls.\n\nChanged\n^^^^^^^\n\n- Generalize Google OAuth2 email functionality.\n- Improve use and testing of exceptions.\n\n22.1.0 (2022-01-22)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add `init` command and `--init` flag to initialize secrets base directory.\n- Ensure overridden values via flags are exported to process environment\n for subprocesses to use.\n- Add missing tests for features added in a previous release.\n- Add and start using application-specific exception classes.\n\nChanged\n^^^^^^^\n\n- Move functions and variables to `utils` to improve reuse ability.\n- Use `get_` prefix more consistently for getter method/function names.\n- Over-ride cliff formatter class globally in app parser setup.\n- Use `pathlib.Path` for paths for cleaner code.\n- Fix bugs in `environments delete` command.\n- Fix bugs in `--from-options` feature of `secrets get` and `secrets set`.\n- Improvements to source code, test, and vscode launch configuration quality.\n\n21.11.0 (2021-11-22)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add `Help` attribute to descriptions for URL to more information.\n\nChanged\n^^^^^^^\n\n- General code quality, documentation, and testing enhancements\n- Move `tmpdir` path creation to `secrets_environment.SecretsEnvironment()`.\n- Move `umask()` function and variables to `utils`.\n\nRemoved\n^^^^^^^\n\n- Drop Python 3.6 support due to it being EOL.\n\n21.9.1 (2021-09-15)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Added `secrets tree` subcommand.\n\nChanged\n^^^^^^^\n\n- Fixed bugs with `environments path --tmpdir` subcommand and\n `run` subcommand with `--elapsed` option when no environment exists.\n- Changed license file name.\n- Improved documentation.\n\n21.9.0 (2021-09-07)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Increased test coverage to address bugs (below) being fixed.\n\nChanged\n^^^^^^^\n\n- Fixed bugs in `Makefile` and `tox.ini` file.\n- Fixed bug setting undefined variables.\n- Switched from `numpy` to Python `secrets` module for random bytes.\n- Increased key size from 16 to 32 bits for `consul_key`, `token_hex` and `token_urlsafe`.\n\n21.8.0 (2021-08-12)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Fixed bug in setup.py+setup.cfg\n\n21.7.0 (2021-07-30)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Secrets descriptions for demoing HypriotOS Flash mods Medium article\n\nChanged\n^^^^^^^\n\n- Improve `secrets set --from-options`\n- General code quality, documentation, and testing enhancements\n\n21.6.0 (2021-06-23)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Ability to set and generate secrets from defaults options\n- Ability to create an alias for an existing environment\n- Allow retroactive mirroring of new secrets\n\nChanged\n^^^^^^^\n\n- Switched from `pbr` to `setuptools_scm` for version numbering\n- Switched to more secure random number generation\n\n21.2.0 (20201-02-23)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Improve GitHub Actions workflows\n- Overall documentation and code enhancements\n- Improve handling of wildcards in options list\n\nChanged\n^^^^^^^\n\n- Fix bugs with handling empty lists, cloning environments, BATS tests\n- Increase password complexity a bit more\n- Fix ReadTheDocs\n\n20.11.0 (2020-11-17)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add `secrets create` and `secrets delete` commands\n\nChanged\n^^^^^^^\n\n- Normalize all logger and exception output text\n- Refactoring code for better modulatiry\n- Normalize `group create` and `group delete` code\n- Normalize `secrets show` and `secrets describe` code\n- Fix bug that left variables missing after cloning\n- Add Python 3.9 to testing matrix\n- Switch from .yml to .json format for secrets\n- Expand IP address support in `utils` subcommand\n\n20.8.1 (2020-08-11)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Fixes to v20.8.0\n\n20.8.0 (2020-08-11)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add GitHub workflow to publish to test.pypi.org\n- Add `secrets backup` and `secrets restore` logic\n- Open web browser to documentation for help\n\nChanged\n^^^^^^^\n\n- Go back to date-based version numbering\n- General CI/CD workflow updates\n- Improve directory handling in `environments path`\n\n20.2.15 (2012-02-15)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Added Python 3.8 support to test matrix\n\nChanged\n^^^^^^^\n\n- Fix bug in `environments default`\n- Put elapsed time (and BELL) on stdout\n- Fix bug in `environments tree`\n- Allow setting vars using diff names+environment\n\n19.12.0 (2019-12-16)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add and document new `boolean` data type\n- Add `groups delete` command\n\n\nChanged\n^^^^^^^\n\n- Improve default environment handling\n- Improve tox+BATS testing\n- Address security issue per \"Your xkcd passwords are pwned\" article\n- General code quality and test improvements\n- Add protection from over-writing existing env vars\n- Add `Options` attribute\n\n19.11.1 (2019-11-29)\n~~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Enhancements to better support Windows 10\n- Allow cloning group descriptions from environment\n- Fix tty/no-tty handling with `environments delete`\n- Expose terraform command on `-v`\n- Validate variable exists in environment\n- Fix broken `environments tree` code\n\n19.10.1 (2019-10-20)\n~~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Move BATS unit tests into tox testing\n- Avoid attempting interactive things when no tty\n- Improve file and directory permissions logic\n\n19.10.0 (2019-10-14)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Working SSH key and configuration management\n- Use `bullet` for interactive list selection\n- Elapsed timer feature\n- Parsing of terraform output to extract SSH public keys\n- `umask` control for better new file permission settings\n- Support configuring terraform `tfstate` backend\n- Allow setting secrets by copying from another environment\n\nChanged\n^^^^^^^\n\n- Numerous bug fixes\n- Refine testing\n- Option to only show undefined variables\n- Sort environments when listing\n\n19.9.0 (2019-09-05)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add `environments delete` subcommand\n- Allow cloning environment from an existing one\n\nChanged\n^^^^^^^\n\n19.8.3 (2019-08-28)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Dynamically get version number\n- General testing enhancements\n- General code quality enhancements\n- Ensure more secure file permissions\n\n19.8.2 (2019-08-23)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- General code quality enhancements\n\n19.8.0 (2019-08-22)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- IP address determination\n- Allow cloning new group in an empty environment\n- Make `python -m psec` work\n- JSON output method\n- Environment aliasing feature\n\nChanged\n^^^^^^^\n\n- General code quality and testing enhancements\n- Be more explicit about default environment\n- Tighten permissions on cloned environments/groups\n- Add insecure permissions checking\n\n19.5.1 (2019-05-08)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\nAdd `HISTORY.rst` file\n\n19.4.5 (2019-05-08)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add command `ssh config` to manage SSH configuration snippet\n for use by `update-dotdee` to generate ~/.ssh/config file\n- Add command `ssh known-hosts add` and `ssh known-hosts remove`\n to manage system known_hosts file(s)\n\nChanged\n^^^^^^^\n\n- Generalized exception to fix --version bug\n- Clean up temporary docs/psec_help.txt file\n\n19.4.4 (2019-04-21)\n~~~~~~~~~~~~~~~~~~~\n\nChanged\n^^^^^^^\n\n- Fix Bats dependencies/tests\n- Fix broken documentation (wt?)\n- Fix messed up release tagging\n\n19.4.0 (2019-04-19)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Python 3.7 coverage for Travis CI\n\nChanged\n^^^^^^^\n\n- Complete --help output (epilog text) in all commands\n- Install a script 'psec' to complement console_script entry point\n- Clarify arguments in --help output\n\nDeprecated\n^^^^^^^^^^\n\n- The 'python_secrets' command is now just 'psec'\n\n19.3.1 (2019-04-06)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add ``environments rename`` command\n- Add ``utils set-aws-credentials`` command to mirror AWS CLI credentials\n- Use ``autoprogram_cliff`` for self-documentation\n- Add ``cliff.sphinxext`` for documentation\n\nChanged\n^^^^^^^\n\n- Refactored ``SecretsEnvironment()`` so ``autoprogram_cliff`` works\n\n18.11.0 (2018-11-09)\n~~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Add \"--type\" option to \"secrets describe\"\n- Improve visibility into default environment\n- Add screencasts to documenation\n- Add RST checks to ensure PyPi documentation works\n- Add feedback about minimum Python version\n- Add ``--json`` output to ``environments path``\n- Add reference to proof-of-concept using goSecure fork\n\nChanged\n^^^^^^^\n\n- The \"secrets describe\" command now describes variables and types\n- Allow ``secrets set`` to set any type (not just ``string``)\n\n\n18.9.0 (2018-09-27)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Switched to calendar version numbering\n- Finish GPG encrypted email delivery of secrets\n- ``groups create`` command\n- Improve error handling consistency when no environment exists\n\n\n0.16.0 (2018-09-12)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Use attribute maps instead of lookup loops\n- Add Prompt attribute in descriptions for better UX when setting variables\n- Note new undefined variables when adding groups or ``environments create --clone-from``\n- When exporting vars, also export PYTHON_SECRETS_ENVIRONMENT w/environment name\n- Add reference to Python Security coding information\n- ``environments tree`` command\n- ``environments path`` command with features supporting Ansible Lookup Plugin\n- ``secrets get`` command\n- ``groups path`` command\n- ``environments default`` command\n\n0.14.0 (2018-08-30)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- Option to export secrets as environment variables (with optional prefix)\n- Can now set secrets (any specified or all undefined) via command line\n- ``utils myip`` command returns routable IP address (with CIDR option)\n- ``run`` command allows running commands with exported environment variables\n\nChanged\n^^^^^^^\n\n- Renamed ``template`` comamnd to ``utils tfoutput``\n\nRemoved\n^^^^^^^\n\n- Dropped support for Python 3.4, 3.5, since ``secrets`` module only in Python >= 3.6\n\n\n0.10.0 (2018-08-23)\n~~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- New ``string`` type for manually set secrets\n- ``secrets path`` command provides path to secrets ``.yml`` file\n- ``template`` command (Jinja templating)\n- Default environment to basename of cwd\n- Clone environment from skeleton directory in repo\n\n0.9.1 (2018-08-19)\n~~~~~~~~~~~~~~~~~~\n\nAdded\n^^^^^\n\n- ``secrets describe`` command\n- ``environments create`` command\n- ``environments list`` command\n- Expand secrets types and generation methods\n- Add initial feature for sending secrets via email using Google OAuth2 SMTP\n\nRemoved\n^^^^^^^\n\n- Drop Python 2.7 support (at least for now...)\n\nSecurity\n^^^^^^^^\n\n- Add ``six`` for securing ``input`` call\n\n0.8.0 (2018-05-11)\n~~~~~~~~~~~~~~~~~~\n\n(TBD)\n\n0.4.0 (2018-05-01)\n~~~~~~~~~~~~~~~~~~\n\n(TBD)\n\n0.3.6 (2018-04-29)\n~~~~~~~~~~~~~~~~~~\n\n(TBD)\n\n0.3.0 (2018-04-27)\n~~~~~~~~~~~~~~~~~~\n\n* First release on PyPI.\n",
"bugtrack_url": null,
"license": "",
"summary": "",
"version": "23.4.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8a908abe22bec6b857470c9345a7046a45623f0fd29f295b34627a90be8282e9",
"md5": "ee4a4ff7efb8987b4742403f352b87d7",
"sha256": "31ec1e1aa30b63b6b4e3aee0a2d8b8cccebb58df030a3fadedbfb97fb20c25a0"
},
"downloads": -1,
"filename": "python_secrets-23.4.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "ee4a4ff7efb8987b4742403f352b87d7",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.6",
"size": 127160,
"upload_time": "2023-04-19T21:53:20",
"upload_time_iso_8601": "2023-04-19T21:53:20.984017Z",
"url": "https://files.pythonhosted.org/packages/8a/90/8abe22bec6b857470c9345a7046a45623f0fd29f295b34627a90be8282e9/python_secrets-23.4.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bd2ad8d3d657f5b4c028c159f042a6d072274583da3b9abec40cea5fa9148665",
"md5": "aba587c88786996ea19eead6ce1bcf73",
"sha256": "37a2b56a4e181f92b8fefcabb628dbbc7983cd7823702fe2995822b7029264b5"
},
"downloads": -1,
"filename": "python_secrets-23.4.1-py3.9.egg",
"has_sig": false,
"md5_digest": "aba587c88786996ea19eead6ce1bcf73",
"packagetype": "bdist_egg",
"python_version": "23.4.1",
"requires_python": ">=3.6",
"size": 250694,
"upload_time": "2023-04-19T21:53:24",
"upload_time_iso_8601": "2023-04-19T21:53:24.533016Z",
"url": "https://files.pythonhosted.org/packages/bd/2a/d8d3d657f5b4c028c159f042a6d072274583da3b9abec40cea5fa9148665/python_secrets-23.4.1-py3.9.egg",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "30614001a0f35881bdbb2753a14420cf805baa977883da0050ec0cda3abe1a5f",
"md5": "b5dac85ececca9126f92c246ec7782ce",
"sha256": "4c50958c1a4c61eade45cd9d928abcda11092fc603c7c0308b1339883d6d7ace"
},
"downloads": -1,
"filename": "python_secrets-23.4.1.tar.gz",
"has_sig": false,
"md5_digest": "b5dac85ececca9126f92c246ec7782ce",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 176348,
"upload_time": "2023-04-19T21:53:26",
"upload_time_iso_8601": "2023-04-19T21:53:26.259708Z",
"url": "https://files.pythonhosted.org/packages/30/61/4001a0f35881bdbb2753a14420cf805baa977883da0050ec0cda3abe1a5f/python_secrets-23.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-19 21:53:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "davedittrich",
"github_project": "python_secrets",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "python-secrets"
}
JSON
