# Secret Transfer
YaML-based secret manager for secret load/transfer from different sources to destinations.
## Base idea
The base idea is to have a simple and flexible way to transfer secrets from different sources to different destinations.
The main components are:
- **Source** - A place where secrets are stored. It can be a file, environment variables, a database, a cloud service, etc.
- **Destination** - A place where secrets are transferred. It can be environment variables, a file, a cloud service, etc.
- **Collection** - A collection of secrets from sources. It can be used to combine secrets from different sources.
- **Transfer** - A transfer of secrets from a collection to a destination.
## Features
- **YaML-based** - Define sources, destinations, collections, and transfers in a YaML file.
- **Lazy loading** - Define objects in no particular order and reference them from one another.
- **Cross-referencing** - Reference sources, destinations, collections, and transfers from one another.
- **Built-in classes** - Use built-in classes for sources, destinations, collections, and transfers.
- **Extensible** - Add custom classes for sources, destinations, collections, and transfers.
## Usage
### CLI Commands
#### Run
Run all transfers in the given YaML file.
```bash
secret-transfer run -f <path_to_yaml_file>
```
Options:
- `-f, --file <path_to_yaml_file>` - Path to the YaML file with secrets. [required]
- `-n, --name <name>` - Name of the transfer to run. [optional]
#### Clean
Clean all secrets in all transfer destinations in the given YaML file.
```bash
secret-transfer clean -f <path_to_yaml_file>
```
Options:
- `-f, --file <path_to_yaml_file>` - Path to the YaML file with secrets. [required]
- `-n, --name <name>` - Name of the transfer to clean. [optional]
### YaML schema
The YaML file should contain the following optional sections:
Custom class definitions for sources, destinations, collections, and transfers.
```yaml
source_classes:
<name>: # Name to be registered
module: <module> # Module to import from
class_name: <class_name> # Class to be registered
destination_classes: ...
collection_classes: ...
transfer_classes: ...
```
Sources, destinations, collections, and transfers.
```yaml
sources:
<name>:
class_name: <source_class_name> # Class to use, either from source_classes or built-in
init_args: # Arguments to pass to the class constructor, can be omitted if no arguments are needed
<arg_name>: <arg_value>
...
destinations:
...
collections:
...
transfers:
...
```
### Lasy loading
All items are lazily loaded, so they are not created until they are used. This allows to define objects in no particular order and reference them from one another. The only exception is built-in classes, which are registered at import-time.
Cross-referencing is allowed, so you can reference sources, destinations, collections, transfers and their values from init_args. Circular references are not allowed.
### Cross-referencing
#### Instance references
To reference instances of sources, destinations, collections, and transfers, use the following syntax:
```yaml
$<type>[<name>]
```
Where:
- `<type>` - Type of the instance (source, destination, collection, or transfer).
- `<name>` - Name of the instance.
#### Value references
To reference values from sources or collections, use the following syntax:
```yaml
$<type>[<name>][<key>]
```
Where:
- `<type>` - Type of the gettable instance (usually, but not necessarily source or collection).
- `<name>` - Name of the instance.
- `<key>` - Key of the value.
Where:
### Built-in classes
#### Sources
##### DotEnvSource
Load secrets from a .env file using `dotenv` package.
Arguments:
- `file_path` - Path to the .env file. [required]
Example:
```yaml
sources:
dotenv:
class_name: DotEnvSource
init_args:
file_path: .env
```
##### EnvSource
Load secrets from environment variables.
Arguments: none.
Already registered with name `env`, no need to define in the YaML file.
Example:
```yaml
collections:
default:
init_args:
TEST_KEY:
source: $sources[env]
```
##### PresetSource
Load secrets from a preset dictionary.
Arguments: key-value pairs of variables.
Example:
```yaml
sources:
preset:
class_name: PresetSource
init_args:
TEST_KEY: test_value
```
##### UserInputSource
Asks the user for input.
Arguments: none.
Already registered with name `user_input`, no need to define in the YaML file.
Example:
```yaml
collections:
default:
init_args:
TEST_KEY:
source: $sources[user_input]
```
##### VaultCLIKVSource
Load secrets from HashiCorp Vault KV using `vault kv get` CLI command.
Arguments:
- `address` - Address of the Vault server. [required]
- `mount` - Mount point of the KV engine. [required]
- `secret_name` - Name(path) of the secret. [required]
Pre-requisites:
- `vault` CLI installed and authenticated.
Example:
```yaml
sources:
vault:
class_name: VaultCLIKVSource
init_args:
address: https://vault.example.com
mount: secrets
secret_name: TEST_SECRET
```
##### YCCLILockboxSource
Load secrets from Yandex.Cloud Lockbox using `yc lockbox payload get` CLI command.
Arguments:
- `profile` - Name of the Yandex.Cloud CLI profile. [required]
- `folder` - Folder name. [required]
- `lockbox` - Lockbox name. [required]
Pre-requisites:
- `yc` CLI installed and authenticated.
Example:
```yaml
sources:
yc_lockbox:
class_name: YCCLILockboxSource
init_args:
profile: my-profile
folder: my-folder
lockbox: my-lockbox
```
#### Destinations
##### BashExportDestination
Print secrets as `export` commands to the console. Useful for setting environment variables. Never let stdout to be captured by a process, as it will expose the secrets.
Arguments: none.
Already registered with name `bash_export`, no need to define in the YaML file.
Example:
```yaml
transfers:
default:
init_args:
source: ...
destination: $destinations[bash_export]
```
##### EnvDestination
Set secrets as environment variables.
Arguments: none.
Already registered with name `env`, no need to define in the YaML file.
Example:
```yaml
transfers:
default:
init_args:
source: ...
destination: $destinations[env]
```
##### GithubCliSecretsDestination
Set secrets as GitHub repository secrets using `gh secret set` CLI command.
Arguments:
- `repo_name` - Name of the repository. [required]
- `owner_name` - Name of the repository owner. [required]
- `base_url` - Base URL of the GitHub API. [optional] (default: https://github.com)
Pre-requisites:
- `gh` CLI installed and authenticated.
Example:
```yaml
destinations:
github:
class_name: GithubCliSecretsDestination
init_args:
repo_name: my-repo
owner_name: my-org
```
#### Collections
##### DefaultCollection
Default collection to combine secrets from sources. Default collection class, so class_name can be omitted.
Example:
```yaml
collections:
default:
init_args:
COLLECTION_KEY:
source: $sources[env]
key: SOURCE_KEY
```
#### Transfers
##### DefaultTransfer
Default transfer to transfer secrets from collection to destination. Default transfer class, so class_name can be omitted.
Example:
```yaml
transfers:
default:
init_args:
collection: $collections[default]
destination: $destinations[env]
```
### Custom class definitions
You can define custom classes for sources, destinations, collections, and transfers.
#### Any resource
```python
import typing
import typing_extensions
import secret_transfer.core as secret_transfer_core
class CustomResource(...):
__register__ = ... # bool: Should the class be registered on creation (optional, default: True)
__default__ = ... # bool: Should the class be used as default for its type (optional, default: False)
@classmethod
def parse_init_arguments(
cls,
**arguments: secret_transfer_core.InitArgumentType
) -> typing.Mapping[str, typing.Any]:
# Optional to implement
# Parse and validate init arguments passed from the YaML definition for the future instance creation
...
@classmethod
def get_default_instances(cls) -> typing.Mapping[str, typing_extensions.Self]:
# Optional to implement
# Return default instances of the class to be registered at import-time
...
```
#### Source
```python
import secret_transfer.core as secret_transfer_core
import secret_transfer.utils.types as secret_transfer_types
class CustomSource(secret_transfer_core.AbstractSource):
def __getitem__(self, key: str) -> secret_transfer_types.Literal:
# Required to implement
# Return the value of the secret by key
...
```
#### Destination
```python
import secret_transfer.core as secret_transfer_core
import secret_transfer.utils.types as secret_transfer_types
class CustomDestination(secret_transfer_core.AbstractDestination):
def __setitem__(self, key: str, value: secret_transfer_types.Literal) -> None:
# Required to implement
# Set the value of the secret by key
...
def __delitem__(self, key: str) -> None:
# Optional to implement
# Clean all secrets in the destination
...
```
#### Collection
```python
import typing
import secret_transfer.core as secret_transfer_core
import secret_transfer.utils.types as secret_transfer_types
class CustomCollection(secret_transfer_core.AbstractCollection):
def __getitem__(self, key: str) -> secret_transfer_types.Literal:
# Required to implement
# Return the value of the secret by key
...
def __iter__(self) -> typing.Iterator[str]:
# Required to implement
# Return an iterator over the keys of the collection
...
def items(self) -> typing.Iterator[tuple[str, secret_transfer_types.Literal]]:
# Required to implement
# Return an iterator over the items of the collection
...
```
#### Transfer
```python
import secret_transfer.core as secret_transfer_core
class CustomTransfer(secret_transfer_core.AbstractTransfer):
def run(self) -> None:
# Required to implement
# Transfer secrets
...
def clean(self) -> None:
# Required to implement
# Clean all secrets in the transfer
...
```
### Usage Examples
Check [examples](examples/README.md) for usage examples.
## Development
### Taskfile commands
For all commands see [Taskfile](Taskfile.yaml) or `task --list-all`.
Raw data
{
"_id": null,
"home_page": "https://github.com/ovsds/secret-transfer",
"name": "secret-transfer",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<3.13",
"maintainer_email": "",
"keywords": "secret,transfer,yaml",
"author": "ovsds",
"author_email": "github@ovsds.ru",
"download_url": "https://files.pythonhosted.org/packages/9f/07/ea0b939af2c0b09ac0a0052ffb6e6cf8ad41ddf6a0a6820864be6d78dada/secret_transfer-0.3.0.tar.gz",
"platform": null,
"description": "# Secret Transfer\n\nYaML-based secret manager for secret load/transfer from different sources to destinations.\n\n## Base idea\n\nThe base idea is to have a simple and flexible way to transfer secrets from different sources to different destinations.\n\nThe main components are:\n\n- **Source** - A place where secrets are stored. It can be a file, environment variables, a database, a cloud service, etc.\n- **Destination** - A place where secrets are transferred. It can be environment variables, a file, a cloud service, etc.\n- **Collection** - A collection of secrets from sources. It can be used to combine secrets from different sources.\n- **Transfer** - A transfer of secrets from a collection to a destination.\n\n## Features\n\n- **YaML-based** - Define sources, destinations, collections, and transfers in a YaML file.\n- **Lazy loading** - Define objects in no particular order and reference them from one another.\n- **Cross-referencing** - Reference sources, destinations, collections, and transfers from one another.\n- **Built-in classes** - Use built-in classes for sources, destinations, collections, and transfers.\n- **Extensible** - Add custom classes for sources, destinations, collections, and transfers.\n\n## Usage\n\n### CLI Commands\n\n#### Run\n\nRun all transfers in the given YaML file.\n\n```bash\nsecret-transfer run -f <path_to_yaml_file>\n```\n\nOptions:\n\n- `-f, --file <path_to_yaml_file>` - Path to the YaML file with secrets. [required]\n- `-n, --name <name>` - Name of the transfer to run. [optional]\n\n#### Clean\n\nClean all secrets in all transfer destinations in the given YaML file.\n\n```bash\nsecret-transfer clean -f <path_to_yaml_file>\n```\n\nOptions:\n\n- `-f, --file <path_to_yaml_file>` - Path to the YaML file with secrets. [required]\n- `-n, --name <name>` - Name of the transfer to clean. [optional]\n\n### YaML schema\n\nThe YaML file should contain the following optional sections:\n\nCustom class definitions for sources, destinations, collections, and transfers.\n\n```yaml\nsource_classes:\n <name>: # Name to be registered\n module: <module> # Module to import from\n class_name: <class_name> # Class to be registered\ndestination_classes: ...\ncollection_classes: ...\ntransfer_classes: ...\n```\n\nSources, destinations, collections, and transfers.\n\n```yaml\nsources:\n <name>:\n class_name: <source_class_name> # Class to use, either from source_classes or built-in\n init_args: # Arguments to pass to the class constructor, can be omitted if no arguments are needed\n <arg_name>: <arg_value>\n ...\ndestinations:\n ...\ncollections:\n ...\ntransfers:\n ...\n```\n\n### Lasy loading\n\nAll items are lazily loaded, so they are not created until they are used. This allows to define objects in no particular order and reference them from one another. The only exception is built-in classes, which are registered at import-time.\n\nCross-referencing is allowed, so you can reference sources, destinations, collections, transfers and their values from init_args. Circular references are not allowed.\n\n### Cross-referencing\n\n#### Instance references\n\nTo reference instances of sources, destinations, collections, and transfers, use the following syntax:\n\n```yaml\n$<type>[<name>]\n```\n\nWhere:\n\n- `<type>` - Type of the instance (source, destination, collection, or transfer).\n- `<name>` - Name of the instance.\n\n#### Value references\n\nTo reference values from sources or collections, use the following syntax:\n\n```yaml\n$<type>[<name>][<key>]\n```\n\nWhere:\n\n- `<type>` - Type of the gettable instance (usually, but not necessarily source or collection).\n- `<name>` - Name of the instance.\n- `<key>` - Key of the value.\n\nWhere:\n\n### Built-in classes\n\n#### Sources\n\n##### DotEnvSource\n\nLoad secrets from a .env file using `dotenv` package.\nArguments:\n\n- `file_path` - Path to the .env file. [required]\n\nExample:\n\n```yaml\nsources:\n dotenv:\n class_name: DotEnvSource\n init_args:\n file_path: .env\n```\n\n##### EnvSource\n\nLoad secrets from environment variables.\nArguments: none.\n\nAlready registered with name `env`, no need to define in the YaML file.\n\nExample:\n\n```yaml\ncollections:\n default:\n init_args:\n TEST_KEY:\n source: $sources[env]\n```\n\n##### PresetSource\n\nLoad secrets from a preset dictionary.\nArguments: key-value pairs of variables.\n\nExample:\n\n```yaml\nsources:\n preset:\n class_name: PresetSource\n init_args:\n TEST_KEY: test_value\n```\n\n##### UserInputSource\n\nAsks the user for input.\nArguments: none.\n\nAlready registered with name `user_input`, no need to define in the YaML file.\n\nExample:\n\n```yaml\ncollections:\n default:\n init_args:\n TEST_KEY:\n source: $sources[user_input]\n```\n\n##### VaultCLIKVSource\n\nLoad secrets from HashiCorp Vault KV using `vault kv get` CLI command.\nArguments:\n\n- `address` - Address of the Vault server. [required]\n- `mount` - Mount point of the KV engine. [required]\n- `secret_name` - Name(path) of the secret. [required]\n\nPre-requisites:\n\n- `vault` CLI installed and authenticated.\n\nExample:\n\n```yaml\nsources:\n vault:\n class_name: VaultCLIKVSource\n init_args:\n address: https://vault.example.com\n mount: secrets\n secret_name: TEST_SECRET\n```\n\n##### YCCLILockboxSource\n\nLoad secrets from Yandex.Cloud Lockbox using `yc lockbox payload get` CLI command.\nArguments:\n\n- `profile` - Name of the Yandex.Cloud CLI profile. [required]\n- `folder` - Folder name. [required]\n- `lockbox` - Lockbox name. [required]\n\nPre-requisites:\n\n- `yc` CLI installed and authenticated.\n\nExample:\n\n```yaml\nsources:\n yc_lockbox:\n class_name: YCCLILockboxSource\n init_args:\n profile: my-profile\n folder: my-folder\n lockbox: my-lockbox\n```\n\n#### Destinations\n\n##### BashExportDestination\n\nPrint secrets as `export` commands to the console. Useful for setting environment variables. Never let stdout to be captured by a process, as it will expose the secrets.\nArguments: none.\n\nAlready registered with name `bash_export`, no need to define in the YaML file.\n\nExample:\n\n```yaml\ntransfers:\n default:\n init_args:\n source: ...\n destination: $destinations[bash_export]\n```\n\n##### EnvDestination\n\nSet secrets as environment variables.\nArguments: none.\n\nAlready registered with name `env`, no need to define in the YaML file.\n\nExample:\n\n```yaml\ntransfers:\n default:\n init_args:\n source: ...\n destination: $destinations[env]\n```\n\n##### GithubCliSecretsDestination\n\nSet secrets as GitHub repository secrets using `gh secret set` CLI command.\nArguments:\n\n- `repo_name` - Name of the repository. [required]\n- `owner_name` - Name of the repository owner. [required]\n- `base_url` - Base URL of the GitHub API. [optional] (default: https://github.com)\n\nPre-requisites:\n\n- `gh` CLI installed and authenticated.\n\nExample:\n\n```yaml\ndestinations:\n github:\n class_name: GithubCliSecretsDestination\n init_args:\n repo_name: my-repo\n owner_name: my-org\n```\n\n#### Collections\n\n##### DefaultCollection\n\nDefault collection to combine secrets from sources. Default collection class, so class_name can be omitted.\n\nExample:\n\n```yaml\ncollections:\n default:\n init_args:\n COLLECTION_KEY:\n source: $sources[env]\n key: SOURCE_KEY\n```\n\n#### Transfers\n\n##### DefaultTransfer\n\nDefault transfer to transfer secrets from collection to destination. Default transfer class, so class_name can be omitted.\n\nExample:\n\n```yaml\ntransfers:\n default:\n init_args:\n collection: $collections[default]\n destination: $destinations[env]\n```\n\n### Custom class definitions\n\nYou can define custom classes for sources, destinations, collections, and transfers.\n\n#### Any resource\n\n```python\nimport typing\nimport typing_extensions\n\nimport secret_transfer.core as secret_transfer_core\n\n\nclass CustomResource(...):\n __register__ = ... # bool: Should the class be registered on creation (optional, default: True)\n __default__ = ... # bool: Should the class be used as default for its type (optional, default: False)\n\n @classmethod\n def parse_init_arguments(\n cls,\n **arguments: secret_transfer_core.InitArgumentType\n ) -> typing.Mapping[str, typing.Any]:\n # Optional to implement\n # Parse and validate init arguments passed from the YaML definition for the future instance creation\n ...\n\n @classmethod\n def get_default_instances(cls) -> typing.Mapping[str, typing_extensions.Self]:\n # Optional to implement\n # Return default instances of the class to be registered at import-time\n ...\n```\n\n#### Source\n\n```python\nimport secret_transfer.core as secret_transfer_core\nimport secret_transfer.utils.types as secret_transfer_types\n\n\nclass CustomSource(secret_transfer_core.AbstractSource):\n def __getitem__(self, key: str) -> secret_transfer_types.Literal:\n # Required to implement\n # Return the value of the secret by key\n ...\n```\n\n#### Destination\n\n```python\nimport secret_transfer.core as secret_transfer_core\nimport secret_transfer.utils.types as secret_transfer_types\n\n\nclass CustomDestination(secret_transfer_core.AbstractDestination):\n def __setitem__(self, key: str, value: secret_transfer_types.Literal) -> None:\n # Required to implement\n # Set the value of the secret by key\n ...\n\n def __delitem__(self, key: str) -> None:\n # Optional to implement\n # Clean all secrets in the destination\n ...\n```\n\n#### Collection\n\n```python\nimport typing\n\nimport secret_transfer.core as secret_transfer_core\nimport secret_transfer.utils.types as secret_transfer_types\n\n\nclass CustomCollection(secret_transfer_core.AbstractCollection):\n def __getitem__(self, key: str) -> secret_transfer_types.Literal:\n # Required to implement\n # Return the value of the secret by key\n ...\n\n def __iter__(self) -> typing.Iterator[str]:\n # Required to implement\n # Return an iterator over the keys of the collection\n ...\n\n def items(self) -> typing.Iterator[tuple[str, secret_transfer_types.Literal]]:\n # Required to implement\n # Return an iterator over the items of the collection\n ...\n```\n\n#### Transfer\n\n```python\nimport secret_transfer.core as secret_transfer_core\n\n\nclass CustomTransfer(secret_transfer_core.AbstractTransfer):\n def run(self) -> None:\n # Required to implement\n # Transfer secrets\n ...\n\n def clean(self) -> None:\n # Required to implement\n # Clean all secrets in the transfer\n ...\n```\n\n### Usage Examples\n\nCheck [examples](examples/README.md) for usage examples.\n\n## Development\n\n### Taskfile commands\n\nFor all commands see [Taskfile](Taskfile.yaml) or `task --list-all`.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Yaml-based secret manager for secret load/set/clean.",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/ovsds/secret-transfer",
"Repository": "https://github.com/ovsds/secret-transfer"
},
"split_keywords": [
"secret",
"transfer",
"yaml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1be0544f86fd6cae550dc6bb1689d780682b1e34e66ff6930811892b7e37e05c",
"md5": "bde5b9e1c2238c423f0b6f0715e90fab",
"sha256": "e94cb1b7d7fe28e265685580afca6347a31d359077f396834f8c50c9830c68c5"
},
"downloads": -1,
"filename": "secret_transfer-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bde5b9e1c2238c423f0b6f0715e90fab",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<3.13",
"size": 33800,
"upload_time": "2024-02-21T20:27:07",
"upload_time_iso_8601": "2024-02-21T20:27:07.176730Z",
"url": "https://files.pythonhosted.org/packages/1b/e0/544f86fd6cae550dc6bb1689d780682b1e34e66ff6930811892b7e37e05c/secret_transfer-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9f07ea0b939af2c0b09ac0a0052ffb6e6cf8ad41ddf6a0a6820864be6d78dada",
"md5": "9b1b651d1910ec99dcf943007ddbf3c2",
"sha256": "eb446b02528cfd3a297ee82a71f5fbf7c77b2e907cc480e9cbcc9eb388723aa9"
},
"downloads": -1,
"filename": "secret_transfer-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "9b1b651d1910ec99dcf943007ddbf3c2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<3.13",
"size": 19271,
"upload_time": "2024-02-21T20:27:08",
"upload_time_iso_8601": "2024-02-21T20:27:08.238936Z",
"url": "https://files.pythonhosted.org/packages/9f/07/ea0b939af2c0b09ac0a0052ffb6e6cf8ad41ddf6a0a6820864be6d78dada/secret_transfer-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-21 20:27:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ovsds",
"github_project": "secret-transfer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "secret-transfer"
}
JSON
