airflow-sops-secrets-backend


Nameairflow-sops-secrets-backend JSON
Version 0.0.2 PyPI version JSON
download
home_pagehttps://github.com/paulvitic/airflow-sops-secrets-backend
SummaryA airflow_sops secrets backend with sops
upload_time2022-12-05 16:54:02
maintainer
docs_urlNone
authorPaul Vitic
requires_python>=3.8, <4
license
keywords airflow sops secrets-backend
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # Airflow SOPS Secrets Backend for GCP KMS
This packages enables Airflow to pull connections and variables
from files in GCP bucket that are encrypted by [SOPS](https://github.com/mozilla/sops) 
using GCP KMS.

## Configure Airflow
Add following to *airflow.cfg*.
```text
[secrets]
backend = airflow_sops.secrets_backend.GcsSopsSecretsBackend
backend_kwargs = {"project_id": "your-project-id"}
```
Available parameters to backend_kwargs:
* project_id: Optional. GCP project id where the GCS bucket which holds the encrypted connections/variables files reside., 
* bucket_name: Optional. If not submitted tries retrieving from Composer GCS_BUCKET environment variable 
* connections_prefix. Optional. Default is "sops/connections". The folder in GCS bucket that holds encrypted connections.
* variables_prefix: Optional. Default is "sops/variables". The folder in GCS bucket that holds encrypted variables., 
* encrypted_file_ext: Optional. Default is "enc". The file extension for encrypted sops files. The format is <connection_id or variable_key>.<encrypted_file_ext>.yaml
* ignore_mac: Optional. Default is True. Ignores file checksum when true.

## GCP Config
```terraform
locals {
  gcp_project_id = "your-project-id"
  service_account_name = "your-composer-service-account-name"
}

resource "google_service_account" "composer" {
  account_id   = local.service_account_name
  display_name = local.service_account_name
  project      = local.gcp_project_id
}

resource "google_project_iam_member" "composer_worker" {
  project = local.gcp_project_id
  role   = "roles/composer.worker"
  member = "serviceAccount:${google_service_account.composer.email}"
}

resource "google_kms_key_ring" "secrets" {
  name     = local.gcp_project_id
  location = "europe-west1"
  project  = local.gcp_project_id
}

resource "google_kms_crypto_key" "secrets_sops" {
  name            = "secrets_sops"
  key_ring        = google_kms_key_ring.secrets.id
  rotation_period = "7776000s" // 90 days
}

resource "google_kms_crypto_key_iam_member" "composer_sops_decrypter" {
  crypto_key_id = google_kms_crypto_key.secrets_sops.id
  role          = "roles/cloudkms.cryptoKeyDecrypter"
  member        = "serviceAccount:${google_service_account.composer.email}"
}

# some mandatory attributes omitted
resource "google_composer_environment" "composer" {
  name     = "your-composer-environment-name"
  region   = "europe-west1"
  project  = local.gcp_project_id
  config {
    software_config {
      airflow_config_overrides = {
        secrets-backend                          = "airflow_sops.secrets_backend.GcsSopsSecretsBackend"
      }
      pypi_packages = {
        airflow-secrets-sops                   = "==0.0.1"
      }
    }
    node_config {
      service_account = google_service_account.composer.email
    }
  }
}
```

## SOPS
Install [SOPS](https://github.com/mozilla/sops). Encrypt files
using GCP KMS and upload to GCP bucket sops/connections directory
```shell
export KMS_PATH=$(gcloud kms keys list --location europe-west1 --keyring your-keyring --project your-gcp-project | awk 'FNR == 2 {print $1}')
sops --encrypt --encrypted-regex '^(password|extra)$' --gcp-kms $KMS_PATH some-connection.yaml > some-connection.enc.yaml
```

## Setup
```shell
python -m venv .venv
source .venv/bin/activate
pip config set --site global.extra-index-url https://pypi.org/simple
pip install -r requirements.txt
```

## Test
```shell
pip install . airflow-sops-secrets-backend[test]
pytest
```

## Build
```shell
pip install airflow-sops-secrets-backend[dev]
python -m build
```

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/paulvitic/airflow-sops-secrets-backend",
    "name": "airflow-sops-secrets-backend",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8, <4",
    "maintainer_email": "",
    "keywords": "airflow,sops,secrets-backend",
    "author": "Paul Vitic",
    "author_email": "pvitic@comatch.com",
    "download_url": "https://files.pythonhosted.org/packages/04/02/28e02c1c14a77a409a1ca6c67d70309106f1657f9df95ccba3d5cdaf2860/airflow-sops-secrets-backend-0.0.2.tar.gz",
    "platform": null,
    "description": "# Airflow SOPS Secrets Backend for GCP KMS\nThis packages enables Airflow to pull connections and variables\nfrom files in GCP bucket that are encrypted by [SOPS](https://github.com/mozilla/sops) \nusing GCP KMS.\n\n## Configure Airflow\nAdd following to *airflow.cfg*.\n```text\n[secrets]\nbackend = airflow_sops.secrets_backend.GcsSopsSecretsBackend\nbackend_kwargs = {\"project_id\": \"your-project-id\"}\n```\nAvailable parameters to backend_kwargs:\n* project_id: Optional. GCP project id where the GCS bucket which holds the encrypted connections/variables files reside., \n* bucket_name: Optional. If not submitted tries retrieving from Composer GCS_BUCKET environment variable \n* connections_prefix. Optional. Default is \"sops/connections\". The folder in GCS bucket that holds encrypted connections.\n* variables_prefix: Optional. Default is \"sops/variables\". The folder in GCS bucket that holds encrypted variables., \n* encrypted_file_ext: Optional. Default is \"enc\". The file extension for encrypted sops files. The format is <connection_id or variable_key>.<encrypted_file_ext>.yaml\n* ignore_mac: Optional. Default is True. Ignores file checksum when true.\n\n## GCP Config\n```terraform\nlocals {\n  gcp_project_id = \"your-project-id\"\n  service_account_name = \"your-composer-service-account-name\"\n}\n\nresource \"google_service_account\" \"composer\" {\n  account_id   = local.service_account_name\n  display_name = local.service_account_name\n  project      = local.gcp_project_id\n}\n\nresource \"google_project_iam_member\" \"composer_worker\" {\n  project = local.gcp_project_id\n  role   = \"roles/composer.worker\"\n  member = \"serviceAccount:${google_service_account.composer.email}\"\n}\n\nresource \"google_kms_key_ring\" \"secrets\" {\n  name     = local.gcp_project_id\n  location = \"europe-west1\"\n  project  = local.gcp_project_id\n}\n\nresource \"google_kms_crypto_key\" \"secrets_sops\" {\n  name            = \"secrets_sops\"\n  key_ring        = google_kms_key_ring.secrets.id\n  rotation_period = \"7776000s\" // 90 days\n}\n\nresource \"google_kms_crypto_key_iam_member\" \"composer_sops_decrypter\" {\n  crypto_key_id = google_kms_crypto_key.secrets_sops.id\n  role          = \"roles/cloudkms.cryptoKeyDecrypter\"\n  member        = \"serviceAccount:${google_service_account.composer.email}\"\n}\n\n# some mandatory attributes omitted\nresource \"google_composer_environment\" \"composer\" {\n  name     = \"your-composer-environment-name\"\n  region   = \"europe-west1\"\n  project  = local.gcp_project_id\n  config {\n    software_config {\n      airflow_config_overrides = {\n        secrets-backend                          = \"airflow_sops.secrets_backend.GcsSopsSecretsBackend\"\n      }\n      pypi_packages = {\n        airflow-secrets-sops                   = \"==0.0.1\"\n      }\n    }\n    node_config {\n      service_account = google_service_account.composer.email\n    }\n  }\n}\n```\n\n## SOPS\nInstall [SOPS](https://github.com/mozilla/sops). Encrypt files\nusing GCP KMS and upload to GCP bucket sops/connections directory\n```shell\nexport KMS_PATH=$(gcloud kms keys list --location europe-west1 --keyring your-keyring --project your-gcp-project | awk 'FNR == 2 {print $1}')\nsops --encrypt --encrypted-regex '^(password|extra)$' --gcp-kms $KMS_PATH some-connection.yaml > some-connection.enc.yaml\n```\n\n## Setup\n```shell\npython -m venv .venv\nsource .venv/bin/activate\npip config set --site global.extra-index-url https://pypi.org/simple\npip install -r requirements.txt\n```\n\n## Test\n```shell\npip install . airflow-sops-secrets-backend[test]\npytest\n```\n\n## Build\n```shell\npip install airflow-sops-secrets-backend[dev]\npython -m build\n```\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "A airflow_sops secrets backend with sops",
    "version": "0.0.2",
    "split_keywords": [
        "airflow",
        "sops",
        "secrets-backend"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "f433369beffa784fcbef6a6cbc8f775e",
                "sha256": "212f2db87acb432261bdf0b97462e0a8817968918135ae6e577c6346f362b040"
            },
            "downloads": -1,
            "filename": "airflow_sops_secrets_backend-0.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "f433369beffa784fcbef6a6cbc8f775e",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8, <4",
            "size": 9431,
            "upload_time": "2022-12-05T16:53:59",
            "upload_time_iso_8601": "2022-12-05T16:53:59.925014Z",
            "url": "https://files.pythonhosted.org/packages/14/5e/a6873ec784325e4eeaeaca7aad61991212030a804f1223a8344ded019af0/airflow_sops_secrets_backend-0.0.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "3e118849c4cda10845186c8a7c481be0",
                "sha256": "7a2c3e5db00b9447434929de4e44751832de47c2c24b4ecaf8c3fc7fd5622b39"
            },
            "downloads": -1,
            "filename": "airflow-sops-secrets-backend-0.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "3e118849c4cda10845186c8a7c481be0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8, <4",
            "size": 12558,
            "upload_time": "2022-12-05T16:54:02",
            "upload_time_iso_8601": "2022-12-05T16:54:02.421145Z",
            "url": "https://files.pythonhosted.org/packages/04/02/28e02c1c14a77a409a1ca6c67d70309106f1657f9df95ccba3d5cdaf2860/airflow-sops-secrets-backend-0.0.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-12-05 16:54:02",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "paulvitic",
    "github_project": "airflow-sops-secrets-backend",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [],
    "lcname": "airflow-sops-secrets-backend"
}
Paul Vitic
Elapsed time: 0.06759s