<div align="center">
# 🚀 ampy-config
**Typed Configuration & Secrets Façade for AmpyFin**
[](https://www.python.org/downloads/)
[](LICENSE)
[](https://github.com/AmpyFin/ampy-config/actions)
[](https://pypi.org/project/ampy-config/)
[](https://github.com/psf/black)
> 🎯 **Single, safe source of truth** for configuration and secrets across AmpyFin services
> 🔗 Built to integrate with **ampy-bus** (control plane over NATS/JetStream) and **ampy-proto** (payload contracts)
[📖 Documentation](#-highlights-what-you-get) • [🚀 Quick Start](#-install-python--pypi) • [🔧 Usage](#-cli-usage) • [🤝 Contributing](#-contributing)
</div>
---
## 📋 Table of Contents
- [🎯 Why this exists](#-why-this-exists-the-problem)
- [✨ Highlights](#-highlights-what-you-get)
- [🚀 Install](#-install-python--pypi)
- [🎮 Control Plane](#-control-plane-natsjetstream)
- [📚 Layering Model](#-layering-model)
- [🔐 Secrets](#-secrets-indirection-caching-rotation-redaction)
- [💻 CLI Usage](#-cli-usage)
- [🐍 Python Integration](#-use-from-a-service-python-example)
- [📊 Schema Examples](#-schema-notes-metrics-example)
- [🌍 Environment Variables](#-environment-variables)
- [🔧 Troubleshooting](#-troubleshooting)
- [🛡️ Security](#-security-notes)
- [🤝 Contributing](#-contributing)
---
## 🎯 Why this exists (the problem)
Without a unified configuration layer, distributed trading systems tend to develop:
> ⚠️ **Common Issues:**
> - **ENV/YAML sprawl** → drift, surprises, outages
> - **Secret handling risks** → credentials in logs, brittle rotations, no redaction
> - **Non-reproducibility** → can't reconstruct exactly which parameters were live for a given trade/run
> - **Inconsistent runtime behavior** → some services reload, others require restarts
**ampy-config** provides a single, typed, validated, observable configuration view with clean secret indirection and a runtime control plane for safe updates.
---
## ✨ Highlights (what you get)
| Feature | Description |
|---------|-------------|
| 🔍 **Typed schema + validation** | JSON Schema + semantic cross-field checks |
| 📚 **Layering & precedence** | defaults → environment profile → overlays → ENV allowlist → runtime overrides |
| 🔐 **Secret indirection** | `secret://…`, `aws-sm://…`, `gcp-sm://…` with caching, rotation, and universal redaction |
| 🎮 **Control plane for updates** | `config_preview` → `config_apply` → `config_applied` events on NATS (JetStream) |
| 📊 **Auditability & observability** | provenance for each key; logs/metrics/traces (no secrets) |
| 🌐 **Language-agnostic** | produces plain YAML effective config for Python, Go, C++, etc. |
---
## 🚀 Install
### 🐍 Python / PyPI
```bash
pip install ampy-config
```
**Developer mode** (local repo):
```bash
pip install -e .
```
### 🐹 Go Client
**Library:**
```bash
go get github.com/AmpyFin/ampy-config/go/ampyconfig@v0.1.0
```
**Binaries:**
```bash
cd go/ampyconfig
make # builds bin/ampyconfig-{ops,agent,listener}
```
> 📦 **Available on [pkg.go.dev](https://pkg.go.dev/github.com/AmpyFin/ampy-config/go/ampyconfig)**
### 🔧 Optional secret backends
| Backend | Install Command | Use Case |
|---------|----------------|----------|
| 🔐 **HashiCorp Vault** | `pip install hvac` | Enterprise secret management |
| ☁️ **AWS Secrets Manager** | `pip install boto3` | AWS-native secret storage |
| 🌐 **GCP Secret Manager** | `pip install google-cloud-secret-manager` | Google Cloud secret storage |
> 💡 **Tip:** You **do not** need to sign up for all of these. Choose one or more real backends for your deployment; the library gracefully falls back to a local JSON file in development.
---
## 🎮 Control plane (NATS/JetStream)
**Start a local NATS with JetStream:**
```bash
docker run --rm -d --name nats -p 4222:4222 nats:2.10 -js
export NATS_URL="nats://127.0.0.1:4222"
```
**Provision the stream and durable consumers** (once). Using the `nats` CLI:
```bash
# Stream to cover all control-plane subjects
nats --server "$NATS_URL" stream add ampy-control \
--subjects "ampy.*.control.v1.*" \
--retention limits --max-age 24h --storage file \
--max-msgs 10000 --max-bytes 100MB --discard old --defaults
# Agent durables (pull + explicit ack)
nats --server "$NATS_URL" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-preview \
--filter "ampy.dev.control.v1.config_preview" --pull --deliver all --ack explicit --defaults
nats --server "$NATS_URL" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-apply \
--filter "ampy.dev.control.v1.config_apply" --pull --deliver all --ack explicit --defaults
nats --server "$NATS_URL" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-secret-rotated \
--filter "ampy.dev.control.v1.secret_rotated" --pull --deliver all --ack explicit --defaults
```
**Verify setup:**
```bash
nats --server "$NATS_URL" stream ls
nats --server "$NATS_URL" consumer ls ampy-control
```
> ⚡ **Note:** The library can also auto-provision if permitted, but explicit creation is more predictable for local dev and CI.
---
## 📚 Layering model
Effective config = **merge** in this order (later overrides earlier):
```mermaid
graph TD
A[📄 Defaults<br/>config/defaults.yaml] --> B[🌍 Environment Profile<br/>examples/dev.yaml]
B --> C[📋 Overlays<br/>--overlay path]
C --> D[🔧 ENV Allowlist<br/>env_allowlist.txt]
D --> E[⚡ Runtime Overrides<br/>runtime/overrides.yaml]
E --> F[✅ Final Config]
```
| Layer | Description | Example |
|-------|-------------|---------|
| 1️⃣ **Defaults** | Checked-in base config | `config/defaults.yaml` |
| 2️⃣ **Environment profile** | Environment-specific settings | `examples/dev.yaml`, `examples/paper.yaml`, `examples/prod.yaml` |
| 3️⃣ **Overlays** | Region/cluster/service YAMLs | `--overlay path` (repeatable) |
| 4️⃣ **ENV allowlist** | Environment variable mapping | `env_allowlist.txt` maps allowed env keys |
| 5️⃣ **Runtime overrides** | Live configuration updates | `runtime/overrides.yaml` (written by agent) |
Each key tracks **provenance**: where it came from (defaults/profile/overlay/ENV/runtime).
### 📏 Units & types
| Type | Format | Examples |
|------|--------|----------|
| ⏱️ **Durations** | String format | `150ms`, `2s`, `5m`, `1h` |
| 📊 **Sizes** | String format | `128KiB`, `1MiB` |
| 🏷️ **Domains** | Explicit prefixes | `oms.*`, `ingest.*`, `broker.*`, `ml.*`, `warehouse.*`, `fx.*`, `metrics`, `logging`, `tracing`, `security.*`, `feature_flags.*` |
---
## 🔐 Secrets (indirection, caching, rotation, redaction)
Use **references**, not literal values:
| Backend | Format | Example |
|---------|--------|---------|
| 🔐 **Vault** | `secret://vault/<path>#<key>` | `secret://vault/tiingo#token` |
| ☁️ **AWS SM** | `aws-sm://<name>?versionStage=AWSCURRENT` | `aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT` |
| 🌐 **GCP SM** | `gcp-sm://projects/<project>/secrets/<name>/versions/latest` | `gcp-sm://projects/demo/secrets/AMPY_API/versions/latest` |
**Local development fallback file** (`.secrets.local.json`):
```json
{
"secret://vault/tiingo#token": "TIINGO_LOCAL_DEV_TOKEN",
"aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT": "ALPACA_LOCAL_DEV_SECRET",
"gcp-sm://projects/demo/secrets/AMPY_API/versions/latest": "AMPY_LOCAL_DEV_API"
}
```
> 🔒 **Security:** Secrets are **always redacted** in logs/metrics/traces; rotation is signaled via `secret_rotated` events.
---
## 💻 CLI usage
All commands are available via `python -m ampy_config.cli …` (works without global entrypoints).
### 🎨 Render effective config
```bash
python -m ampy_config.cli render \
--profile dev \
--resolve-secrets redacted \
--provenance
```
**Write it to a file:**
```bash
python -m ampy_config.cli render \
--profile dev \
--resolve-secrets redacted \
--output /tmp/effective.yaml
```
**Resolve values** (dev only; requires `.secrets.local.json` or configured backends):
```bash
AMPY_CONFIG_LOCAL_SECRETS=.secrets.local.json \
python -m ampy_config.cli render --profile dev --resolve-secrets values
```
### ✅ Validate (schema + semantic checks)
```bash
python tools/validate.py examples/dev.yaml
# Or explicitly:
python tools/validate.py --schema schema/ampy-config.schema.json examples/*.yaml
```
### 🔧 Secrets utilities
```bash
# Resolve (redacted by default)
python -m ampy_config.cli secret get "aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT"
# Print plain (development only)
python -m ampy_config.cli secret get --plain "secret://vault/tiingo#token"
# Invalidate cache entry
python -m ampy_config.cli secret rotate "gcp-sm://projects/demo/secrets/AMPY_API/versions/latest"
```
### 🤖 Run the agent
```bash
export NATS_URL="nats://127.0.0.1:4222"
export AMPY_CONFIG_SERVICE="ampy-config-agent"
python -m ampy_config.cli agent --profile dev
```
**It subscribes to:**
```
ampy.dev.control.v1.config_preview
ampy.dev.control.v1.config_apply
ampy.dev.control.v1.secret_rotated
```
### ⚡ Ops: preview & apply a runtime override
**Create an overlay:**
```bash
cat >/tmp/overlay.yaml <<'YAML'
oms:
risk:
max_order_notional_usd: 77777
YAML
```
**Preview** (validate only):
```bash
python -m ampy_config.cli ops preview \
--profile dev \
--overlay-file /tmp/overlay.yaml \
--expires-at "2025-12-31T23:59:59Z" \
--reason "intraday risk tightening" \
--dry-run
```
**Apply** (persist) and **wait** until it's effective in the resolved view:
```bash
python -m ampy_config.cli ops apply \
--profile dev \
--overlay-file /tmp/overlay.yaml \
--wait-applied --timeout 20
```
**Then verify:**
```bash
python -m ampy_config.cli render \
--profile dev \
--runtime runtime/overrides.yaml \
--resolve-secrets redacted \
--provenance
```
---
## 🐍 Use from a service (Python example)
```python
# examples/service_skel.py
import asyncio, os
from ampy_config.layering import build_effective_config
from ampy_config.bus.ampy_bus import AmpyBus
from ampy_config.control.events import subjects
async def main():
cfg, _ = build_effective_config(
schema_path="schema/ampy-config.schema.json",
defaults_path="config/defaults.yaml",
profile_yaml="examples/dev.yaml",
overlays=[],
service_overrides=[],
env_allowlist_path="env_allowlist.txt",
env_file=None,
runtime_overrides_path="runtime/overrides.yaml",
)
print("[service] max_order_notional_usd =", cfg["oms"]["risk"]["max_order_notional_usd"])
bus = AmpyBus(os.environ.get("NATS_URL"))
await bus.connect()
subs = subjects(cfg["bus"]["topic_prefix"])
async def on_apply(subject, data):
# Re-build after apply; in real code, you’d update state atomically & validate
new_cfg, _ = build_effective_config(
"schema/ampy-config.schema.json",
"config/defaults.yaml",
"examples/dev.yaml",
[], [], "env_allowlist.txt", None, "runtime/overrides.yaml"
)
print("[service] updated max_order_notional_usd =", new_cfg["oms"]["risk"]["max_order_notional_usd"])
await bus.subscribe_json(subs["apply"], on_apply)
while True:
await asyncio.sleep(1)
if __name__ == "__main__":
os.environ.setdefault("AMPY_CONFIG_SERVICE", "ampy-service-demo")
os.environ.setdefault("NATS_URL", "nats://127.0.0.1:4222")
asyncio.run(main())
```
### 🐹 Go Client Usage
**Start the agent:**
```bash
./bin/ampyconfig-agent \
-nats "$NATS_URL" \
-topic ampy/dev \
-runtime runtime/overrides.yaml \
-service ampy-config-agent \
-log info
```
**Apply configuration changes:**
```bash
cat >/tmp/overlay.yaml <<'YAML'
oms:
risk:
max_order_notional_usd: 123456
YAML
./bin/ampyconfig-ops \
-nats "$NATS_URL" \
-topic ampy/dev \
-overlay-file /tmp/overlay.yaml \
-wait-applied -timeout 20 \
-runtime runtime/overrides.yaml
```
**Available binaries:**
- `ampyconfig-ops` — publish `config_preview`, `config_apply`, `secret_rotated`
- `ampyconfig-agent` — consume control events and persist `runtime/overrides.yaml`
- `ampyconfig-listener` — example service listener that reacts to changes
> 📝 **Status:** v0 thin client — Python `ampy-config` remains the source of truth for schema validation and layering. This Go module focuses on control-plane parity and operational UX.
### 🌐 Go / C++ services
- Parse the **effective YAML** (rendered by ops at boot or on a schedule)
- Subscribe to the same control-plane subjects and re-load your resolved config (or just read `runtime/overrides.yaml`) when a `config_apply` is observed
- Keep reloads **transactional** for safety-critical domains
---
## 📊 Schema notes (metrics example)
The schema allows **either** OTLP (with endpoint) **or** Prometheus (with port):
```json
"metrics": {
"type": "object",
"additionalProperties": false,
"properties": {
"exporter": { "type": "string", "enum": ["otlp", "prom"] },
"endpoint": { "type": "string" },
"sampling_ratio": { "type": "number", "minimum": 0, "maximum": 1 },
"port": { "type": "integer", "minimum": 1, "maximum": 65535 }
},
"required": ["exporter"]
}
```
**Examples:**
```yaml
# OTLP
metrics:
exporter: otlp
endpoint: https://otel.dev.ampyfin.com:4317
sampling_ratio: 0.25
# Prometheus
metrics:
exporter: prom
port: 9464
```
---
## 🌍 Environment variables
| Variable | Description | Example |
|----------|-------------|---------|
| `NATS_URL` | NATS server URL | `nats://127.0.0.1:4222` |
| `AMPY_CONFIG_SERVICE` | Logical service name (used to derive durable names) | `ampy-config-agent` |
| `AMPY_CONFIG_RUNTIME_OVERRIDES` | Path for persisted runtime overrides | `runtime/overrides.yaml` |
| `AMPY_CONFIG_LOCAL_SECRETS` | Path to local dev secrets JSON | `.secrets.local.json` |
| `AMPY_CONFIG_SECRET_TTL_MS` | Secrets cache TTL in milliseconds | `120000` |
| `AMPY_CONFIG_JS_FALLBACK` | Force direct NATS subscription fallback | `1` (skip JetStream) |
**Secret Backend Variables:**
- **🔐 Vault**: `VAULT_ADDR`, `VAULT_TOKEN` (if using `secret://`)
- **☁️ AWS**: `AWS_DEFAULT_REGION` + credentials (if using `aws-sm://`)
- **🌐 GCP**: `GOOGLE_APPLICATION_CREDENTIALS` (if using `gcp-sm://`)
---
## 🔧 Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| **🤖 Agent only shows one subscription** | Blocked while initializing a secret backend (e.g., boto3 waiting for metadata/creds) | Unset or configure that backend properly, or run with only local secrets in dev |
| **⏰ No messages consumed / timeouts** | `NATS_URL` points to wrong port, JetStream disabled, or missing stream/consumers | Check `NATS_URL`, enable JetStream, verify `ampy-control` stream & consumers exist |
| **❌ Apply says OK but value didn't change** | Agent didn't write `runtime/overrides.yaml` or service doesn't reload config | Verify file path via `AMPY_CONFIG_RUNTIME_OVERRIDES` and service reloads on `config_apply` |
| **⚠️ Schema validation passes but semantic check fails** | Semantic checks run after schema validation | Fix the offending values called out in the error |
---
## 🛡️ Security notes
- 🔒 **Secrets are never logged**; redaction is enforced throughout the library
- 🚨 **Prefer fail-shut** for safety-critical domains (OMS risk, broker creds) and **fail-open** for low-risk knobs (metric sampling)
- 🔐 **Ensure access to secret backends** is locked down with least privilege
---
## 🤝 Contributing
PRs welcome! Please include tests for new config keys, validation rules, and control-plane flows.
**Before submitting:**
```bash
pytest -q
python tools/validate.py examples/*.yaml
```
---
## 📄 License
Apache-2.0 (proposed). See `LICENSE` for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "ampy-config",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ampyfin, config, secrets, trading, yaml",
"author": "AmpyFin contributors",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/a9/e4/8b7987434e2418e1b0a9b233d009942b252057291c4079cb236370149ab4/ampy_config-1.1.3.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n# \ud83d\ude80 ampy-config\n\n**Typed Configuration & Secrets Fa\u00e7ade for AmpyFin**\n\n[](https://www.python.org/downloads/)\n[](LICENSE)\n[](https://github.com/AmpyFin/ampy-config/actions)\n[](https://pypi.org/project/ampy-config/)\n[](https://github.com/psf/black)\n\n> \ud83c\udfaf **Single, safe source of truth** for configuration and secrets across AmpyFin services \n> \ud83d\udd17 Built to integrate with **ampy-bus** (control plane over NATS/JetStream) and **ampy-proto** (payload contracts)\n\n[\ud83d\udcd6 Documentation](#-highlights-what-you-get) \u2022 [\ud83d\ude80 Quick Start](#-install-python--pypi) \u2022 [\ud83d\udd27 Usage](#-cli-usage) \u2022 [\ud83e\udd1d Contributing](#-contributing)\n\n</div>\n\n---\n\n## \ud83d\udccb Table of Contents\n\n- [\ud83c\udfaf Why this exists](#-why-this-exists-the-problem)\n- [\u2728 Highlights](#-highlights-what-you-get)\n- [\ud83d\ude80 Install](#-install-python--pypi)\n- [\ud83c\udfae Control Plane](#-control-plane-natsjetstream)\n- [\ud83d\udcda Layering Model](#-layering-model)\n- [\ud83d\udd10 Secrets](#-secrets-indirection-caching-rotation-redaction)\n- [\ud83d\udcbb CLI Usage](#-cli-usage)\n- [\ud83d\udc0d Python Integration](#-use-from-a-service-python-example)\n- [\ud83d\udcca Schema Examples](#-schema-notes-metrics-example)\n- [\ud83c\udf0d Environment Variables](#-environment-variables)\n- [\ud83d\udd27 Troubleshooting](#-troubleshooting)\n- [\ud83d\udee1\ufe0f Security](#-security-notes)\n- [\ud83e\udd1d Contributing](#-contributing)\n\n---\n\n## \ud83c\udfaf Why this exists (the problem)\n\nWithout a unified configuration layer, distributed trading systems tend to develop:\n\n> \u26a0\ufe0f **Common Issues:**\n> - **ENV/YAML sprawl** \u2192 drift, surprises, outages\n> - **Secret handling risks** \u2192 credentials in logs, brittle rotations, no redaction\n> - **Non-reproducibility** \u2192 can't reconstruct exactly which parameters were live for a given trade/run\n> - **Inconsistent runtime behavior** \u2192 some services reload, others require restarts\n\n**ampy-config** provides a single, typed, validated, observable configuration view with clean secret indirection and a runtime control plane for safe updates.\n\n---\n\n## \u2728 Highlights (what you get)\n\n| Feature | Description |\n|---------|-------------|\n| \ud83d\udd0d **Typed schema + validation** | JSON Schema + semantic cross-field checks |\n| \ud83d\udcda **Layering & precedence** | defaults \u2192 environment profile \u2192 overlays \u2192 ENV allowlist \u2192 runtime overrides |\n| \ud83d\udd10 **Secret indirection** | `secret://\u2026`, `aws-sm://\u2026`, `gcp-sm://\u2026` with caching, rotation, and universal redaction |\n| \ud83c\udfae **Control plane for updates** | `config_preview` \u2192 `config_apply` \u2192 `config_applied` events on NATS (JetStream) |\n| \ud83d\udcca **Auditability & observability** | provenance for each key; logs/metrics/traces (no secrets) |\n| \ud83c\udf10 **Language-agnostic** | produces plain YAML effective config for Python, Go, C++, etc. |\n\n---\n\n## \ud83d\ude80 Install\n\n### \ud83d\udc0d Python / PyPI\n\n```bash\npip install ampy-config\n```\n\n**Developer mode** (local repo):\n```bash\npip install -e .\n```\n\n### \ud83d\udc39 Go Client\n\n**Library:**\n```bash\ngo get github.com/AmpyFin/ampy-config/go/ampyconfig@v0.1.0\n```\n\n**Binaries:**\n```bash\ncd go/ampyconfig\nmake # builds bin/ampyconfig-{ops,agent,listener}\n```\n\n> \ud83d\udce6 **Available on [pkg.go.dev](https://pkg.go.dev/github.com/AmpyFin/ampy-config/go/ampyconfig)**\n\n### \ud83d\udd27 Optional secret backends\n\n| Backend | Install Command | Use Case |\n|---------|----------------|----------|\n| \ud83d\udd10 **HashiCorp Vault** | `pip install hvac` | Enterprise secret management |\n| \u2601\ufe0f **AWS Secrets Manager** | `pip install boto3` | AWS-native secret storage |\n| \ud83c\udf10 **GCP Secret Manager** | `pip install google-cloud-secret-manager` | Google Cloud secret storage |\n\n> \ud83d\udca1 **Tip:** You **do not** need to sign up for all of these. Choose one or more real backends for your deployment; the library gracefully falls back to a local JSON file in development.\n\n---\n\n## \ud83c\udfae Control plane (NATS/JetStream)\n\n**Start a local NATS with JetStream:**\n```bash\ndocker run --rm -d --name nats -p 4222:4222 nats:2.10 -js\nexport NATS_URL=\"nats://127.0.0.1:4222\"\n```\n\n**Provision the stream and durable consumers** (once). Using the `nats` CLI:\n\n```bash\n# Stream to cover all control-plane subjects\nnats --server \"$NATS_URL\" stream add ampy-control \\\n --subjects \"ampy.*.control.v1.*\" \\\n --retention limits --max-age 24h --storage file \\\n --max-msgs 10000 --max-bytes 100MB --discard old --defaults\n\n# Agent durables (pull + explicit ack)\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-preview \\\n --filter \"ampy.dev.control.v1.config_preview\" --pull --deliver all --ack explicit --defaults\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-config-apply \\\n --filter \"ampy.dev.control.v1.config_apply\" --pull --deliver all --ack explicit --defaults\nnats --server \"$NATS_URL\" consumer add ampy-control ampy-config-agent-ampy-dev-control-v1-secret-rotated \\\n --filter \"ampy.dev.control.v1.secret_rotated\" --pull --deliver all --ack explicit --defaults\n```\n\n**Verify setup:**\n```bash\nnats --server \"$NATS_URL\" stream ls\nnats --server \"$NATS_URL\" consumer ls ampy-control\n```\n\n> \u26a1 **Note:** The library can also auto-provision if permitted, but explicit creation is more predictable for local dev and CI.\n\n---\n\n## \ud83d\udcda Layering model\n\nEffective config = **merge** in this order (later overrides earlier):\n\n```mermaid\ngraph TD\n A[\ud83d\udcc4 Defaults<br/>config/defaults.yaml] --> B[\ud83c\udf0d Environment Profile<br/>examples/dev.yaml]\n B --> C[\ud83d\udccb Overlays<br/>--overlay path]\n C --> D[\ud83d\udd27 ENV Allowlist<br/>env_allowlist.txt]\n D --> E[\u26a1 Runtime Overrides<br/>runtime/overrides.yaml]\n E --> F[\u2705 Final Config]\n```\n\n| Layer | Description | Example |\n|-------|-------------|---------|\n| 1\ufe0f\u20e3 **Defaults** | Checked-in base config | `config/defaults.yaml` |\n| 2\ufe0f\u20e3 **Environment profile** | Environment-specific settings | `examples/dev.yaml`, `examples/paper.yaml`, `examples/prod.yaml` |\n| 3\ufe0f\u20e3 **Overlays** | Region/cluster/service YAMLs | `--overlay path` (repeatable) |\n| 4\ufe0f\u20e3 **ENV allowlist** | Environment variable mapping | `env_allowlist.txt` maps allowed env keys |\n| 5\ufe0f\u20e3 **Runtime overrides** | Live configuration updates | `runtime/overrides.yaml` (written by agent) |\n\nEach key tracks **provenance**: where it came from (defaults/profile/overlay/ENV/runtime).\n\n### \ud83d\udccf Units & types\n\n| Type | Format | Examples |\n|------|--------|----------|\n| \u23f1\ufe0f **Durations** | String format | `150ms`, `2s`, `5m`, `1h` |\n| \ud83d\udcca **Sizes** | String format | `128KiB`, `1MiB` |\n| \ud83c\udff7\ufe0f **Domains** | Explicit prefixes | `oms.*`, `ingest.*`, `broker.*`, `ml.*`, `warehouse.*`, `fx.*`, `metrics`, `logging`, `tracing`, `security.*`, `feature_flags.*` |\n\n---\n\n## \ud83d\udd10 Secrets (indirection, caching, rotation, redaction)\n\nUse **references**, not literal values:\n\n| Backend | Format | Example |\n|---------|--------|---------|\n| \ud83d\udd10 **Vault** | `secret://vault/<path>#<key>` | `secret://vault/tiingo#token` |\n| \u2601\ufe0f **AWS SM** | `aws-sm://<name>?versionStage=AWSCURRENT` | `aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT` |\n| \ud83c\udf10 **GCP SM** | `gcp-sm://projects/<project>/secrets/<name>/versions/latest` | `gcp-sm://projects/demo/secrets/AMPY_API/versions/latest` |\n\n**Local development fallback file** (`.secrets.local.json`):\n```json\n{\n \"secret://vault/tiingo#token\": \"TIINGO_LOCAL_DEV_TOKEN\",\n \"aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT\": \"ALPACA_LOCAL_DEV_SECRET\",\n \"gcp-sm://projects/demo/secrets/AMPY_API/versions/latest\": \"AMPY_LOCAL_DEV_API\"\n}\n```\n\n> \ud83d\udd12 **Security:** Secrets are **always redacted** in logs/metrics/traces; rotation is signaled via `secret_rotated` events.\n\n---\n\n## \ud83d\udcbb CLI usage\n\nAll commands are available via `python -m ampy_config.cli \u2026` (works without global entrypoints).\n\n### \ud83c\udfa8 Render effective config\n\n```bash\npython -m ampy_config.cli render \\\n --profile dev \\\n --resolve-secrets redacted \\\n --provenance\n```\n\n**Write it to a file:**\n```bash\npython -m ampy_config.cli render \\\n --profile dev \\\n --resolve-secrets redacted \\\n --output /tmp/effective.yaml\n```\n\n**Resolve values** (dev only; requires `.secrets.local.json` or configured backends):\n```bash\nAMPY_CONFIG_LOCAL_SECRETS=.secrets.local.json \\\npython -m ampy_config.cli render --profile dev --resolve-secrets values\n```\n\n### \u2705 Validate (schema + semantic checks)\n\n```bash\npython tools/validate.py examples/dev.yaml\n# Or explicitly:\npython tools/validate.py --schema schema/ampy-config.schema.json examples/*.yaml\n```\n\n### \ud83d\udd27 Secrets utilities\n\n```bash\n# Resolve (redacted by default)\npython -m ampy_config.cli secret get \"aws-sm://ALPACA_SECRET?versionStage=AWSCURRENT\"\n\n# Print plain (development only)\npython -m ampy_config.cli secret get --plain \"secret://vault/tiingo#token\"\n\n# Invalidate cache entry\npython -m ampy_config.cli secret rotate \"gcp-sm://projects/demo/secrets/AMPY_API/versions/latest\"\n```\n\n### \ud83e\udd16 Run the agent\n\n```bash\nexport NATS_URL=\"nats://127.0.0.1:4222\"\nexport AMPY_CONFIG_SERVICE=\"ampy-config-agent\"\n\npython -m ampy_config.cli agent --profile dev\n```\n\n**It subscribes to:**\n```\nampy.dev.control.v1.config_preview\nampy.dev.control.v1.config_apply\nampy.dev.control.v1.secret_rotated\n```\n\n### \u26a1 Ops: preview & apply a runtime override\n\n**Create an overlay:**\n```bash\ncat >/tmp/overlay.yaml <<'YAML'\noms:\n risk:\n max_order_notional_usd: 77777\nYAML\n```\n\n**Preview** (validate only):\n```bash\npython -m ampy_config.cli ops preview \\\n --profile dev \\\n --overlay-file /tmp/overlay.yaml \\\n --expires-at \"2025-12-31T23:59:59Z\" \\\n --reason \"intraday risk tightening\" \\\n --dry-run\n```\n\n**Apply** (persist) and **wait** until it's effective in the resolved view:\n```bash\npython -m ampy_config.cli ops apply \\\n --profile dev \\\n --overlay-file /tmp/overlay.yaml \\\n --wait-applied --timeout 20\n```\n\n**Then verify:**\n```bash\npython -m ampy_config.cli render \\\n --profile dev \\\n --runtime runtime/overrides.yaml \\\n --resolve-secrets redacted \\\n --provenance\n```\n\n---\n\n## \ud83d\udc0d Use from a service (Python example)\n\n```python\n# examples/service_skel.py\nimport asyncio, os\nfrom ampy_config.layering import build_effective_config\nfrom ampy_config.bus.ampy_bus import AmpyBus\nfrom ampy_config.control.events import subjects\n\nasync def main():\n cfg, _ = build_effective_config(\n schema_path=\"schema/ampy-config.schema.json\",\n defaults_path=\"config/defaults.yaml\",\n profile_yaml=\"examples/dev.yaml\",\n overlays=[],\n service_overrides=[],\n env_allowlist_path=\"env_allowlist.txt\",\n env_file=None,\n runtime_overrides_path=\"runtime/overrides.yaml\",\n )\n print(\"[service] max_order_notional_usd =\", cfg[\"oms\"][\"risk\"][\"max_order_notional_usd\"])\n\n bus = AmpyBus(os.environ.get(\"NATS_URL\"))\n await bus.connect()\n subs = subjects(cfg[\"bus\"][\"topic_prefix\"])\n\n async def on_apply(subject, data):\n # Re-build after apply; in real code, you\u2019d update state atomically & validate\n new_cfg, _ = build_effective_config(\n \"schema/ampy-config.schema.json\",\n \"config/defaults.yaml\",\n \"examples/dev.yaml\",\n [], [], \"env_allowlist.txt\", None, \"runtime/overrides.yaml\"\n )\n print(\"[service] updated max_order_notional_usd =\", new_cfg[\"oms\"][\"risk\"][\"max_order_notional_usd\"])\n\n await bus.subscribe_json(subs[\"apply\"], on_apply)\n while True:\n await asyncio.sleep(1)\n\nif __name__ == \"__main__\":\n os.environ.setdefault(\"AMPY_CONFIG_SERVICE\", \"ampy-service-demo\")\n os.environ.setdefault(\"NATS_URL\", \"nats://127.0.0.1:4222\")\n asyncio.run(main())\n```\n\n### \ud83d\udc39 Go Client Usage\n\n**Start the agent:**\n```bash\n./bin/ampyconfig-agent \\\n -nats \"$NATS_URL\" \\\n -topic ampy/dev \\\n -runtime runtime/overrides.yaml \\\n -service ampy-config-agent \\\n -log info\n```\n\n**Apply configuration changes:**\n```bash\ncat >/tmp/overlay.yaml <<'YAML'\noms:\n risk:\n max_order_notional_usd: 123456\nYAML\n\n./bin/ampyconfig-ops \\\n -nats \"$NATS_URL\" \\\n -topic ampy/dev \\\n -overlay-file /tmp/overlay.yaml \\\n -wait-applied -timeout 20 \\\n -runtime runtime/overrides.yaml\n```\n\n**Available binaries:**\n- `ampyconfig-ops` \u2014 publish `config_preview`, `config_apply`, `secret_rotated`\n- `ampyconfig-agent` \u2014 consume control events and persist `runtime/overrides.yaml`\n- `ampyconfig-listener` \u2014 example service listener that reacts to changes\n\n> \ud83d\udcdd **Status:** v0 thin client \u2014 Python `ampy-config` remains the source of truth for schema validation and layering. This Go module focuses on control-plane parity and operational UX.\n\n### \ud83c\udf10 Go / C++ services\n\n- Parse the **effective YAML** (rendered by ops at boot or on a schedule)\n- Subscribe to the same control-plane subjects and re-load your resolved config (or just read `runtime/overrides.yaml`) when a `config_apply` is observed\n- Keep reloads **transactional** for safety-critical domains\n\n---\n\n## \ud83d\udcca Schema notes (metrics example)\n\nThe schema allows **either** OTLP (with endpoint) **or** Prometheus (with port):\n\n```json\n\"metrics\": {\n \"type\": \"object\",\n \"additionalProperties\": false,\n \"properties\": {\n \"exporter\": { \"type\": \"string\", \"enum\": [\"otlp\", \"prom\"] },\n \"endpoint\": { \"type\": \"string\" },\n \"sampling_ratio\": { \"type\": \"number\", \"minimum\": 0, \"maximum\": 1 },\n \"port\": { \"type\": \"integer\", \"minimum\": 1, \"maximum\": 65535 }\n },\n \"required\": [\"exporter\"]\n}\n```\n\n**Examples:**\n\n```yaml\n# OTLP\nmetrics:\n exporter: otlp\n endpoint: https://otel.dev.ampyfin.com:4317\n sampling_ratio: 0.25\n\n# Prometheus\nmetrics:\n exporter: prom\n port: 9464\n```\n\n---\n\n## \ud83c\udf0d Environment variables\n\n| Variable | Description | Example |\n|----------|-------------|---------|\n| `NATS_URL` | NATS server URL | `nats://127.0.0.1:4222` |\n| `AMPY_CONFIG_SERVICE` | Logical service name (used to derive durable names) | `ampy-config-agent` |\n| `AMPY_CONFIG_RUNTIME_OVERRIDES` | Path for persisted runtime overrides | `runtime/overrides.yaml` |\n| `AMPY_CONFIG_LOCAL_SECRETS` | Path to local dev secrets JSON | `.secrets.local.json` |\n| `AMPY_CONFIG_SECRET_TTL_MS` | Secrets cache TTL in milliseconds | `120000` |\n| `AMPY_CONFIG_JS_FALLBACK` | Force direct NATS subscription fallback | `1` (skip JetStream) |\n\n**Secret Backend Variables:**\n- **\ud83d\udd10 Vault**: `VAULT_ADDR`, `VAULT_TOKEN` (if using `secret://`)\n- **\u2601\ufe0f AWS**: `AWS_DEFAULT_REGION` + credentials (if using `aws-sm://`)\n- **\ud83c\udf10 GCP**: `GOOGLE_APPLICATION_CREDENTIALS` (if using `gcp-sm://`)\n\n---\n\n## \ud83d\udd27 Troubleshooting\n\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| **\ud83e\udd16 Agent only shows one subscription** | Blocked while initializing a secret backend (e.g., boto3 waiting for metadata/creds) | Unset or configure that backend properly, or run with only local secrets in dev |\n| **\u23f0 No messages consumed / timeouts** | `NATS_URL` points to wrong port, JetStream disabled, or missing stream/consumers | Check `NATS_URL`, enable JetStream, verify `ampy-control` stream & consumers exist |\n| **\u274c Apply says OK but value didn't change** | Agent didn't write `runtime/overrides.yaml` or service doesn't reload config | Verify file path via `AMPY_CONFIG_RUNTIME_OVERRIDES` and service reloads on `config_apply` |\n| **\u26a0\ufe0f Schema validation passes but semantic check fails** | Semantic checks run after schema validation | Fix the offending values called out in the error |\n\n---\n\n## \ud83d\udee1\ufe0f Security notes\n\n- \ud83d\udd12 **Secrets are never logged**; redaction is enforced throughout the library\n- \ud83d\udea8 **Prefer fail-shut** for safety-critical domains (OMS risk, broker creds) and **fail-open** for low-risk knobs (metric sampling)\n- \ud83d\udd10 **Ensure access to secret backends** is locked down with least privilege\n\n---\n\n## \ud83e\udd1d Contributing\n\nPRs welcome! Please include tests for new config keys, validation rules, and control-plane flows.\n\n**Before submitting:**\n```bash\npytest -q\npython tools/validate.py examples/*.yaml\n```\n\n---\n\n## \ud83d\udcc4 License\n\nApache-2.0 (proposed). See `LICENSE` for details.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Typed configuration & secrets facade for AmpyFin (layering, validation, secrets, control-plane)",
"version": "1.1.3",
"project_urls": null,
"split_keywords": [
"ampyfin",
" config",
" secrets",
" trading",
" yaml"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "d646756f4b88cd0cd255dcc5a21492cb3a6713721f0cba9298d89194d3121cc0",
"md5": "6dc2a168e572a26416113bbd35f5b3b3",
"sha256": "1ae18f80cb6a8c040ae14b65014b666a409e1947712441e734cd4da9a6f5e8ac"
},
"downloads": -1,
"filename": "ampy_config-1.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6dc2a168e572a26416113bbd35f5b3b3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 55344,
"upload_time": "2025-09-09T17:28:19",
"upload_time_iso_8601": "2025-09-09T17:28:19.965593Z",
"url": "https://files.pythonhosted.org/packages/d6/46/756f4b88cd0cd255dcc5a21492cb3a6713721f0cba9298d89194d3121cc0/ampy_config-1.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "a9e48b7987434e2418e1b0a9b233d009942b252057291c4079cb236370149ab4",
"md5": "8c9d9df0693d1e4d465a0f5cc5d2c71a",
"sha256": "75d408e640a6027b00df7fdf1621fb1aa9c73a9992236e91f2ff49a0975b9354"
},
"downloads": -1,
"filename": "ampy_config-1.1.3.tar.gz",
"has_sig": false,
"md5_digest": "8c9d9df0693d1e4d465a0f5cc5d2c71a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 36057,
"upload_time": "2025-09-09T17:28:21",
"upload_time_iso_8601": "2025-09-09T17:28:21.371426Z",
"url": "https://files.pythonhosted.org/packages/a9/e4/8b7987434e2418e1b0a9b233d009942b252057291c4079cb236370149ab4/ampy_config-1.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-09 17:28:21",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ampy-config"
}
JSON