# What is this?
A config system that doesn't waste your time
- all values in the hierarchy can be overridden with CLI args
- select multiple profiles from (ex: GPU & DEV or UNIX & GPU & PROD) from the CLI, profile settings get merged recursively
- configs supports secrets (unsynced file along side git-committed config)
- provides a consistent central way to handle filepaths
- profiles can inherit other profiles
- default works along side `argparse`, but also can replace it
- can combine/import multiple config files
# How do I use this?
`pip install quik-config`
In a `config.py`:
```python
from quik_config import find_and_load
info = find_and_load(
"info.yaml", # walks up folders until it finds a file with this name
cd_to_filepath=True, # helpful if using relative paths
fully_parse_args=True, # if you already have argparse, use parse_args=True instead
show_help_for_no_args=False, # change if you want
)
# info.path_to
# info.absolute_path_to
# info.unused_args
# info.secrets
# info.available_profiles
# info.selected_profiles
# info.root_path
# info.project
# info.local_data
# info.as_dict
print(info.config) # dictionary
```
Create `info.yaml` with a structure like this:
```yaml
# names in parentheses are special, all other names are not!
# (e.g. add/extend this with any custom fields)
(project):
# the local_data file will be auto-generated
# (its for machine-specific data)
# so probably git-ignore whatever path you pick
(local_data): ./local_data.ignore.yaml
# example profiles
(profiles):
(default):
blah: "blah blah blah"
mode: development # or production. Same thing really
has_gpu: maybe
constants:
pi: 3 # its 'bout 3
PROFILE1:
constants:
e: 2.7182818285
PROD:
mode: production
constants:
pi: 3.1415926536
problems: true
```
Then run it:
```shell
python ./config.py
```
Which will print out this config:
```py
{
"blah": "blah blah blah", # from (default)
"mode": "development", # from (default)
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3, # from (default)
},
}
```
# Features
### Builtin Help
```shell
python ./config.py --help --profiles
```
```
available profiles:
- DEV
- GPU
- PROD
as cli argument:
-- --profiles='["DEV"]'
-- --profiles='["GPU"]'
-- --profiles='["PROD"]'
```
```
---------------------------------------------------------------------------------
QUIK CONFIG HELP
---------------------------------------------------------------------------------
open the file below and look for "(profiles)" for more information:
$PWD/info.yaml
examples:
python3 ./ur_file.py -- --help --profiles
python3 ./ur_file.py -- --help key1
python3 ./ur_file.py -- --help key1:subKey
python3 ./ur_file.py -- --help key1:subKey key2
python3 ./ur_file.py -- --profiles='[YOUR_PROFILE, YOUR_OTHER_PROFILE]'
python3 ./ur_file.py -- thing1:"Im a new value" part2:"10000"
python3 ./ur_file.py -- thing1:"I : cause errors" part2:10000
python3 ./ur_file.py -- 'thing1:"I : dont cause errors" part2:10000
python3 ./ur_file.py -- 'thing1:["Im in a list"]'
python3 ./ur_file.py -- 'thing1:part_A:"Im nested"'
python3 ./ur_file.py "I get sent to ./ur_file.py" -- part2:"new value"
python3 ./ur_file.py "I get ignored" "me too" -- part2:10000
how it works:
- the "--" is a required argument, quik config only looks after the --
- given "thing1:10", "thing1" is the key, "10" is the value
- All values are parsed as json/yaml
- "true" is boolean true
- "10" is a number
- '"10"' is a string (JSON escaping)
- '"10\n"' is a string with a newline
- '[10,11,hello]' is a list with two numbers and an unquoted string
- '{"thing": 10}' is a map/object
- "blah blah" is an un-quoted string with a space. Yes its valid YAML
- multiline values are valid, you can dump an whole JSON doc as 1 arg
- "thing1:10" overrides the "thing1" in the (profiles) of the info.yaml
- "thing:subThing:10" is shorthand, 10 is the value, the others are keys
it will only override the subThing (and will create it if necessary)
- '{"thing": {"subThing":10} }' is long-hand for "thing:subThing:10"
- '"thing:subThing":10' will currently not work for shorthand (parse error)
options:
--help
--profiles
---------------------------------------------------------------------------------
your default top-level keys:
- mode
- has_gpu
- constants
your local defaults file:
./local_data.ignore.yaml
your default profiles:
- DEV
---------------------------------------------------------------------------------
```
### Select Profiles from CLI
```shell
python ./config.py @PROFILE1
```
prints:
```py
{
"blah": "blah blah blah", # from (default)
"mode": "development", # from (default)
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
},
}
```
```shell
python ./config.py @PROFILE1 @PROD
```
prints:
```py
{
"blah": "blah blah blah", # from (default)
"mode": "production", # from PROD
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
"problems": True, # from PROD
},
}
```
### Override Values from CLI
```shell
python ./config.py @PROFILE1 mode:custom constants:problems:99
```
prints:
```py
{
"blah": "blah blah blah", # from (default)
"mode": "custom", # from CLI
"has_gpu": "maybe", # from (default)
"constants": {
"pi": 3.1415926536, # from (default)
"e": 2.7182818285, # from PROFILE1
"problems": 99, # from CLI
},
}
```
Again but with really complicated arguments: <br>
(each argument is parsed as yaml)
```shell
python ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'
```
prints:
```py
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3.1415926,
"tau": 6.2831853072,
"reserved_letters": ["C", "K", "i", ],
},
}
unused_args: ["arg1"]
```
### Working Alongside Argparse (quick)
Remove `fully_parse_args` and replace it with just `parse_args`
```py
info = find_and_load(
"info.yaml",
parse_args=True, # <- will only parse after --
)
```
Everthing in the CLI is the same, but it waits for `--`
For example:
```shell
# quik_config ignores arg1 --arg2 arg3, so argparse can do its thing with them
python ./config.py arg1 --arg2 arg3 -- @PROD
```
### Working Alongside Argparse (advanced)
Arguments can simply be passed as a list of strings, which can be useful for running many combinations of configs.
```py
info = find_and_load(
"info.yaml",
args=[ "@PROD" ],
)
```
### Relative and absolute paths
Add them to the info.yaml
```yaml
(project):
(local_data): ./local_data.ignore.yaml
# filepaths (relative to location of info.yaml)
(path_to):
this_file: "./info.yaml"
blah_file: "./data/results.txt"
# example profiles
(profiles):
(default):
blah: "blah blah blah"
```
Access them in python
```py
info = find_and_load("info.yaml")
info.path_to.blah_file
info.absolute_path_to.blah_file # nice when then PWD != folder of the info file
```
### Import other yaml files
You can import multiple profiles by specifying profile sources.<br>
NOTE: the last profile source will override (merge keys) with the previous ones, but the main config will take the priority over any/all profile sources.
```yaml
(project):
(profile_sources):
- ./comments.yaml
- ./camera_profiles.yaml
(profiles):
# you can also load a single profile as a file
(GPU): !load_yaml_file ./profiles/gpu.yaml
```
## Different Profiles For Different Machines
Lets say you've several machines and an info.yaml like this:
```yaml
(project):
(profiles):
DEV:
cores: 1
database_ip: 192.168.10.10
mode: dev
LAPTOP:
cores: 2
DESKTOP:
cores: 8
UNIX:
line_endings: "\n"
WINDOWS:
line_endings: "\r\n"
PROD:
database_ip: 183.177.10.83
mode: prod
cores: 32
```
And lets say you have a `config.py` like this:
```python
from quik_config import find_and_load
info = find_and_load(
"info.yaml",
defaults_for_local_data=["DEV", ],
# if the ./local_data.ignore.yaml doesnt exist,
# => create it and add DEV as the default no-argument choice
)
```
Run the code once to get a `./local_data.ignore.yaml` file. <br>
Each machine gets to pick the profiles it defaults to.<br>
So, on your Macbook you can edit the `./local_data.ignore.yaml` to include something like the following:
```yaml
(selected_profiles):
- LAPTOP # the cores:2 will be used (instead of cores:1 from DEV)
- UNIX # because LAPTOP is higher in the list than DEV
- DEV
```
On your Windows laptop you can edit it and put:
```yaml
(selected_profiles):
- LAPTOP
- WINDOWS
- DEV
```
## Command Line Arguments
If you have `run.py` like this:
```python
from quik_config import find_and_load
info = find_and_load("info.yaml", parse_args=True)
print("config:", info.config )
print("unused_args:", info.unused_args)
#
# call some other function you've got
#
#from your_code import run
#run(*info.unused_args)
```
### Example 0
Using the python file and config file above
```shell
python ./run.py
```
Running that will output:
```py
config: {
"mode": "development",
"has_gpu": False,
"constants": {
"pi": 3
}
}
unused_args: []
```
### Example 1
Show help. This output can be overridden in the info.yaml by setting `(help):` under the `(project):` key.
```shell
python ./run.py -- --help
```
Note the `--` is needed in front of the help.
You can also add `show_help_for_no_args=True` if you want that behavior. <br>
Ex:
```python
from quik_config import find_and_load
info = find_and_load(
"info.yaml",
show_help_for_no_args=True
parse_args=True,
)
```
### Example 2
Again but selecting some profiles
```shell
python ./run.py arg1 -- --profiles='[PROD]'
# or
python ./run.py arg1 -- @PROD
```
Output:
```py
config: {
"mode": "production",
"has_gpu": False,
"constants": {
"pi": 3.1415926536,
"problems": True,
},
}
unused_args: ["arg1"]
```
### Example 3
Again but with custom arguments:
```shell
python ./run.py arg1 -- mode:my_custom_mode constants:tau:6.2831853072
```
```py
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3,
"tau": 6.2831853072,
},
}
unused_args: ["arg1"]
```
### Example 4
Again but with really complicated arguments: <br>
(each argument is parsed as yaml)
```shell
python ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'
```
prints:
```py
config: {
"mode": "my_custom_mode",
"has_gpu": False,
"constants": {
"pi": 3.1415926,
"tau": 6.2831853072,
"reserved_letters": ["C", "K", "i", ],
},
}
unused_args: ["arg1"]
```
## Auto Generate Config-Specific Log Folders
If you add `path_from_config_to_log_folder="../logs",` as an argument to `find_and_load`
- The config is hashed
- A folder will be generated for each config
- A "run_index" is created (incrementes by 1 for each run with the same config)
- A nested folder will be created for each run
- `info.unique_run_path` is probably what you care about the most; the absolute path to log folder for this run
- `info.unique_config_path` is an absolute path to the unique config folder (summary stats for a specific config)
By default:
- config path is `./logs/[created_date]__[hash_of_config]/specific_config.yaml`
- run path is `./logs/[created_date]__[hash_of_config]/[run_index]__[date]__[git_commit]/`
- run index is padded with zeros (ex: `0001`)
- If you don't have git, the `__[git_commit]` won't be there
- The `./logs/[created_date]__[hash_of_config]/running_data.yaml` will look like this:
```yaml
# guaranteed one line per run, each line is JSON
- {"run_index": 0, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.327494", "inital_run_name": "run/hi2_0000"}
- {"run_index": 1, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.341325", "inital_run_name": "0001__2024-07-01--15-22__1a69c8"}
- {"run_index": 2, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.541018", "inital_run_name": "0002__2024-07-01--15-22__1a69c8"}
```
You can customize almost everything:
- You can rename* any of the folders manually (and it'll keep working)
- *the config path NEEDS to end with the hash of the config, but thats the only constraint
- Ex: `./logs/your_name[hash_of_config]/`
- padding of the run index (ex: `0001`) with `run_index_padding=4,`
- the length of the config hash (ex: `6`) with `config_hash_length=6,`
- the length of the git commit (ex: `6`) with `default_git_commit_length=6,`
- using `config_renamer`, `run_namer`, and `config_initial_namer` like so:
```py
from quik_config import find_and_load
config_renamer1 = lambda **_: "blahblah_"
config_renamer2 = lambda config, **_: f"{config.experiment_name}_"
config_renamer3 = lambda config, time, **_: f"{config.experiment_name}_{time}_"
config_renamer4 = lambda config, date, time, **_: f"{config.experiment_name}_{date}_{time}_"
config_renamer5 = lambda config, date, **_: f"{config.experiment_name}_{date}_"
config_renamer6 = lambda run_index, **_: f"{time}_{run_index}__"
# NOTE: run_index is a string, because its padded-out with zeros
run_namer1 = lambda **_: "blahblah"
run_namer2 = lambda run_index, **_: f"{run_index}"
run_namer3 = lambda run_index, **_: f"runs/{run_index}"
run_namer4 = lambda info, run_index, date, time, **_: f"runs_{date}/{run_index}"
# NOTE: for "run_namer", its okay to return "thing/{run_index}"
# The sub-folders will be created
# only use this if you want to manually rename these folders
# (config_renamer would undo your manual rename)
config_initial_namer1 = lambda **_: "blahblah_"
config_initial_namer2 = lambda info, time, **_: f"0_{time}_"
#
# basic example
#
info = find_and_load(
"info.yaml",
path_from_config_to_log_folder="../logs",
run_index_padding=4,
config_hash_length=6,
default_git_commit_length=6,
config_renamer: lambda date, **_: f"{date}_",
run_namer: lambda info, run_index, date, time, **_: f"{run_index}_{date}_{time}_",
config_initial_namer: lambda info, time, **_: f"{time}_",
config_renamer: lambda info, **_: f"blahblah_", # can "update" the name
)
print(info.this_run.run_index) # starts at 0 if this is a new/unique config
print(info.this_run.git_commit) # full 40-character git commit hash
print(info.this_run.start_time) # datetime.datetime.now() object (not a string)
print(info.unique_run_path)
print(info.unique_config_path)
print(info.log_folder)
#
# lambda arg options
#
lambda date, **_: "blah" # string: "2024-07-01"
lambda time, **_: "blah" # string: "17-52" (NOTE: "-" because colon is not valid filename character)
lambda datetime, **_: "blah" # string: '2024-06-28--17-26'
lambda now, **_: "blah" # datetime.datetime object
lambda config_hash, **_: "blah" # string: "lak4fa"
lambda git_commit, **_: "blah" # 40-character git commit hash as a string
lambda time_with_seconds, **_: "blah" # string: "17:52:00"
lambda time_with_milliseconds, **_: "blah" # string: "17:52:00.000"
lambda year, **_: "blah" # int
lambda month, **_: "blah" # int
lambda day, **_: "blah" # int
lambda hour, **_: "blah" # int
lambda minute, **_: "blah" # int
lambda second, **_: "blah" # int
lambda unix_seconds, **_: "blah" # int
lambda unix_milliseconds, **_: "blah" # int
```
<!--
TODO:
- make argument parsing not need the -- by default
- allow for multiple back to back runs with different args
-->
Raw data
{
"_id": null,
"home_page": "https://github.com/jeff-hykin/quik_config_python.git",
"name": "quik-config",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Jeff Hykin",
"author_email": "jeff.hykin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d5/d1/a1b7e6b2d4ff57b5d994c888b922d61ce068d0b4485ae4ed552339480335/quik_config-1.9.1.tar.gz",
"platform": null,
"description": "# What is this?\n\nA config system that doesn't waste your time\n- all values in the hierarchy can be overridden with CLI args\n- select multiple profiles from (ex: GPU & DEV or UNIX & GPU & PROD) from the CLI, profile settings get merged recursively\n- configs supports secrets (unsynced file along side git-committed config)\n- provides a consistent central way to handle filepaths\n- profiles can inherit other profiles\n- default works along side `argparse`, but also can replace it\n- can combine/import multiple config files\n\n# How do I use this?\n\n`pip install quik-config`\n\nIn a `config.py`: \n```python\nfrom quik_config import find_and_load\n\ninfo = find_and_load(\n \"info.yaml\", # walks up folders until it finds a file with this name\n cd_to_filepath=True, # helpful if using relative paths\n fully_parse_args=True, # if you already have argparse, use parse_args=True instead\n show_help_for_no_args=False, # change if you want\n)\n# info.path_to\n# info.absolute_path_to\n# info.unused_args\n# info.secrets\n# info.available_profiles\n# info.selected_profiles\n# info.root_path\n# info.project\n# info.local_data\n# info.as_dict\nprint(info.config) # dictionary\n```\n\nCreate `info.yaml` with a structure like this:\n```yaml\n# names in parentheses are special, all other names are not!\n# (e.g. add/extend this with any custom fields)\n(project):\n # the local_data file will be auto-generated\n # (its for machine-specific data)\n # so probably git-ignore whatever path you pick\n (local_data): ./local_data.ignore.yaml\n \n # example profiles\n (profiles):\n (default):\n blah: \"blah blah blah\"\n mode: development # or production. Same thing really\n has_gpu: maybe\n constants:\n pi: 3 # its 'bout 3 \n \n PROFILE1:\n constants:\n e: 2.7182818285\n \n PROD:\n mode: production\n constants:\n pi: 3.1415926536\n problems: true\n```\n\nThen run it:\n```shell\npython ./config.py\n```\n\nWhich will print out this config:\n```py\n{\n \"blah\": \"blah blah blah\", # from (default)\n \"mode\": \"development\", # from (default)\n \"has_gpu\": \"maybe\", # from (default)\n \"constants\": {\n \"pi\": 3, # from (default)\n },\n}\n```\n\n# Features\n\n### Builtin Help\n\n```shell\npython ./config.py --help --profiles\n```\n\n```\navailable profiles:\n - DEV\n - GPU\n - PROD\n\nas cli argument:\n -- --profiles='[\"DEV\"]'\n -- --profiles='[\"GPU\"]'\n -- --profiles='[\"PROD\"]'\n```\n\n\n```\n ---------------------------------------------------------------------------------\n QUIK CONFIG HELP\n ---------------------------------------------------------------------------------\n \n open the file below and look for \"(profiles)\" for more information:\n $PWD/info.yaml\n \n examples:\n python3 ./ur_file.py -- --help --profiles\n python3 ./ur_file.py -- --help key1\n python3 ./ur_file.py -- --help key1:subKey\n python3 ./ur_file.py -- --help key1:subKey key2\n python3 ./ur_file.py -- --profiles='[YOUR_PROFILE, YOUR_OTHER_PROFILE]'\n python3 ./ur_file.py -- thing1:\"Im a new value\" part2:\"10000\"\n python3 ./ur_file.py -- thing1:\"I : cause errors\" part2:10000\n python3 ./ur_file.py -- 'thing1:\"I : dont cause errors\" part2:10000\n python3 ./ur_file.py -- 'thing1:[\"Im in a list\"]'\n python3 ./ur_file.py -- 'thing1:part_A:\"Im nested\"'\n python3 ./ur_file.py \"I get sent to ./ur_file.py\" -- part2:\"new value\"\n python3 ./ur_file.py \"I get ignored\" \"me too\" -- part2:10000\n \n how it works:\n - the \"--\" is a required argument, quik config only looks after the --\n - given \"thing1:10\", \"thing1\" is the key, \"10\" is the value\n - All values are parsed as json/yaml\n - \"true\" is boolean true\n - \"10\" is a number\n - '\"10\"' is a string (JSON escaping)\n - '\"10\\n\"' is a string with a newline\n - '[10,11,hello]' is a list with two numbers and an unquoted string\n - '{\"thing\": 10}' is a map/object\n - \"blah blah\" is an un-quoted string with a space. Yes its valid YAML\n - multiline values are valid, you can dump an whole JSON doc as 1 arg\n - \"thing1:10\" overrides the \"thing1\" in the (profiles) of the info.yaml\n - \"thing:subThing:10\" is shorthand, 10 is the value, the others are keys\n it will only override the subThing (and will create it if necessary)\n - '{\"thing\": {\"subThing\":10} }' is long-hand for \"thing:subThing:10\"\n - '\"thing:subThing\":10' will currently not work for shorthand (parse error)\n \n options:\n --help\n --profiles\n \n ---------------------------------------------------------------------------------\n \n your default top-level keys:\n - mode\n - has_gpu\n - constants\n your local defaults file:\n ./local_data.ignore.yaml\n your default profiles:\n - DEV\n \n ---------------------------------------------------------------------------------\n\n```\n\n\n### Select Profiles from CLI\n\n```shell\npython ./config.py @PROFILE1\n```\n\nprints:\n```py\n{\n \"blah\": \"blah blah blah\", # from (default)\n \"mode\": \"development\", # from (default)\n \"has_gpu\": \"maybe\", # from (default)\n \"constants\": {\n \"pi\": 3.1415926536, # from (default)\n \"e\": 2.7182818285, # from PROFILE1\n },\n}\n```\n\n```shell\npython ./config.py @PROFILE1 @PROD\n```\n\nprints:\n```py\n{\n \"blah\": \"blah blah blah\", # from (default)\n \"mode\": \"production\", # from PROD\n \"has_gpu\": \"maybe\", # from (default)\n \"constants\": {\n \"pi\": 3.1415926536, # from (default)\n \"e\": 2.7182818285, # from PROFILE1\n \"problems\": True, # from PROD\n },\n}\n```\n\n### Override Values from CLI\n\n```shell\npython ./config.py @PROFILE1 mode:custom constants:problems:99\n```\n\nprints:\n```py\n{\n \"blah\": \"blah blah blah\", # from (default)\n \"mode\": \"custom\", # from CLI\n \"has_gpu\": \"maybe\", # from (default)\n \"constants\": {\n \"pi\": 3.1415926536, # from (default)\n \"e\": 2.7182818285, # from PROFILE1\n \"problems\": 99, # from CLI\n },\n}\n```\n\nAgain but with really complicated arguments: <br>\n(each argument is parsed as yaml)\n\n```shell\npython ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ \"C\", \"K\", \"i\" ] }'\n```\n\nprints:\n```py\nconfig: {\n \"mode\": \"my_custom_mode\", \n \"has_gpu\": False, \n \"constants\": {\n \"pi\": 3.1415926, \n \"tau\": 6.2831853072, \n \"reserved_letters\": [\"C\", \"K\", \"i\", ], \n }, \n}\nunused_args: [\"arg1\"]\n```\n\n### Working Alongside Argparse (quick)\n\nRemove `fully_parse_args` and replace it with just `parse_args`\n\n```py\ninfo = find_and_load(\n \"info.yaml\",\n parse_args=True, # <- will only parse after -- \n)\n```\n\nEverthing in the CLI is the same, but it waits for `--`\nFor example:\n\n```shell\n# quik_config ignores arg1 --arg2 arg3, so argparse can do its thing with them\npython ./config.py arg1 --arg2 arg3 -- @PROD\n```\n\n### Working Alongside Argparse (advanced)\n\nArguments can simply be passed as a list of strings, which can be useful for running many combinations of configs.\n\n```py\ninfo = find_and_load(\n \"info.yaml\",\n args=[ \"@PROD\" ],\n)\n```\n\n### Relative and absolute paths\n\nAdd them to the info.yaml\n\n```yaml\n(project):\n (local_data): ./local_data.ignore.yaml\n \n # filepaths (relative to location of info.yaml)\n (path_to):\n this_file: \"./info.yaml\"\n blah_file: \"./data/results.txt\"\n \n # example profiles\n (profiles):\n (default):\n blah: \"blah blah blah\"\n```\n\nAccess them in python\n```py\ninfo = find_and_load(\"info.yaml\")\ninfo.path_to.blah_file\ninfo.absolute_path_to.blah_file # nice when then PWD != folder of the info file\n```\n\n### Import other yaml files\n\nYou can import multiple profiles by specifying profile sources.<br>\nNOTE: the last profile source will override (merge keys) with the previous ones, but the main config will take the priority over any/all profile sources.\n\n```yaml\n(project):\n (profile_sources):\n - ./comments.yaml\n - ./camera_profiles.yaml\n \n (profiles):\n # you can also load a single profile as a file\n (GPU): !load_yaml_file ./profiles/gpu.yaml\n```\n\n\n## Different Profiles For Different Machines\n\nLets say you've several machines and an info.yaml like this:\n```yaml\n(project):\n (profiles):\n DEV:\n cores: 1\n database_ip: 192.168.10.10\n mode: dev\n LAPTOP:\n cores: 2\n DESKTOP:\n cores: 8\n UNIX:\n line_endings: \"\\n\"\n WINDOWS:\n line_endings: \"\\r\\n\"\n PROD:\n database_ip: 183.177.10.83\n mode: prod\n cores: 32\n```\n\nAnd lets say you have a `config.py` like this:\n```python\nfrom quik_config import find_and_load\ninfo = find_and_load(\n \"info.yaml\",\n defaults_for_local_data=[\"DEV\", ],\n # if the ./local_data.ignore.yaml doesnt exist,\n # => create it and add DEV as the default no-argument choice\n)\n```\n\nRun the code once to get a `./local_data.ignore.yaml` file. <br>\n\nEach machine gets to pick the profiles it defaults to.<br>\nSo, on your Macbook you can edit the `./local_data.ignore.yaml` to include something like the following:\n```yaml\n(selected_profiles):\n - LAPTOP # the cores:2 will be used (instead of cores:1 from DEV)\n - UNIX # because LAPTOP is higher in the list than DEV\n - DEV\n```\n\nOn your Windows laptop you can edit it and put:\n```yaml\n(selected_profiles):\n - LAPTOP\n - WINDOWS\n - DEV\n```\n\n## Command Line Arguments\n\nIf you have `run.py` like this:\n\n```python\nfrom quik_config import find_and_load\n\ninfo = find_and_load(\"info.yaml\", parse_args=True)\n\nprint(\"config:\", info.config )\nprint(\"unused_args:\", info.unused_args)\n\n# \n# call some other function you've got\n# \n#from your_code import run\n#run(*info.unused_args)\n```\n\n### Example 0\n\nUsing the python file and config file above\n\n```shell\npython ./run.py\n```\n\nRunning that will output:\n\n```py\nconfig: {\n \"mode\": \"development\",\n \"has_gpu\": False,\n \"constants\": {\n \"pi\": 3\n }\n}\nunused_args: []\n```\n\n### Example 1\n\nShow help. This output can be overridden in the info.yaml by setting `(help):` under the `(project):` key.\n\n```shell\npython ./run.py -- --help\n```\n\nNote the `--` is needed in front of the help.\n\nYou can also add `show_help_for_no_args=True` if you want that behavior. <br>\nEx:\n\n```python\nfrom quik_config import find_and_load\ninfo = find_and_load(\n \"info.yaml\",\n show_help_for_no_args=True\n parse_args=True,\n)\n```\n\n### Example 2\n\nAgain but selecting some profiles\n\n```shell\npython ./run.py arg1 -- --profiles='[PROD]'\n# or\npython ./run.py arg1 -- @PROD\n```\n\nOutput:\n\n```py\nconfig: {\n \"mode\": \"production\",\n \"has_gpu\": False,\n \"constants\": {\n \"pi\": 3.1415926536,\n \"problems\": True,\n },\n}\nunused_args: [\"arg1\"]\n```\n\n### Example 3\n\nAgain but with custom arguments:\n\n```shell\npython ./run.py arg1 -- mode:my_custom_mode constants:tau:6.2831853072\n```\n\n```py\nconfig: {\n \"mode\": \"my_custom_mode\",\n \"has_gpu\": False,\n \"constants\": {\n \"pi\": 3,\n \"tau\": 6.2831853072,\n },\n}\nunused_args: [\"arg1\"]\n```\n\n### Example 4\n\nAgain but with really complicated arguments: <br>\n(each argument is parsed as yaml)\n\n```shell\npython ./run.py arg1 -- mode:my_custom_mode 'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ \"C\", \"K\", \"i\" ] }'\n```\n\nprints:\n\n```py\nconfig: {\n \"mode\": \"my_custom_mode\", \n \"has_gpu\": False, \n \"constants\": {\n \"pi\": 3.1415926, \n \"tau\": 6.2831853072, \n \"reserved_letters\": [\"C\", \"K\", \"i\", ], \n }, \n}\nunused_args: [\"arg1\"]\n```\n\n## Auto Generate Config-Specific Log Folders\n\nIf you add `path_from_config_to_log_folder=\"../logs\",` as an argument to `find_and_load`\n- The config is hashed\n- A folder will be generated for each config\n- A \"run_index\" is created (incrementes by 1 for each run with the same config)\n- A nested folder will be created for each run\n- `info.unique_run_path` is probably what you care about the most; the absolute path to log folder for this run\n- `info.unique_config_path` is an absolute path to the unique config folder (summary stats for a specific config)\n\nBy default:\n- config path is `./logs/[created_date]__[hash_of_config]/specific_config.yaml`\n- run path is `./logs/[created_date]__[hash_of_config]/[run_index]__[date]__[git_commit]/`\n - run index is padded with zeros (ex: `0001`)\n - If you don't have git, the `__[git_commit]` won't be there\n- The `./logs/[created_date]__[hash_of_config]/running_data.yaml` will look like this:\n```yaml\n# guaranteed one line per run, each line is JSON\n- {\"run_index\": 0, \"git_commit\": \"1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5\", \"start_time\": \"2024-07-01T15:22:09.327494\", \"inital_run_name\": \"run/hi2_0000\"}\n- {\"run_index\": 1, \"git_commit\": \"1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5\", \"start_time\": \"2024-07-01T15:22:09.341325\", \"inital_run_name\": \"0001__2024-07-01--15-22__1a69c8\"}\n- {\"run_index\": 2, \"git_commit\": \"1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5\", \"start_time\": \"2024-07-01T15:22:09.541018\", \"inital_run_name\": \"0002__2024-07-01--15-22__1a69c8\"}\n```\n\nYou can customize almost everything:\n- You can rename* any of the folders manually\u00a0 (and it'll keep working)\n - *the config path NEEDS to end with the hash of the config, but thats the only constraint\n - Ex: `./logs/your_name[hash_of_config]/`\n- padding of the run index (ex: `0001`) with `run_index_padding=4,`\n- the length of the config hash (ex: `6`) with `config_hash_length=6,`\n- the length of the git commit (ex: `6`) with `default_git_commit_length=6,`\n- using `config_renamer`, `run_namer`, and `config_initial_namer` like so:\n\n```py\nfrom quik_config import find_and_load\n\nconfig_renamer1 = lambda **_: \"blahblah_\"\nconfig_renamer2 = lambda config, **_: f\"{config.experiment_name}_\"\nconfig_renamer3 = lambda config, time, **_: f\"{config.experiment_name}_{time}_\"\nconfig_renamer4 = lambda config, date, time, **_: f\"{config.experiment_name}_{date}_{time}_\"\nconfig_renamer5 = lambda config, date, **_: f\"{config.experiment_name}_{date}_\"\nconfig_renamer6 = lambda run_index, **_: f\"{time}_{run_index}__\"\n# NOTE: run_index is a string, because its padded-out with zeros\n\nrun_namer1 = lambda **_: \"blahblah\"\nrun_namer2 = lambda run_index, **_: f\"{run_index}\"\nrun_namer3 = lambda run_index, **_: f\"runs/{run_index}\"\nrun_namer4 = lambda info, run_index, date, time, **_: f\"runs_{date}/{run_index}\"\n# NOTE: for \"run_namer\", its okay to return \"thing/{run_index}\"\n# The sub-folders will be created\n\n# only use this if you want to manually rename these folders\n# (config_renamer would undo your manual rename)\nconfig_initial_namer1 = lambda **_: \"blahblah_\"\nconfig_initial_namer2 = lambda info, time, **_: f\"0_{time}_\"\n\n# \n# basic example\n# \ninfo = find_and_load(\n \"info.yaml\",\n path_from_config_to_log_folder=\"../logs\",\n run_index_padding=4,\n config_hash_length=6,\n default_git_commit_length=6,\n config_renamer: lambda date, **_: f\"{date}_\",\n run_namer: lambda info, run_index, date, time, **_: f\"{run_index}_{date}_{time}_\",\n config_initial_namer: lambda info, time, **_: f\"{time}_\",\n config_renamer: lambda info, **_: f\"blahblah_\", # can \"update\" the name\n)\nprint(info.this_run.run_index) # starts at 0 if this is a new/unique config\nprint(info.this_run.git_commit) # full 40-character git commit hash\nprint(info.this_run.start_time) # datetime.datetime.now() object (not a string)\n\nprint(info.unique_run_path)\nprint(info.unique_config_path)\nprint(info.log_folder)\n\n# \n# lambda arg options\n# \n lambda date, **_: \"blah\" # string: \"2024-07-01\"\n lambda time, **_: \"blah\" # string: \"17-52\" (NOTE: \"-\" because colon is not valid filename character)\n lambda datetime, **_: \"blah\" # string: '2024-06-28--17-26'\n lambda now, **_: \"blah\" # datetime.datetime object\n lambda config_hash, **_: \"blah\" # string: \"lak4fa\"\n lambda git_commit, **_: \"blah\" # 40-character git commit hash as a string\n lambda time_with_seconds, **_: \"blah\" # string: \"17:52:00\"\n lambda time_with_milliseconds, **_: \"blah\" # string: \"17:52:00.000\"\n lambda year, **_: \"blah\" # int\n lambda month, **_: \"blah\" # int\n lambda day, **_: \"blah\" # int\n lambda hour, **_: \"blah\" # int\n lambda minute, **_: \"blah\" # int\n lambda second, **_: \"blah\" # int\n lambda unix_seconds, **_: \"blah\" # int\n lambda unix_milliseconds, **_: \"blah\" # int\n```\n\n<!-- \nTODO:\n- make argument parsing not need the -- by default\n- allow for multiple back to back runs with different args\n -->\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Project config files",
"version": "1.9.1",
"project_urls": {
"Homepage": "https://github.com/jeff-hykin/quik_config_python.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c63d8c9be4814451ce8de91828aaf89d2410ea48a2f7ec5de1f8d87969fedb40",
"md5": "4f457dbbb61dc940ab717ade28c014c6",
"sha256": "46508a7abbec10e097028340e7e67c3264d8dc9699ef299f8f54ec7948779201"
},
"downloads": -1,
"filename": "quik_config-1.9.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4f457dbbb61dc940ab717ade28c014c6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 4679539,
"upload_time": "2024-07-01T20:35:40",
"upload_time_iso_8601": "2024-07-01T20:35:40.233575Z",
"url": "https://files.pythonhosted.org/packages/c6/3d/8c9be4814451ce8de91828aaf89d2410ea48a2f7ec5de1f8d87969fedb40/quik_config-1.9.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d5d1a1b7e6b2d4ff57b5d994c888b922d61ce068d0b4485ae4ed552339480335",
"md5": "f4c7bab925411ef66a959d53c8dbf081",
"sha256": "f82f0717667cd2f1ffcd556b0ea3cd942b23c5fc1b7b21c9e01e025d6ae05601"
},
"downloads": -1,
"filename": "quik_config-1.9.1.tar.gz",
"has_sig": false,
"md5_digest": "f4c7bab925411ef66a959d53c8dbf081",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 4955559,
"upload_time": "2024-07-01T20:35:42",
"upload_time_iso_8601": "2024-07-01T20:35:42.649111Z",
"url": "https://files.pythonhosted.org/packages/d5/d1/a1b7e6b2d4ff57b5d994c888b922d61ce068d0b4485ae4ed552339480335/quik_config-1.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-01 20:35:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jeff-hykin",
"github_project": "quik_config_python",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "quik-config"
}
JSON
