# Project description
#### easy_configer version : 2.3.2
---
### Configeruating the program in an easy-way
I'm willing to provide a light-weight solution for configurating your python program. Hope this repository make every user control their large project more easier ~ ~
### Introduction 📝
With the python project go into large-scale, a lot of argument will be required to control the complex business logic, user may need a simple way to load configurations through a file eventually. Their exists various package cover part of function and offer some solution to tackle the mentioned problem.
**Unfortunately, I can not find a solution for load & use the argument in simple manner at least.** Instead, most of the config-tools seems only works for the specific goal, then cause the code more longer and hard to read.
For example :
## ConfigParser
import ConfigParser
Config = ConfigParser.ConfigParser()
Config.read("c:\\tomorrow.ini")
# get arg via method
Config.get(section, option)
# or get arg with converter
int(Config['lucky_num'])
## Argparse
import argparse
parse = argparse.ArgumentParser("description string")
parse.add_argument("--lucky_num", type=int)
...
args = parser.parse_args()
args.lucky_num
That leverage me to package my solution for solving this issue. The easy_config will cover the following attributes :
1. **Hierachical section config (nested dictionary)**
2. **Accept multiple config file in dynamic loading manner**
3. **Support customized class (initialized by keyword arguments)**
4. **Commend-line update all declared-value wherever it belong, even in hierachical section**
5. **Support the absl style FLAGS functionality (declare once, use anywhere)**
And, of course the following attribute will also be supported :
* dot-access of any arguments (even in nested dictionary)
* inline comment '#', now you can write comment in everyline ~
* support arguments interpolation!!
* support config conversion, which turn easy_config into the other kind of config package (omegaconf, argparse, ..., etc.)
* support hierachical configurating system with dynamic override ~
---
### Newly update features 🚀
1. Enable all test case (automatically ci by git-action)
2. Support dot-access of any arguments
3. Consistent import syntax..
4. New document is released ~
---
### Bug Fixed 🐛
#### Fix-up import syntax.. plz open an issue if you find a bug ~
---
### Dependencies 🏗️
This package is written for Python 3.8. After refactor in this version, this package also support down to python 3.6!!
Of course, light-weight solution **do not** contain any 3-rd package complex dependencies.
The python standard package (such as pathlib, sys, .., etc) is the only source of dependencies, so you don't need to worry about that ~ ~
> However, if you want to use the IO_Converter for converting config into omegaconf, you still need to install omegaconf for this functionality ~
---
### Installation ⚙️<br>
1. **pypi install** <br>
simply type the `pip install easy_configer` (due to name conflict of pypi pkg, we use different pkg name)
2. **install from source code** <br>
clone the project from github : `git clone repo-link`
Chage to the root directory of the cloned project, and type `pip install -e .`
3. **import syntax** <br>
Wherever you install, pypi or source. Now, you just need a simple import : `from easy_configer.Configer import Configer`
---
### Quick start 🥂
#### **1. Handy example of config file**
#### Let's say we have an easy-config for development enviroment on jupyter notebook. we want to define several variable for configurating a simple math calculation.
# config string
cfg_str = '''
title = 'math calculation'@str
coef = 1e-3@float
with_intercept = True@bool
intercept = 3@int
'''
# look's good, let's get the config!
from easy_configer.Config import Config
cfg = Config(description="math calculation config!", cmd_args=False)
cfg.cfg_from_str(cfg_str)
# oh.. wait, could we do it more easier ?
ez_cfg_str = '''
# opps.. let's change some value
title = 'linear equation'
coef = 15
'''
# Note : every time you load the config, if you have the same variable name,
# it will override the value of the variable!
cfg.cfg_from_str(ez_cfg_str)
lin_equ = lambda x : cfg.coef * x + cfg.intercept if cfg.with_intercept else (cfg.coef * x)
x = 15
print( f"Linear equation with x={x} : { lin_equ(x) }" )
#### In larger project, we may write a config file to control the program, so that the config will become easy to trace, check and debug. In here, we first prepare an config called `test_cfg.ini` in the working directory.
For easy-config file, there're two type of argument : flatten argument, hierachical argument. You can see that flatten argument is placed in first shallow level, and the argument could be easily accessed by dot operator. Besides flatten argument, all of hierachical argument will be placed in python dict object, thus accessing each argument by key string!
# ./test_cfg.ini
# '#' denote comment line, the inline comment is also supported!
# define 'flatten' arguments :
serv_host = '127.0.0.1'
serv_port = 9478@int # specific type is also allowed!!
api_keys = {'TW_REGION':'SV92VS92N20', 'US_REGION':'W92N8WN029N2'}
# define 'hierachical' arguments :
# the 'section' is the key of accessing dict value and could be defined as follows :
[db_setup]
db_host = $serv_host
# first `export mongo_port=5566` in your bash, then support os.env interpolation!
db_port = $Env.mongo_port
snap_shot = True
# and then define second section for backend server..
[bknd_srv]
load_8bit = True
async_req = True
chat_mode = 'inference'
model_type = 'LlaMa2.0'
[bknd_srv.mod_params]
log_hist = False
tempeture = 1e-4
model_mode = $bknd_srv.chat_mode # hierachical args interpolation
<br>
Now, we're free to launch the chatbot via `python quick_start.py` (*quick_start.py in work directory*)!
However, you can also override the arguemnts via commendline `python quick_start.py serv_port=7894`
import sys
# main_block
if __name__ == "__main__":
from easy_configer.Configer import Configer
cfger = Configer(description="chat-bot configuration", cmd_args=True)
# we have defined a config file, let's try to load it!
cfger.cfg_from_ini("./test_cfg.ini")
# Display the Namespace, it will display all flatten arguemnts and first-level sections
print(cfger)
... # for building chat-bot instance `Chat_server`
chat_serv = Chat_server(host=cfger.serv_host, port=cfger.serv_port, api_keys=cfger.api_keys)
... # build mongo-db instance `mongo_serv` for logging chat history..
mongo_serv.init_setup( **cfger.db_setup )
... # loading llm model instance `Llama` ~
llm_mod = Llama(
ld_8bit=cfger.bknd_srv.load_8bit,
chat_mode=cfger.chat_mode,
model_type=cfger.model_type
)
# you can access nested-dict by dot access ~
llm_mod.init_mod_param( **cfger.bknd_srv.mod_params )
# or you can keep the dict fashion ~
if cfger.bknd_srv['async_req']:
chat_serv.chat_mod = llm_mod
chat_serv.hist_db = mongo_serv
else:
... # write sync conversation by yourself..
sys.exit( chat_serv.server_forever() )
> Note that the recommended way to access the argument is **still** key-string access `cfger.args['#4$%-var']`, as you may notice, dot-access doesn't support **ugly** variable name.
<br>
---
### More detail tutorial about each topic is as follows :
#### **2. How to declare hierachical config**
There have two kind of way to prepare the arguments in easy-config : we can either define flatten argument or groupping the multiple arguments in an hierachical manner (begin from second level). In most of time, we define the flatten argument as global setup, and arrange the rest of arguments into the corresponding dictionary for easy to assign it to the subroutine.
#### Let's give a deep-learning example ~
#### *hier_cfg.ini in work directory*
glb_seed = 42
exp_id = '0001'
# we call '...' in [...] as section name,
# i.e. we can assign dict dataset to subroutine by `build_dataset(**cfg.dataset)`, just such easy!!
[dataset]
service_port = 65536
path = '/data/kitti'
# of course, nested dict is also supported! it just the native python dictionary in dictionary!
[dataset.loader]
batch_size = 32
[model]
[model.backbone]
mod_typ = 'resnet'
[model.backbone.optimizer]
lay_seed = 42
[train_cfg]
batch_size = 32
[train_cfg.opt]
opt_typ = 'Adam'
lr = 1e-4
sched = 'cos_anneal'
#### We have defined the config file, now let's see how to access any agruments! Execute `python quick_hier.py` in work directory*.
from easy_configer.Configer import Configer
if __name__ == "__main__":
cfger = Configer(cmd_args=True)
# omit cfg_from_str, hier-config also could be declared in str though ~
cfger.cfg_from_ini("./hier_cfg.ini")
print(cfger.dataset)
# output nested dict : { 'service_port':65536, 'path':'/data/kitti', 'loader':{'batch_size':32} }
print(f"key-string access bz : {cfger.dataset['loader']['batch_size']}")
# output - "key-string access bz : 32"
print(f"bz : {cfger.dataset.loader.batch_size}")
# output - "dot-access bz : 32"
# we usually conduct initialization such simple & elegant!
ds = build_dataset(**cfger.dataset)
mod = build_model(**cfger.model)
... # get torch Trainer
Trainer(mod).fit(ds)
However, the syntax of above config file could be improved, isn't it !? For example, the batch_size is defined twice under `dataset.loader` and `train_cfg`, so as layer seed. Moreover, path is defined as python string, it need to be further converted by Path object in python standard package. Could we regist our customized data type for easy-config ?
#### Glade to say : Yes! it's possible to elegantly deal with above mentioned issue. We can solve the first issue by using argument interpolation, and solve the second issue by using the customized register!!
#### *config interpolation with $ symbol* and *customized register method `regist_cnvtor`*
> Currently we support interpolation mechnaism to interpolate **ANY** arguemnts belong the different level of nested dictionary. Moreover, we also support **$Env** for accessing enviroment variables exported in bash!!
# For convience, we define string-config!
def get_str_cfg():
''' # `export glb_seed=42` in bash!!
glb_seed = $Env.glb_seed
exp_id = '0001'
[dataset]
service_port = 65536
# Don't forgot to regist Path object first and the typename will be the given name!!
path = {'path':'/data/kitti'}@pyPath
[dataset.loader]
batch_size = 32
[model]
[model.backbone]
mod_typ = 'resnet'
[model.backbone.optimizer]
lay_seed = $glb_seed
[train_cfg]
batch_size = $dataset.loader.batch_size
[train_cfg.opt]
opt_typ = 'Adam'
lr = 1e-4
sched = 'cos_anneal'
'''
# main_block
if __name__ == "__main__":
from pathlib import Path
cfger = Configer(description="sample for arguments interpolation")
cfger.regist_cnvtor("pyPath", Path) # regist customer class 'Path'
cfg_str = get_str_cfg()
cfger.cfg_from_str(cfg_str)
# do whatever you want to do!
#### **3. Access all arguments flexibly**
We simple set a breakpoint to feel how flexible does `easy_configer.utils.Container.AttributeDict` support.
from easy_configer.Configer import Configer
if __name__ == "__main__":
cfger = Configer()
cfger.cfg_from_ini("./hier_cfg.ini")
breakpoint()
> We write a special example `hier_cfg.ini`!!
# nested-dict
[secA] # test depth ((sub^4)-section under secA)
lev = 1
[secA.secB]
lev = 2
[secA.secB.secC]
lev = 3
[secA.secB.secC.secD]
lev = 4
Now you can access each `lev` :
1. `(pdb) cfger.secA.lev `, output `lev : 1`
2. `(pdb) cfger['secA'].secB['lev'] `, output `lev : 2`, and so on..
3. Most crazy one ~ `(pdb) cfger.secA.['secB'].secC['secD'].lev `, output `lev : 4`
#### **4. Commmend-line Support**
> We also take `hier_cfg.ini` as example!
# hier_cfg.ini
glb_var = 42@int
[dataset]
ds_type = None
path = {'root':'/data/kitti'}@Path
[dataset.loader]
batch_size = 32@int
# Hier-Cell cfg written by Josef-Huang..
Execute python program and print out the helper information <br>
`python quick_hier.py -h`
Update flatten argument and print out the helper information <br>
`python quick_hier.py glb_var=404 -h`
Especially update **non-flatten argument**, you can access any argument at any level by dot-access in commend-line!! (with combining any argument update). Now, try to change any nested argument <br>
`python quick_hier.py dataset.ds_type="'kitti'" dataset.path="{'path':'/root/ds'}" dataset.loader.batch_size=48`
( Note that the commendline declaration for string is tricky, but currently we only support two way for that :
`dataset.ds_type="'kitti'"` or `dataset.ds_type=kitti@str`, pick up one of you like ~ )
#### **5. Import Sub-Config**
Like `omegaconf`, most of user expect to seperate the config based on their type and dynamically merge it in runtime. It's a rational requirement and the previous version of easy-config provide two way to conduct it, but both have it's limit :
1. you can call the `cfg_from_ini` twice, for example, `cfg.cfg_from_ini('./base_cfg') ; cfg.cfg_from_ini('./override_cfg')`. But it's not explicitly load the config thus reducing readability.
2. you can use the config merging, for example, `new_cfg = base_cfg | override_cfg`. But it's not elegant solution while you have to merge several config..
#### Now, we provide the thrid way : **sub-config**. you can import the sub-config in any depth of hierachical config by simply placing the `>` symbol at the beginning of line.
# ./base_cfg.ini
glb_seed = 42@int
[dataset]
> ./config/ds_config.ini
[model]
> ./root/config/model_config.ini
# ./config/ds_config.ini
ds_type = None
path = {'root':'/data/kitti'}@Path
[dataset.loader]
batch_size = 32@int
# ./root/config/model_config.ini
[model.backbone]
mod_typ = 'resnet'
[model.backbone.optimizer]
# and yes, interpolation is still valid "after" the reference argument is declared!
lay_seed = $glb_seed
#### **6. Config Operation**
Config operation is one of the core technique for dynamic configuration system!!
In the following example, you can see that the merging config system already provided a impressive hierachical merging funtionality!
> For example, `ghyu.opop.add` in cfg_a can be replaced by the cfg_b in **same** section with the same variable name, while the different namespace keep their variable safely ~ so the value of `ghyu.opop.add` will be 67 and `ghyu.opop.tueo.inpo` refer the flatten variable `inpo` and the value will be 46.
from easy_configer.Configer import Configer
def build_cfg_text_a():
return '''
# Initial config file :
inpo = 46@int
[test]
mrg_var_tst = [1, 3, 5]@list
[test.ggap]
gtgt = haha@str
[ghyu]
[ghyu.opop]
add = 32@int
[ghyu.opop.tueo]
salt = $inpo
# Cell cfg written by Josef-Huang..
'''
def build_cfg_text_b():
return '''
# Initial config file :
inop = 32@int
[test]
mrg_var_tst = [1, 3, 5]@list
[test.ggap]
gtgt = overrides@str
[test.ggap.conf]
secert = 42@int
[ghyu]
[ghyu.opop]
add = 67@int
div = 1e-4@float
[new]
[new.new]
newsec = wpeo@str
# Cell cfg written by Josef-Huang..
'''
if __name__ == "__main__":
cfg_a = Configer(cmd_args=True)
cfg_a.cfg_from_str(build_cfg_text_a())
cfg_b = Configer()
cfg_b.cfg_from_str(build_cfg_text_b())
# default, override falg is turn off ~
cfg_a.merge_conf(cfg_b, override=True)
# `cfg_b = cfg_b | cfg_a`, operator support, warn to decrease the read-ability...
# cfg_a will override the argument of cfg_b which share the identitical variable name in cfg_b!
# operator support : `cfg_b |= cfg_a` == `cfg_b = cfg_b | cfg_a`
---
### **Miscellnous features**
#### **7. IO Converter**
from dataclasses import dataclass
from typing import Optional
@dataclass
class TableConfig:
rows: int = 1
@dataclass
class DatabaseConfig:
table_cfg: TableConfig = TableConfig()
@dataclass
class ModelConfig:
data_source: Optional[TableConfig] = None
@dataclass
class ServerConfig:
db: DatabaseConfig = DatabaseConfig()
model: ModelConfig = ModelConfig()
if __name__ == '__main__':
from easy_configer.IO_Converter import IO_Converter
# first import the IO_converter
from easy_config.IO_Converter import IO_Converter
cnvt = IO_Converter()
# convert easy_config instance into the argparse instance
argp_cfg = cnvt.cnvt_cfg_to(cfger, 'argparse')
uargp_cfg = cnvt.cnvt_cfg_to(cfger, 'argparse', parse_arg=False)
argp_cfg = uargp_cfg.parse_args()
## convert config INTO..
# convert easy_config instance into the omegaconf instance
ome_cfg = cnvt.cnvt_cfg_to(cfger, 'omegaconf')
# convert easy_config instance into the "yaml string"
yaml_cfg = cnvt.cnvt_cfg_to(cfger, 'yaml')
# convert easy_config instance into the "dict"
yaml_cfg = cnvt.cnvt_cfg_to(cfger, 'dict')
## convert into easy-config FROM..
# argparse, omegaconf, yaml, dict ... is supported
ez_cfg = cnvt.cnvt_cfg_from(argp_cfg, 'omegaconf')
# Especially, it support "dataclass"!
ds_cfg = ServerConfig()
ez_cfg = cnvt.cnvt_cfg_from(ds_cfg, 'dataclass')
#### **8. Absl style flag**
> easy_config also support that you can access the 'same' config file in different python file without re-declare the config. test_flag.py under the same work directory
Suppose you have executed `main.py`:
from easy_configer.Configer import Configer
from utils import get_var_from_flag
if __name__ == "__main__":
cfg = Configer()
cfg.cfg_from_str("var = 32")
# both should output 32 ~
print(f"var from main : {cfg.var}")
print(f"var from flag : { get_var_from_flag() }")
Now, when you step in `get_var_from_flag` function in different file..
from easy_configer.Configer import Configer
def get_var_from_flag():
new_cfger = Configer()
flag = new_cfger.get_cfg_flag()
# test to get the pre-defined 'var'
return flag.var
---
#### **The documentation of easy_configer is also released in read doc** [🔗](https://easy-configer.readthedocs.io/en/latest/)
---
### Simple Unittest 🧪
If you clone this repo and built from source, you can try to run the unittest.
`python -m unittest discover`
> I have placed all test file under test folder.
---
### License
MIT License. More information of each term, please see LICENSE.md
### Author
Josef-Huang, a3285556aa@gmail.com
### Footer
~ Hope God bless everyone in the world to know his word ~ <br>
**The fear of the LORD is the beginning of knowledge; fools despise wisdom and instruction. by Proverbs 1:7**
Raw data
{
"_id": null,
"home_page": "https://github.com/HuangChiEn/easy_config",
"name": "easy-configer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "configuration, commendline argument, argument",
"author": "JosefHuang",
"author_email": "a3285556aa@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d0/9a/58f10f02fc0c8e0e7f7bc03cb01ad35a8842b4c430db7e58f16f70d0bf59/easy_configer-2.3.2.tar.gz",
"platform": null,
"description": "# Project description\n#### easy_configer version : 2.3.2\n\n\n\n\n---\n\n### Configeruating the program in an easy-way \nI'm willing to provide a light-weight solution for configurating your python program. Hope this repository make every user control their large project more easier ~ ~ \n\n### Introduction \ud83d\udcdd\nWith the python project go into large-scale, a lot of argument will be required to control the complex business logic, user may need a simple way to load configurations through a file eventually. Their exists various package cover part of function and offer some solution to tackle the mentioned problem. \n\n**Unfortunately, I can not find a solution for load & use the argument in simple manner at least.** Instead, most of the config-tools seems only works for the specific goal, then cause the code more longer and hard to read.\n\nFor example :\n \n ## ConfigParser\n import ConfigParser \n Config = ConfigParser.ConfigParser()\n Config.read(\"c:\\\\tomorrow.ini\")\n # get arg via method\n Config.get(section, option)\n # or get arg with converter\n int(Config['lucky_num'])\n \n ## Argparse\n import argparse\n parse = argparse.ArgumentParser(\"description string\")\n parse.add_argument(\"--lucky_num\", type=int)\n ...\n args = parser.parse_args()\n args.lucky_num\n \n\nThat leverage me to package my solution for solving this issue. The easy_config will cover the following attributes :\n1. **Hierachical section config (nested dictionary)**\n\n2. **Accept multiple config file in dynamic loading manner**\n\n3. **Support customized class (initialized by keyword arguments)**\n\n4. **Commend-line update all declared-value wherever it belong, even in hierachical section**\n\n5. **Support the absl style FLAGS functionality (declare once, use anywhere)** \n\nAnd, of course the following attribute will also be supported :\n\n* dot-access of any arguments (even in nested dictionary)\n\n* inline comment '#', now you can write comment in everyline ~\n\n* support arguments interpolation!!\n\n* support config conversion, which turn easy_config into the other kind of config package (omegaconf, argparse, ..., etc.)\n\n* support hierachical configurating system with dynamic override ~\n\n---\n\n### Newly update features \ud83d\ude80\n1. Enable all test case (automatically ci by git-action)\n2. Support dot-access of any arguments\n3. Consistent import syntax.. \n4. New document is released ~\n\n---\n\n### Bug Fixed \ud83d\udc1b\n#### Fix-up import syntax.. plz open an issue if you find a bug ~\n---\n\n### Dependencies \ud83c\udfd7\ufe0f\nThis package is written for Python 3.8. After refactor in this version, this package also support down to python 3.6!!\nOf course, light-weight solution **do not** contain any 3-rd package complex dependencies.\nThe python standard package (such as pathlib, sys, .., etc) is the only source of dependencies, so you don't need to worry about that ~ ~\n> However, if you want to use the IO_Converter for converting config into omegaconf, you still need to install omegaconf for this functionality ~\n\n---\n\n### Installation \u2699\ufe0f<br>\n1. **pypi install** <br>\n simply type the `pip install easy_configer` (due to name conflict of pypi pkg, we use different pkg name)\n\n2. **install from source code** <br>\n clone the project from github : `git clone repo-link` \n Chage to the root directory of the cloned project, and type `pip install -e .`\n\n3. **import syntax** <br>\n Wherever you install, pypi or source. Now, you just need a simple import : `from easy_configer.Configer import Configer`\n \n---\n\n### Quick start \ud83e\udd42\n\n#### **1. Handy example of config file**\n#### Let's say we have an easy-config for development enviroment on jupyter notebook. we want to define several variable for configurating a simple math calculation.\n\n # config string\n cfg_str = '''\n title = 'math calculation'@str\n coef = 1e-3@float\n with_intercept = True@bool\n intercept = 3@int\n '''\n\n # look's good, let's get the config!\n from easy_configer.Config import Config\n cfg = Config(description=\"math calculation config!\", cmd_args=False)\n cfg.cfg_from_str(cfg_str)\n\n # oh.. wait, could we do it more easier ?\n ez_cfg_str = '''\n # opps.. let's change some value\n title = 'linear equation'\n coef = 15 \n '''\n # Note : every time you load the config, if you have the same variable name, \n # it will override the value of the variable!\n cfg.cfg_from_str(ez_cfg_str)\n\n lin_equ = lambda x : cfg.coef * x + cfg.intercept if cfg.with_intercept else (cfg.coef * x)\n x = 15\n print( f\"Linear equation with x={x} : { lin_equ(x) }\" )\n\n\n#### In larger project, we may write a config file to control the program, so that the config will become easy to trace, check and debug. In here, we first prepare an config called `test_cfg.ini` in the working directory. \nFor easy-config file, there're two type of argument : flatten argument, hierachical argument. You can see that flatten argument is placed in first shallow level, and the argument could be easily accessed by dot operator. Besides flatten argument, all of hierachical argument will be placed in python dict object, thus accessing each argument by key string! \n \n # ./test_cfg.ini\n # '#' denote comment line, the inline comment is also supported!\n\n # define 'flatten' arguments :\n serv_host = '127.0.0.1' \n serv_port = 9478@int # specific type is also allowed!!\n api_keys = {'TW_REGION':'SV92VS92N20', 'US_REGION':'W92N8WN029N2'}\n \n # define 'hierachical' arguments :\n # the 'section' is the key of accessing dict value and could be defined as follows :\n [db_setup]\n db_host = $serv_host\n # first `export mongo_port=5566` in your bash, then support os.env interpolation!\n db_port = $Env.mongo_port \n snap_shot = True\n \n # and then define second section for backend server..\n [bknd_srv]\n load_8bit = True\n async_req = True\n chat_mode = 'inference'\n model_type = 'LlaMa2.0'\n [bknd_srv.mod_params]\n log_hist = False\n tempeture = 1e-4\n model_mode = $bknd_srv.chat_mode # hierachical args interpolation\n \n<br>\n\nNow, we're free to launch the chatbot via `python quick_start.py` (*quick_start.py in work directory*)!\nHowever, you can also override the arguemnts via commendline `python quick_start.py serv_port=7894`\n \n import sys\n \n # main_block \n if __name__ == \"__main__\":\n from easy_configer.Configer import Configer\n\n cfger = Configer(description=\"chat-bot configuration\", cmd_args=True)\n # we have defined a config file, let's try to load it!\n cfger.cfg_from_ini(\"./test_cfg.ini\")\n \n # Display the Namespace, it will display all flatten arguemnts and first-level sections\n print(cfger)\n \n ... # for building chat-bot instance `Chat_server`\n chat_serv = Chat_server(host=cfger.serv_host, port=cfger.serv_port, api_keys=cfger.api_keys)\n\n ... # build mongo-db instance `mongo_serv` for logging chat history..\n mongo_serv.init_setup( **cfger.db_setup )\n\n ... # loading llm model instance `Llama` ~\n llm_mod = Llama(\n ld_8bit=cfger.bknd_srv.load_8bit, \n chat_mode=cfger.chat_mode, \n model_type=cfger.model_type\n )\n\n # you can access nested-dict by dot access ~\n llm_mod.init_mod_param( **cfger.bknd_srv.mod_params )\n\n # or you can keep the dict fashion ~\n if cfger.bknd_srv['async_req']:\n chat_serv.chat_mod = llm_mod\n chat_serv.hist_db = mongo_serv\n else:\n ... # write sync conversation by yourself..\n\n sys.exit( chat_serv.server_forever() )\n\n> Note that the recommended way to access the argument is **still** key-string access `cfger.args['#4$%-var']`, as you may notice, dot-access doesn't support **ugly** variable name. \n\n<br>\n\n---\n\n### More detail tutorial about each topic is as follows :\n\n#### **2. How to declare hierachical config**\nThere have two kind of way to prepare the arguments in easy-config : we can either define flatten argument or groupping the multiple arguments in an hierachical manner (begin from second level). In most of time, we define the flatten argument as global setup, and arrange the rest of arguments into the corresponding dictionary for easy to assign it to the subroutine. \n\n#### Let's give a deep-learning example ~\n#### *hier_cfg.ini in work directory*\n\n glb_seed = 42\n exp_id = '0001'\n\n # we call '...' in [...] as section name,\n # i.e. we can assign dict dataset to subroutine by `build_dataset(**cfg.dataset)`, just such easy!!\n [dataset] \n service_port = 65536\n path = '/data/kitti'\n # of course, nested dict is also supported! it just the native python dictionary in dictionary!\n [dataset.loader]\n batch_size = 32\n\n [model]\n [model.backbone]\n mod_typ = 'resnet'\n [model.backbone.optimizer]\n lay_seed = 42 \n \n [train_cfg]\n batch_size = 32\n [train_cfg.opt]\n opt_typ = 'Adam'\n lr = 1e-4\n sched = 'cos_anneal'\n\n#### We have defined the config file, now let's see how to access any agruments! Execute `python quick_hier.py` in work directory*.\n\n from easy_configer.Configer import Configer\n \n if __name__ == \"__main__\":\n cfger = Configer(cmd_args=True)\n \n # omit cfg_from_str, hier-config also could be declared in str though ~\n cfger.cfg_from_ini(\"./hier_cfg.ini\")\n \n print(cfger.dataset) \n # output nested dict : { 'service_port':65536, 'path':'/data/kitti', 'loader':{'batch_size':32} }\n \n print(f\"key-string access bz : {cfger.dataset['loader']['batch_size']}\")\n # output - \"key-string access bz : 32\"\n\n print(f\"bz : {cfger.dataset.loader.batch_size}\")\n # output - \"dot-access bz : 32\"\n\n # we usually conduct initialization such simple & elegant!\n ds = build_dataset(**cfger.dataset)\n mod = build_model(**cfger.model)\n ... # get torch Trainer\n Trainer(mod).fit(ds)\n\nHowever, the syntax of above config file could be improved, isn't it !? For example, the batch_size is defined twice under `dataset.loader` and `train_cfg`, so as layer seed. Moreover, path is defined as python string, it need to be further converted by Path object in python standard package. Could we regist our customized data type for easy-config ?\n#### Glade to say : Yes! it's possible to elegantly deal with above mentioned issue. We can solve the first issue by using argument interpolation, and solve the second issue by using the customized register!!\n\n#### *config interpolation with $ symbol* and *customized register method `regist_cnvtor`*\n> Currently we support interpolation mechnaism to interpolate **ANY** arguemnts belong the different level of nested dictionary. Moreover, we also support **$Env** for accessing enviroment variables exported in bash!!\n\n # For convience, we define string-config!\n def get_str_cfg():\n ''' # `export glb_seed=42` in bash!!\n glb_seed = $Env.glb_seed\n exp_id = '0001'\n\n [dataset] \n service_port = 65536\n\n # Don't forgot to regist Path object first and the typename will be the given name!!\n path = {'path':'/data/kitti'}@pyPath\n \n [dataset.loader]\n batch_size = 32\n\n [model]\n [model.backbone]\n mod_typ = 'resnet'\n [model.backbone.optimizer]\n lay_seed = $glb_seed\n \n [train_cfg]\n batch_size = $dataset.loader.batch_size\n [train_cfg.opt]\n opt_typ = 'Adam'\n lr = 1e-4\n sched = 'cos_anneal'\n '''\n\n # main_block \n if __name__ == \"__main__\":\n from pathlib import Path\n\n cfger = Configer(description=\"sample for arguments interpolation\")\n cfger.regist_cnvtor(\"pyPath\", Path) # regist customer class 'Path'\n\n cfg_str = get_str_cfg()\n cfger.cfg_from_str(cfg_str)\n # do whatever you want to do!\n \n#### **3. Access all arguments flexibly**\nWe simple set a breakpoint to feel how flexible does `easy_configer.utils.Container.AttributeDict` support.\n\n from easy_configer.Configer import Configer\n \n if __name__ == \"__main__\":\n cfger = Configer()\n cfger.cfg_from_ini(\"./hier_cfg.ini\")\n breakpoint()\n\n> We write a special example `hier_cfg.ini`!!\n # nested-dict\n [secA] # test depth ((sub^4)-section under secA)\n lev = 1\n [secA.secB]\n lev = 2\n [secA.secB.secC]\n lev = 3\n [secA.secB.secC.secD]\n lev = 4\n\nNow you can access each `lev` :\n1. `(pdb) cfger.secA.lev `, output `lev : 1`\n2. `(pdb) cfger['secA'].secB['lev'] `, output `lev : 2`, and so on..\n3. Most crazy one ~ `(pdb) cfger.secA.['secB'].secC['secD'].lev `, output `lev : 4`\n\n\n#### **4. Commmend-line Support**\n> We also take `hier_cfg.ini` as example!\n\n # hier_cfg.ini\n glb_var = 42@int\n [dataset] \n ds_type = None\n path = {'root':'/data/kitti'}@Path\n [dataset.loader]\n batch_size = 32@int\n\n # Hier-Cell cfg written by Josef-Huang..\n\nExecute python program and print out the helper information <br>\n`python quick_hier.py -h`\n\nUpdate flatten argument and print out the helper information <br>\n`python quick_hier.py glb_var=404 -h`\n\nEspecially update **non-flatten argument**, you can access any argument at any level by dot-access in commend-line!! (with combining any argument update). Now, try to change any nested argument <br>\n`python quick_hier.py dataset.ds_type=\"'kitti'\" dataset.path=\"{'path':'/root/ds'}\" dataset.loader.batch_size=48`\n\n( Note that the commendline declaration for string is tricky, but currently we only support two way for that : \n `dataset.ds_type=\"'kitti'\"` or `dataset.ds_type=kitti@str`, pick up one of you like ~ )\n\n#### **5. Import Sub-Config**\nLike `omegaconf`, most of user expect to seperate the config based on their type and dynamically merge it in runtime. It's a rational requirement and the previous version of easy-config provide two way to conduct it, but both have it's limit : \n1. you can call the `cfg_from_ini` twice, for example, `cfg.cfg_from_ini('./base_cfg') ; cfg.cfg_from_ini('./override_cfg')`. But it's not explicitly load the config thus reducing readability.\n2. you can use the config merging, for example, `new_cfg = base_cfg | override_cfg`. But it's not elegant solution while you have to merge several config..\n\n#### Now, we provide the thrid way : **sub-config**. you can import the sub-config in any depth of hierachical config by simply placing the `>` symbol at the beginning of line.\n # ./base_cfg.ini\n glb_seed = 42@int\n [dataset] \n > ./config/ds_config.ini\n \n [model]\n > ./root/config/model_config.ini\n \n # ./config/ds_config.ini\n ds_type = None\n path = {'root':'/data/kitti'}@Path\n [dataset.loader]\n batch_size = 32@int\n \n # ./root/config/model_config.ini\n [model.backbone]\n mod_typ = 'resnet'\n [model.backbone.optimizer]\n # and yes, interpolation is still valid \"after\" the reference argument is declared!\n lay_seed = $glb_seed \n\n\n#### **6. Config Operation**\nConfig operation is one of the core technique for dynamic configuration system!!\nIn the following example, you can see that the merging config system already provided a impressive hierachical merging funtionality! \n\n> For example, `ghyu.opop.add` in cfg_a can be replaced by the cfg_b in **same** section with the same variable name, while the different namespace keep their variable safely ~ so the value of `ghyu.opop.add` will be 67 and `ghyu.opop.tueo.inpo` refer the flatten variable `inpo` and the value will be 46.\n\n from easy_configer.Configer import Configer\n\n def build_cfg_text_a():\n return '''\n # Initial config file :\n inpo = 46@int\n [test] \n mrg_var_tst = [1, 3, 5]@list\n [test.ggap]\n gtgt = haha@str\n\n [ghyu]\n [ghyu.opop]\n add = 32@int\n [ghyu.opop.tueo]\n salt = $inpo\n\n # Cell cfg written by Josef-Huang..\n '''\n\n def build_cfg_text_b():\n return '''\n # Initial config file :\n inop = 32@int\n [test] \n mrg_var_tst = [1, 3, 5]@list\n [test.ggap]\n gtgt = overrides@str\n [test.ggap.conf]\n secert = 42@int\n\n [ghyu]\n [ghyu.opop]\n add = 67@int\n div = 1e-4@float\n\n [new]\n [new.new]\n newsec = wpeo@str\n # Cell cfg written by Josef-Huang..\n '''\n\n if __name__ == \"__main__\":\n cfg_a = Configer(cmd_args=True)\n cfg_a.cfg_from_str(build_cfg_text_a()) \n \n\n cfg_b = Configer()\n cfg_b.cfg_from_str(build_cfg_text_b())\n \n # default, override falg is turn off ~\n cfg_a.merge_conf(cfg_b, override=True)\n\n # `cfg_b = cfg_b | cfg_a`, operator support, warn to decrease the read-ability...\n # cfg_a will override the argument of cfg_b which share the identitical variable name in cfg_b!\n # operator support : `cfg_b |= cfg_a` == `cfg_b = cfg_b | cfg_a`\n\n---\n\n### **Miscellnous features**\n#### **7. IO Converter**\n from dataclasses import dataclass\n from typing import Optional\n\n @dataclass\n class TableConfig:\n rows: int = 1\n\n @dataclass\n class DatabaseConfig:\n table_cfg: TableConfig = TableConfig()\n\n @dataclass\n class ModelConfig:\n data_source: Optional[TableConfig] = None\n\n @dataclass\n class ServerConfig:\n db: DatabaseConfig = DatabaseConfig()\n model: ModelConfig = ModelConfig()\n\n if __name__ == '__main__':\n from easy_configer.IO_Converter import IO_Converter\n\n # first import the IO_converter\n from easy_config.IO_Converter import IO_Converter\n cnvt = IO_Converter()\n\n # convert easy_config instance into the argparse instance\n argp_cfg = cnvt.cnvt_cfg_to(cfger, 'argparse')\n\n uargp_cfg = cnvt.cnvt_cfg_to(cfger, 'argparse', parse_arg=False)\n argp_cfg = uargp_cfg.parse_args()\n\n ## convert config INTO..\n # convert easy_config instance into the omegaconf instance\n ome_cfg = cnvt.cnvt_cfg_to(cfger, 'omegaconf')\n\n # convert easy_config instance into the \"yaml string\"\n yaml_cfg = cnvt.cnvt_cfg_to(cfger, 'yaml')\n\n # convert easy_config instance into the \"dict\"\n yaml_cfg = cnvt.cnvt_cfg_to(cfger, 'dict')\n\n ## convert into easy-config FROM..\n # argparse, omegaconf, yaml, dict ... is supported\n ez_cfg = cnvt.cnvt_cfg_from(argp_cfg, 'omegaconf')\n\n # Especially, it support \"dataclass\"!\n ds_cfg = ServerConfig()\n ez_cfg = cnvt.cnvt_cfg_from(ds_cfg, 'dataclass')\n\n\n#### **8. Absl style flag**\n> easy_config also support that you can access the 'same' config file in different python file without re-declare the config. test_flag.py under the same work directory\n\nSuppose you have executed `main.py`:\n from easy_configer.Configer import Configer\n from utils import get_var_from_flag\n\n if __name__ == \"__main__\":\n cfg = Configer()\n cfg.cfg_from_str(\"var = 32\")\n\n # both should output 32 ~\n print(f\"var from main : {cfg.var}\")\n print(f\"var from flag : { get_var_from_flag() }\")\n\nNow, when you step in `get_var_from_flag` function in different file..\n from easy_configer.Configer import Configer\n\n def get_var_from_flag():\n new_cfger = Configer()\n flag = new_cfger.get_cfg_flag()\n # test to get the pre-defined 'var'\n return flag.var\n\n---\n\n#### **The documentation of easy_configer is also released in read doc** [\ud83d\udd17](https://easy-configer.readthedocs.io/en/latest/)\n\n---\n\n### Simple Unittest \ud83e\uddea\nIf you clone this repo and built from source, you can try to run the unittest.\n`python -m unittest discover`\n> I have placed all test file under test folder.\n\n---\n\n### License\nMIT License. More information of each term, please see LICENSE.md\n\n### Author \nJosef-Huang, a3285556aa@gmail.com \n\n### Footer\n~ Hope God bless everyone in the world to know his word ~ <br>\n**The fear of the LORD is the beginning of knowledge; fools despise wisdom and instruction. by Proverbs 1:7**\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An easy way for configurating python program by the given config file or config str",
"version": "2.3.2",
"project_urls": {
"Homepage": "https://github.com/HuangChiEn/easy_config"
},
"split_keywords": [
"configuration",
" commendline argument",
" argument"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8ed376eed576cf48c5a89020b8dfbd474c74a456a82598171be48abac89b422a",
"md5": "b2f394ab63b19664c3a0540dda97582c",
"sha256": "ae8f667b53ea1e2898306cb9e9bd7098393424e25e32d1136228507be62c6451"
},
"downloads": -1,
"filename": "easy_configer-2.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b2f394ab63b19664c3a0540dda97582c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 17975,
"upload_time": "2024-04-04T14:37:22",
"upload_time_iso_8601": "2024-04-04T14:37:22.488020Z",
"url": "https://files.pythonhosted.org/packages/8e/d3/76eed576cf48c5a89020b8dfbd474c74a456a82598171be48abac89b422a/easy_configer-2.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d09a58f10f02fc0c8e0e7f7bc03cb01ad35a8842b4c430db7e58f16f70d0bf59",
"md5": "a710e3b82921cf2db1a6a24b41127f29",
"sha256": "a662b5600bfa612e8b568e13fbb5db2b76a35c4ef32940ec150099769b652dda"
},
"downloads": -1,
"filename": "easy_configer-2.3.2.tar.gz",
"has_sig": false,
"md5_digest": "a710e3b82921cf2db1a6a24b41127f29",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 23743,
"upload_time": "2024-04-04T14:37:26",
"upload_time_iso_8601": "2024-04-04T14:37:26.603725Z",
"url": "https://files.pythonhosted.org/packages/d0/9a/58f10f02fc0c8e0e7f7bc03cb01ad35a8842b4c430db7e58f16f70d0bf59/easy_configer-2.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-04 14:37:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "HuangChiEn",
"github_project": "easy_config",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "easy-configer"
}
JSON
