# Inlog
Inlog is your solution to managing and tracking results from your Python programs. It simplifies the process of logging parameters and results, making your work more reproducible and organized.
## Motivation
Imagine, you have a python program with some parameters. You make a few experiments with it, producing some results. Traditionally, people write the parameters in the filename:
```txt
Result1_day2_3deg_4km.txt
```
There are a few problems with this:
- The filenames get very long with more parameters.
- The list of parameters is often incomplete.
- The list of parameters is not machine readable.
Things are usually fine, until you try to reproduce your results a few months later...
`inlog` ("Input Logger") addresses those problems by creating a log file with all parameters next to your results.
All you need to do is to put your parameters into a separate file (like YAML or INI format) and with just three additional lines in your python code, inlog will create a comprehensive logfile:
```python
import inlog
config=inlog.load_yaml('Params.yaml', '1.0')
#do your stuff, using the parameters in config and save results into 'Results1.txt'
#...
config.write_log('Results1.txt') #this creates a file 'Results1.log' in the same folder as 'Results1.txt'
```
But there is more: `inlog` stores additional information like the date, runtime or a [hash of your data](https://github.com/Ockenfuss/inlog#hashes), it can keep track of [complex histories](https://github.com/Ockenfuss/inlog#writing-logs) of your results and it can even visualize them as a [flowchart](https://github.com/Ockenfuss/inlog#flowchart)!
That`s how, on the long term, inlog helps you to remember:
- Where did my results come from?
- What do the results depend on?
- What programs did I execute to get the results?
To see a more comprehensive example, look at the [examples folder](https://github.com/Ockenfuss/inlog/tree/master/examples)
## Installation
The basic version of `inlog` only depends on the python standard library.
```
python3 -m pip install inlog
```
If you want YAML support, install the `extras` version of `inlog`:
```
python3 -m pip install inlog[extras]
```
## Usage
### Input Parameters
It is not necessary to read input parameters from a separate file. However, in most cases, this is desirable. Inlog supports currently the 'ini', 'json' and '.yaml' file formats.
#### Ini Format
Parsing is done using the 'configparser' module from the python standard library. ini files consist of sections with options. All options are treated as strings. Ini files allow for value interpolation, see the [manual](https://docs.python.org/3/library/configparser.html#) for further information.
Example of an .ini file:
```ini
[Filepaths_General]
home=/home
bin=/bin
[Filepaths_User]
documents=~/Documents
```
```python
import inlog
config=inlog.load_ini('config.ini',version='1.0')
```
#### JSON Format
Example of a .json file:
```json
{
"Filepaths": {
"Common": {
"home": "/home",
"bin": "/bin"
},
"User": {
"documents": "~/Documents"
}
}
}
```
```python
import inlog
config=inlog.load_json('config.json',version='1.0')
```
#### YAML Format
Parsing YAML files requires the `pyyaml` library.
Example of a .yaml file:
```ini
Filepaths:
Common:
home: /home
bin: /bin
User:
documents: ~/Documents
```
```python
import inlog
config=inlog.load_yaml('config.yaml',version='1.0')
```
#### Dictionary
You can also pass a dictionary directly to the Logger Class:
```python
import inlog
dictionary={'Filepaths': {'Common': {'home': '/home','bin': '/bin'},'User': {'documents': '~/Documents'}}}
config=inlog.Logger(dictionary,version='1.0')
```
### Accessing Parameters
`inlog` stores all parameters in a tree data structure. You can access them using the `.get()` method. This allows `inlog` to keep track which options you used in your program and to write only those in the log file. Take the yaml example from above:
```python
import inlog
config=inlog.load_yaml('config.yaml',version='1.0')
config.get('Filepaths', 'User', 'documents')
```
Similarly, use `.set()` to set a value to a parameter:
```python
config.set('~/MyDocs', 'Filepaths', 'User', 'documents')
```
You can change all the parameters in a subtree using `set_subtree(subtree_dict)`, using a nested dictionary as argument.
```python
config.set_subtree({'documents':'~/MyDocs', 'pictures':'~/MyPictures'}, 'Filepaths', 'User')
```
As a shortcut for accessing parameters, you can use bracket notation []:
```python
config['Filepaths', 'User', 'documents']
```
In this case, `inlog` actually performs a depth-first search in the config tree, so with brackets, you can go even shorter. All of the following commands yield the same result:
```python
config['Filepaths', 'User', 'documents']
config['User', 'documents']
config['documents']
```
### Type conversion
For the `.ini` file format, all options are treated as strings. `inlog` provides the `convert_type` and `convert_array` functions as shortcuts for type conversions:
```python
config.convert_type(int, 'option1')
config.convert_type(pathlib.Path, 'option1') #you can provide an arbitrary conversion function
config.convert_array(int, 'option1', removeSpaces=True, sep=",") #this will split the string and convert the elements, returning a list.
```
### Hashes
You can provide `inlog` the name of the result files your program produced. In this case, `inlog` will store SHA256 hash values of your results in the log.
Therefore, you can later verify that your results truly belong to the parameters given in the log. Use `set_outfile()` and `add_outfile` to set or append the list of filenames.
```python
config.set_outfile('Results1.txt')
config.add_outfile('Results2.txt')
```
### Writing Logs
In order to write a log, you need to specify the file path (or multiple paths) of the new log. Optionally, you can specify existing log files, which will be included in the new log as dependencies. There are two different formats for logs: txt and json.
By default, `write_log()` will append the filename of all given filenames with `.log`.
```python
config.write_log('Results1.txt', old_logs=['Dependency1.txt'])
```
#### JSON Format
A json file. This format is the recommended default, since it allows to capture the tree-like structure of dependencies.
Example:
```json
{
"date": "2023-10-08 20:59:51.019125",
"program": "inlog/examples/Script2.py",
"version": "1.0",
"input": "Config2.ini",
"runtime": "0:00:00.006449",
"options": {
"section1": {
"factor": 2,
"intermediate": "intermediateResult.dat",
"result": "FinalResult.dat"
}
},
"output_files": [
{
"path": "inlog/examples/FinalResult.dat",
"hash": "6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4b"
}
],
"dependencies": {
"inlog/examples/intermediateResult.dat.log": {
"date": "2023-10-08 20:59:49.026310",
"program": "inlog/examples/Script1.py",
"version": "1.0",
"input": "Config1.ini",
"runtime": "0:00:00.005438",
"options": {
"section1": {
"start": 1,
"stop": 10,
"increment": 2,
"intermediate": "intermediateResult.dat"
}
},
"output_files": [
{
"path": "inlog/examples/intermediateResult.dat",
"hash": "22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721"
}
],
"dependencies": {}
}
}
}
```
#### Text Format
A linear text file, where dependencies are listed first and the new log information is appended at the end of the file. This format is straightforward and easy to read, but gets messy if you have multiple (sub-)dependencies. You can execute such a log as a bash-script to reproduce the data.
Example:
```txt
cd inlog/examples
python3 Script1.py Config1.ini
# <Date> 2023-10-08 20:57:47.744149
# <Program> inlog/examples/Script1.py
# <Version> 1.0
# <Input> Config1.ini
# <Runtime> 0:00:00.007015
#**************************
#{
# "section1": {
# "start": 1,
# "stop": 10,
# "increment": 2,
# "intermediate": "intermediateResult.dat"
# }
#}
#**************************
#Output files created:
# <PATH> inlog/examples/intermediateResult.dat
# <HASH> 22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721
# <Logfile> intermediateResult.dat.log_txt
#=========================================
cd inlog/examples
python3 Script2.py Config2.ini
# <Date> 2023-10-08 20:57:54.775511
# <Program> inlog/examples/Script2.py
# <Version> 1.0
# <Input> Config2.ini
# <Runtime> 0:00:00.007339
#**************************
#{
# "section1": {
# "factor": 2,
# "intermediate": "intermediateResult.dat",
# "result": "FinalResult.dat"
# }
#}
#**************************
#Output files created:
# <PATH> inlog/examples/FinalResult.dat
# <HASH> 6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4b
```
## Visualization
Printing the logger object will yield a text version of the current log. Equivalently, you can call `show_data()`.
```python
print(config)
config.show_data()
```
### Flowchart
Calling `inlog-flowchart logfile.dat.log` will convert a log "logfile.dat.log" in json format to a Mermaid flowchart. Mermaid is a charting application, based on a markdown like syntax. You can paste the output of flowchart.py in the [Mermaid Live Editor](https://mermaid.live/) to obtain a flowchart.
Just call:
```bash
inlog/flowchart.py examples/FinalResult.log
```
This willl yield the following output:
```txt
flowchart TD
id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]
id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]
id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]
id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa
id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3
```
Pasting into the [Mermaid Live Editor](https://mermaid.live/) will give the following chart:
```mermaid
flowchart TD
id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]
id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]
id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]
id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa
id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3
```
Raw data
{
"_id": null,
"home_page": "",
"name": "inlog",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "config,ini,log,yaml",
"author": "",
"author_email": "Paul Ockenfu\u00df <paul.ockenfuss@physik.uni-muenchen.de>",
"download_url": "https://files.pythonhosted.org/packages/eb/69/57052af2577c3fd84c8c5643f0bc8f659a280852ffc022b37f63e80cfd75/inlog-2.2.1.tar.gz",
"platform": null,
"description": "# Inlog\n\nInlog is your solution to managing and tracking results from your Python programs. It simplifies the process of logging parameters and results, making your work more reproducible and organized.\n\n## Motivation\nImagine, you have a python program with some parameters. You make a few experiments with it, producing some results. Traditionally, people write the parameters in the filename:\n```txt\nResult1_day2_3deg_4km.txt\n```\nThere are a few problems with this:\n\n- The filenames get very long with more parameters.\n- The list of parameters is often incomplete.\n- The list of parameters is not machine readable.\n\nThings are usually fine, until you try to reproduce your results a few months later...\n\n`inlog` (\"Input Logger\") addresses those problems by creating a log file with all parameters next to your results.\nAll you need to do is to put your parameters into a separate file (like YAML or INI format) and with just three additional lines in your python code, inlog will create a comprehensive logfile:\n```python\nimport inlog\nconfig=inlog.load_yaml('Params.yaml', '1.0')\n#do your stuff, using the parameters in config and save results into 'Results1.txt'\n#...\nconfig.write_log('Results1.txt') #this creates a file 'Results1.log' in the same folder as 'Results1.txt'\n```\n\nBut there is more: `inlog` stores additional information like the date, runtime or a [hash of your data](https://github.com/Ockenfuss/inlog#hashes), it can keep track of [complex histories](https://github.com/Ockenfuss/inlog#writing-logs) of your results and it can even visualize them as a [flowchart](https://github.com/Ockenfuss/inlog#flowchart)!\nThat`s how, on the long term, inlog helps you to remember:\n\n- Where did my results come from?\n- What do the results depend on?\n- What programs did I execute to get the results?\n\nTo see a more comprehensive example, look at the [examples folder](https://github.com/Ockenfuss/inlog/tree/master/examples)\n\n## Installation\nThe basic version of `inlog` only depends on the python standard library.\n```\npython3 -m pip install inlog\n```\nIf you want YAML support, install the `extras` version of `inlog`:\n```\npython3 -m pip install inlog[extras]\n```\n\n## Usage\n\n### Input Parameters\nIt is not necessary to read input parameters from a separate file. However, in most cases, this is desirable. Inlog supports currently the 'ini', 'json' and '.yaml' file formats. \n\n#### Ini Format\nParsing is done using the 'configparser' module from the python standard library. ini files consist of sections with options. All options are treated as strings. Ini files allow for value interpolation, see the [manual](https://docs.python.org/3/library/configparser.html#) for further information.\nExample of an .ini file:\n```ini\n[Filepaths_General]\nhome=/home\nbin=/bin\n[Filepaths_User]\ndocuments=~/Documents\n```\n\n```python\nimport inlog\nconfig=inlog.load_ini('config.ini',version='1.0')\n```\n\n#### JSON Format\nExample of a .json file:\n```json\n{\n \"Filepaths\": {\n \"Common\": {\n \"home\": \"/home\",\n \"bin\": \"/bin\"\n },\n \"User\": {\n \"documents\": \"~/Documents\"\n }\n }\n}\n```\n\n```python\nimport inlog\nconfig=inlog.load_json('config.json',version='1.0')\n```\n\n#### YAML Format\nParsing YAML files requires the `pyyaml` library.\nExample of a .yaml file:\n```ini\nFilepaths:\n Common:\n home: /home\n bin: /bin\n User:\n documents: ~/Documents\n```\n\n```python\nimport inlog\nconfig=inlog.load_yaml('config.yaml',version='1.0')\n```\n\n#### Dictionary\nYou can also pass a dictionary directly to the Logger Class:\n```python\nimport inlog\ndictionary={'Filepaths': {'Common': {'home': '/home','bin': '/bin'},'User': {'documents': '~/Documents'}}}\nconfig=inlog.Logger(dictionary,version='1.0')\n```\n\n\n### Accessing Parameters\n`inlog` stores all parameters in a tree data structure. You can access them using the `.get()` method. This allows `inlog` to keep track which options you used in your program and to write only those in the log file. Take the yaml example from above:\n```python\nimport inlog\nconfig=inlog.load_yaml('config.yaml',version='1.0')\nconfig.get('Filepaths', 'User', 'documents')\n```\nSimilarly, use `.set()` to set a value to a parameter:\n```python\nconfig.set('~/MyDocs', 'Filepaths', 'User', 'documents')\n```\nYou can change all the parameters in a subtree using `set_subtree(subtree_dict)`, using a nested dictionary as argument.\n```python\nconfig.set_subtree({'documents':'~/MyDocs', 'pictures':'~/MyPictures'}, 'Filepaths', 'User')\n```\n\n\nAs a shortcut for accessing parameters, you can use bracket notation []:\n```python\nconfig['Filepaths', 'User', 'documents']\n```\nIn this case, `inlog` actually performs a depth-first search in the config tree, so with brackets, you can go even shorter. All of the following commands yield the same result:\n```python\nconfig['Filepaths', 'User', 'documents']\nconfig['User', 'documents']\nconfig['documents']\n```\n\n\n### Type conversion\nFor the `.ini` file format, all options are treated as strings. `inlog` provides the `convert_type` and `convert_array` functions as shortcuts for type conversions:\n```python\nconfig.convert_type(int, 'option1')\nconfig.convert_type(pathlib.Path, 'option1') #you can provide an arbitrary conversion function\nconfig.convert_array(int, 'option1', removeSpaces=True, sep=\",\") #this will split the string and convert the elements, returning a list.\n```\n\n### Hashes\nYou can provide `inlog` the name of the result files your program produced. In this case, `inlog` will store SHA256 hash values of your results in the log.\nTherefore, you can later verify that your results truly belong to the parameters given in the log. Use `set_outfile()` and `add_outfile` to set or append the list of filenames.\n```python\nconfig.set_outfile('Results1.txt')\nconfig.add_outfile('Results2.txt')\n```\n\n### Writing Logs\nIn order to write a log, you need to specify the file path (or multiple paths) of the new log. Optionally, you can specify existing log files, which will be included in the new log as dependencies. There are two different formats for logs: txt and json.\nBy default, `write_log()` will append the filename of all given filenames with `.log`.\n```python\nconfig.write_log('Results1.txt', old_logs=['Dependency1.txt'])\n```\n\n#### JSON Format\nA json file. This format is the recommended default, since it allows to capture the tree-like structure of dependencies.\nExample:\n```json\n{\n \"date\": \"2023-10-08 20:59:51.019125\",\n \"program\": \"inlog/examples/Script2.py\",\n \"version\": \"1.0\",\n \"input\": \"Config2.ini\",\n \"runtime\": \"0:00:00.006449\",\n \"options\": {\n \"section1\": {\n \"factor\": 2,\n \"intermediate\": \"intermediateResult.dat\",\n \"result\": \"FinalResult.dat\"\n }\n },\n \"output_files\": [\n {\n \"path\": \"inlog/examples/FinalResult.dat\",\n \"hash\": \"6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4b\"\n }\n ],\n \"dependencies\": {\n \"inlog/examples/intermediateResult.dat.log\": {\n \"date\": \"2023-10-08 20:59:49.026310\",\n \"program\": \"inlog/examples/Script1.py\",\n \"version\": \"1.0\",\n \"input\": \"Config1.ini\",\n \"runtime\": \"0:00:00.005438\",\n \"options\": {\n \"section1\": {\n \"start\": 1,\n \"stop\": 10,\n \"increment\": 2,\n \"intermediate\": \"intermediateResult.dat\"\n }\n },\n \"output_files\": [\n {\n \"path\": \"inlog/examples/intermediateResult.dat\",\n \"hash\": \"22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721\"\n }\n ],\n \"dependencies\": {}\n }\n }\n}\n```\n\n#### Text Format\nA linear text file, where dependencies are listed first and the new log information is appended at the end of the file. This format is straightforward and easy to read, but gets messy if you have multiple (sub-)dependencies. You can execute such a log as a bash-script to reproduce the data.\nExample:\n```txt\ncd inlog/examples\npython3 Script1.py Config1.ini\n# <Date> 2023-10-08 20:57:47.744149\n# <Program> inlog/examples/Script1.py\n# <Version> 1.0\n# <Input> Config1.ini\n# <Runtime> 0:00:00.007015\n#**************************\n#{\n# \"section1\": {\n# \"start\": 1,\n# \"stop\": 10,\n# \"increment\": 2,\n# \"intermediate\": \"intermediateResult.dat\"\n# }\n#}\n#**************************\n#Output files created:\n# <PATH> inlog/examples/intermediateResult.dat\n# <HASH> 22a23bcb0798a2b67902a51faad1d04fca6489abdc7c3f1ced983ac22658a721\n# <Logfile> intermediateResult.dat.log_txt\n#=========================================\ncd inlog/examples\npython3 Script2.py Config2.ini\n# <Date> 2023-10-08 20:57:54.775511\n# <Program> inlog/examples/Script2.py\n# <Version> 1.0\n# <Input> Config2.ini\n# <Runtime> 0:00:00.007339\n#**************************\n#{\n# \"section1\": {\n# \"factor\": 2,\n# \"intermediate\": \"intermediateResult.dat\",\n# \"result\": \"FinalResult.dat\"\n# }\n#}\n#**************************\n#Output files created:\n# <PATH> inlog/examples/FinalResult.dat\n# <HASH> 6542c8602f59c351652e382f0448b2caba8c6404a133fca7b137ccd679bd7f4b\n```\n\n\n## Visualization\nPrinting the logger object will yield a text version of the current log. Equivalently, you can call `show_data()`.\n```python\nprint(config)\nconfig.show_data()\n```\n### Flowchart\nCalling `inlog-flowchart logfile.dat.log` will convert a log \"logfile.dat.log\" in json format to a Mermaid flowchart. Mermaid is a charting application, based on a markdown like syntax. You can paste the output of flowchart.py in the [Mermaid Live Editor](https://mermaid.live/) to obtain a flowchart.\nJust call:\n```bash\ninlog/flowchart.py examples/FinalResult.log\n```\nThis willl yield the following output:\n```txt\nflowchart TD\n id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]\n id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]\n id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]\n id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa\n id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3\n\n```\nPasting into the [Mermaid Live Editor](https://mermaid.live/) will give the following chart:\n```mermaid\nflowchart TD\n id_435d36b4a8d3c591b456a027fe49efa3[FinalResult.log]\n id_89f9132e25d1cada0f37669c8e3a2faa[intermediateResult.log]\n id_f1add11e031d031e9f8694f5591c1fc2[No Dependencies]\n id_f1add11e031d031e9f8694f5591c1fc2 --> |Script1.py| id_89f9132e25d1cada0f37669c8e3a2faa\n id_89f9132e25d1cada0f37669c8e3a2faa --> |Script2.py| id_435d36b4a8d3c591b456a027fe49efa3\n```\n\n",
"bugtrack_url": null,
"license": "inlog \u00a9 2023 by Paul Ockenfu\u00df is licensed under CC BY-SA 4.0. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/",
"summary": "Keep track of your data! Reading config files and creating comprehensive logs for your results.",
"version": "2.2.1",
"project_urls": {
"Homepage": "https://github.com/Ockenfuss/inlog"
},
"split_keywords": [
"config",
"ini",
"log",
"yaml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4f3b9068bce8cb509bbc650ef8e4fb33907d79053329631f29724763d672928d",
"md5": "82220b14828726184ab2b5c30200eb68",
"sha256": "8dbcaa83f2f31ca4e69faafb3c217cac5df6daff3887d78d088d609463bcc511"
},
"downloads": -1,
"filename": "inlog-2.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "82220b14828726184ab2b5c30200eb68",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13902,
"upload_time": "2024-02-12T10:34:59",
"upload_time_iso_8601": "2024-02-12T10:34:59.291237Z",
"url": "https://files.pythonhosted.org/packages/4f/3b/9068bce8cb509bbc650ef8e4fb33907d79053329631f29724763d672928d/inlog-2.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "eb6957052af2577c3fd84c8c5643f0bc8f659a280852ffc022b37f63e80cfd75",
"md5": "f634e126c2955f44907df81f0ef349b5",
"sha256": "9a65c73cae11dc790e8a76c2c6aca43ccd42927b415926a7bd95dbbe989d9c81"
},
"downloads": -1,
"filename": "inlog-2.2.1.tar.gz",
"has_sig": false,
"md5_digest": "f634e126c2955f44907df81f0ef349b5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 16345,
"upload_time": "2024-02-12T10:35:00",
"upload_time_iso_8601": "2024-02-12T10:35:00.819497Z",
"url": "https://files.pythonhosted.org/packages/eb/69/57052af2577c3fd84c8c5643f0bc8f659a280852ffc022b37f63e80cfd75/inlog-2.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-12 10:35:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Ockenfuss",
"github_project": "inlog",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "inlog"
}
JSON
