# Treeverse
Treeverse is a Python tool for traversing, processing, and accumulating data from file trees using custom code.
## Installation
```bash
pip install treeverse
```
## Usage
Treeverse operates in three main steps:
1. **Build**: Traverse the directory and build the file tree
2. **Process**: Apply custom processing to each node
3. **Accumulate**: Aggregate data from child nodes to parent nodes
### 1. Build the tree
```bash
treeverse build -p /path/to/your/project -e py -e yaml \
-c your_module:your_filter_function -o tree.yaml
```
Options:
- `-p, --path`: Specify the path to traverse (default is current directory)
- `-e, --extensions`: Specify file extensions to include (can be used multiple times)
- `-c, --callback`: Path to filter callback functions (can be used multiple times)
- `-o, --output`: Output file (default: tree.yaml)
- `--write-to-stream`: Write output to stdout instead of a file
### 2. Process the tree
```bash
treeverse process -i tree.yaml -c your_module:your_processing_function -o processed_tree.yaml
```
Options:
- `-i, --input`: Input file (default: tree.yaml)
- `-c, --callback`: Path to the processing function
- `-o, --output`: Output file (default: processed_tree.yaml)
- `--read-from-stream`: Read input from stdin instead of a file
- `--write-to-stream`: Write output to stdout instead of a file
### 3. Accumulate results
```bash
treeverse accumulate -i processed_tree.yaml -c your_module:your_accumulation_function -o final_result.yaml
```
Options:
- `-i, --input`: Input file (default: processed_tree.yaml)
- `-c, --callback`: Path to the accumulation function
- `-o, --output`: Output file (default: accumulated_tree.yaml)
- `--read-from-stream`: Read input from stdin instead of a file
- `--write-to-stream`: Write output to stdout instead of a file
## Example Use Case
Process all Python and YAML files in a project, analyze them, and accumulate results:
```bash
treeverse build -p ~/path/to/your/project -e py -e yaml -o tree.yaml
treeverse process -i tree.yaml -c ~/path/to/your/tools.py:analyze_file -o processed_tree.yaml
treeverse accumulate -i processed_tree.yaml -c ~/path/to/your/tools.py:accumulate_results -o final_analysis.yaml
```
To use streaming for the entire pipeline:
```bash
treeverse build -p ~/path/to/your/project -e py -e yaml --write-to-stream | \
treeverse process -c ~/path/to/your/tools.py:analyze_file --read-from-stream --write-to-stream | \
treeverse accumulate -c ~/path/to/your/tools.py:accumulate_results --read-from-stream > final_analysis.yaml
```
In this example, `tools.py` would contain custom `analyze_file` and `accumulate_results` functions.
## Reference Implementations
For examples of callable functions that can be used with Treeverse, check the `simple_callbacks.py` file in the project repository. This file contains reference implementations for filtering, processing, and accumulation functions.
Raw data
{
"_id": null,
"home_page": "https://github.com/ryzhakar/treeverse",
"name": "treeverse",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "file-tree, traversal, processing, filtering, cli",
"author": "Arthur Ryzhak",
"author_email": "ryzhakar@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f9/9a/b0844fa5ddbf3d3ff0ba49b945e48ba7b31f08fc857ef6d1f70dca577028/treeverse-0.2.1.tar.gz",
"platform": null,
"description": "# Treeverse\n\nTreeverse is a Python tool for traversing, processing, and accumulating data from file trees using custom code.\n\n## Installation\n\n```bash\npip install treeverse\n```\n\n## Usage\n\nTreeverse operates in three main steps:\n\n1. **Build**: Traverse the directory and build the file tree\n2. **Process**: Apply custom processing to each node\n3. **Accumulate**: Aggregate data from child nodes to parent nodes\n\n### 1. Build the tree\n\n```bash\ntreeverse build -p /path/to/your/project -e py -e yaml \\\n -c your_module:your_filter_function -o tree.yaml\n```\n\nOptions:\n- `-p, --path`: Specify the path to traverse (default is current directory)\n- `-e, --extensions`: Specify file extensions to include (can be used multiple times)\n- `-c, --callback`: Path to filter callback functions (can be used multiple times)\n- `-o, --output`: Output file (default: tree.yaml)\n- `--write-to-stream`: Write output to stdout instead of a file\n\n### 2. Process the tree\n\n```bash\ntreeverse process -i tree.yaml -c your_module:your_processing_function -o processed_tree.yaml\n```\n\nOptions:\n- `-i, --input`: Input file (default: tree.yaml)\n- `-c, --callback`: Path to the processing function\n- `-o, --output`: Output file (default: processed_tree.yaml)\n- `--read-from-stream`: Read input from stdin instead of a file\n- `--write-to-stream`: Write output to stdout instead of a file\n\n### 3. Accumulate results\n\n```bash\ntreeverse accumulate -i processed_tree.yaml -c your_module:your_accumulation_function -o final_result.yaml\n```\n\nOptions:\n- `-i, --input`: Input file (default: processed_tree.yaml)\n- `-c, --callback`: Path to the accumulation function\n- `-o, --output`: Output file (default: accumulated_tree.yaml)\n- `--read-from-stream`: Read input from stdin instead of a file\n- `--write-to-stream`: Write output to stdout instead of a file\n\n## Example Use Case\n\nProcess all Python and YAML files in a project, analyze them, and accumulate results:\n\n```bash\ntreeverse build -p ~/path/to/your/project -e py -e yaml -o tree.yaml\ntreeverse process -i tree.yaml -c ~/path/to/your/tools.py:analyze_file -o processed_tree.yaml\ntreeverse accumulate -i processed_tree.yaml -c ~/path/to/your/tools.py:accumulate_results -o final_analysis.yaml\n```\n\nTo use streaming for the entire pipeline:\n\n```bash\ntreeverse build -p ~/path/to/your/project -e py -e yaml --write-to-stream | \\\ntreeverse process -c ~/path/to/your/tools.py:analyze_file --read-from-stream --write-to-stream | \\\ntreeverse accumulate -c ~/path/to/your/tools.py:accumulate_results --read-from-stream > final_analysis.yaml\n```\n\nIn this example, `tools.py` would contain custom `analyze_file` and `accumulate_results` functions.\n\n## Reference Implementations\n\nFor examples of callable functions that can be used with Treeverse, check the `simple_callbacks.py` file in the project repository. This file contains reference implementations for filtering, processing, and accumulation functions.\n",
"bugtrack_url": null,
"license": "AGPL-3.0-or-later",
"summary": "A flexible tool for traversing, filtering, and processing file trees",
"version": "0.2.1",
"project_urls": {
"Documentation": "https://github.com/ryzhakar/treeverse#readme",
"Homepage": "https://github.com/ryzhakar/treeverse",
"Repository": "https://github.com/ryzhakar/treeverse"
},
"split_keywords": [
"file-tree",
" traversal",
" processing",
" filtering",
" cli"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6280f9c65b0e57ef4133369f32c8b04eadb59070d2a5cae5285e6f52bc8dd492",
"md5": "97f55c88a47bbdaa098001f2c3e63016",
"sha256": "b66ba9a3cdb6b81dc58e9115db0fd87e0d81ba1233c1de6e71743f257a5231d5"
},
"downloads": -1,
"filename": "treeverse-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "97f55c88a47bbdaa098001f2c3e63016",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 19158,
"upload_time": "2024-07-03T19:40:41",
"upload_time_iso_8601": "2024-07-03T19:40:41.748805Z",
"url": "https://files.pythonhosted.org/packages/62/80/f9c65b0e57ef4133369f32c8b04eadb59070d2a5cae5285e6f52bc8dd492/treeverse-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f99ab0844fa5ddbf3d3ff0ba49b945e48ba7b31f08fc857ef6d1f70dca577028",
"md5": "cf91c7907254c1ca9febe4992d0c5085",
"sha256": "e228608b35f26c2ddd3a541563cb97b5c98f6d7922c33c519b84b32ed13cca20"
},
"downloads": -1,
"filename": "treeverse-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "cf91c7907254c1ca9febe4992d0c5085",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 17638,
"upload_time": "2024-07-03T19:40:42",
"upload_time_iso_8601": "2024-07-03T19:40:42.937231Z",
"url": "https://files.pythonhosted.org/packages/f9/9a/b0844fa5ddbf3d3ff0ba49b945e48ba7b31f08fc857ef6d1f70dca577028/treeverse-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-03 19:40:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ryzhakar",
"github_project": "treeverse",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "treeverse"
}
JSON
