| Name | acnestis JSON |
| Version |
0.1.1
JSON |
| download |
| home_page | https://github.com/oduvan/acnestis |
| Summary | data collection, aggregation and convertion tool |
| upload_time | 2023-06-21 18:51:20 |
| maintainer | |
| docs_url | None |
| author | oduvan |
| requires_python | >3.8 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
[](https://pypi.python.org/pypi/acnestis/)
[](https://pypi.python.org/pypi/acnestis/)
[](https://github.com/psf/black)
## Acnestis - collect, aggregate and convert
A very simple tool that allows you collect data from different sources, change it ia different ways
## Install
```bash
$ pip install acnestis
```
## You can do in declarative way
* convert data in the current folder
* aggregate data from different sources
* inheritence - when you collect data from different source - you can replace any file and those files will be used in the child repo
## Ok, this is confusing. Show me some example
first of all, you can find some examples in the [tests/data](https://github.com/oduvan/acnestis/tree/master/tests/data) folder
*(source folders, or from-folders, may contain both yaml and py files, that doesn't mean both are required. We just make alternatives for testing and illustration)*
## Acnestis decloration
in order to declarate folder as acnestis folder you should do one of the following
* create `.acnestis.yaml` file in that folder
* create `.acnestis.py` file in that folder. Py-file has an identical functionality, but some more power (of course)
* create both `.asnestis.yaml` and `.acnestis.py` - in that case yaml-file will be used as the main one, but can use declared global variables from the py-file
* declarate some of the sub-folders (even non-existed), in the current acnestis-decloration
In order to process acnestis folders - you should do
```bash
$ acnesting process from/folder to/folder
```
if no acnestis decloration in the `from/folder` it will just copy files to `to/folder`. if `from/folder` has at least one acnestis decloration - it will be used for processing that folder, the rest will be copied
### Simple convertion
*From: [tests/data/002_concat_poem](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem/) to: [tests/data/002_concat_poem_result](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem_result)*
this is a very simple example, where all files from folder poem.txt will be connected into a single file
* we declarate [poem.txt](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem/poem.txt) folder as acnestis folder
Py version:
```python
from acnestis.processing import Processor
from acnestis.steps import concat_files
PROCESSOR = Processor([concat_files("poem.txt")], as_file="poem.txt")
```
YAML version:
```yaml
steps:
- name: concat_files
into_file: poem.txt
as_file: poem.txt
```
It has one single step `concat_files` with attribute `into_file: poem.txt` - so it simply concat all the files into one
Also processing has attribute `as_file` means the folder will be replaced with one file `poem.txt`
### Docker for more complex convertion
*From [tests/data/006_docker_svgo](https://github.com/oduvan/acnestis/tree/master/tests/data/006_docker_svgo) to: [tests/data/006_docker_svgo_result](https://github.com/oduvan/acnestis/tree/master/tests/data/006_docker_svgo_result)
you can build own convertion tools using docker
Py version:
```python
from acnestis.processing import Processor
from acnestis.steps import docker
PROCESSOR = Processor(
[docker("acnestis_svgo", skip_pull=True)],
)
```
YAML version:
```yaml
steps:
- name: docker
image: acnestis_svgo
skip_pull: true
```
one single step of using docker image `acnestis_svgo` for processing acnestis folder.
[Dockerfile of acnestis_svgo](https://github.com/oduvan/acnestis/tree/master/tests/docker_svgo/Dockerfile) is very simple:
```Dockerfile
# Base image
FROM node:20-alpine
# Install SVGO
RUN npm install -g svgo
# Execute the SVGO command when the container starts
ENTRYPOINT ["svgo", "-f", "/data_input", "-o", "/data_output"]
```
### you can use simple python-code for convertion
From [tests/data/004_code](https://github.com/oduvan/acnestis/tree/master/tests/data/004_code) to [tests/data/004_code_result](https://github.com/oduvan/acnestis/tree/master/tests/data/004_code_result)
## Now let's play with aggregation.
we want not only convert data, but first collect it from different sources
For the testing and examples we will use [oduvan/acnestis-test-repo](https://github.com/oduvan/acnestis-test-repo.git) github repository and its branches.
### simple use of git-repo
From: [data/007_git](https://github.com/oduvan/acnestis/tree/master/tests/data/007_git) to [data/007_git_result](https://github.com/oduvan/acnestis/tree/master/tests/data/007_git_result)
as any folders in the given repo are declarated as acnestis - we will see a simple copy files from the repo
Py version:
```python
from acnestis.processing import Processor
from acnestis.steps import git
PROCESSOR = Processor(
[git("https://github.com/oduvan/acnestis-test-repo.git", branch="main")],
)
```
YAML version:
```yaml
steps:
- name: git
url: https://github.com/oduvan/acnestis-test-repo.git
branch: main
```
### injection in the child
From: [tests/data/008_git_processing](https://github.com/oduvan/acnestis/tree/master/tests/data/008_git_processing) to: [tests/data/008_git_processing_result](https://github.com/oduvan/acnestis/tree/master/tests/data/008_git_processing_result)
YAML version:
```yaml
steps:
- name: git
url: https://github.com/oduvan/acnestis-test-repo.git
```
is very simple, but the master branch of the repo contains an acnestis-folder, which means it will be processed after checkout into the current one.
During the checkout `poem.txt` folder will be created and all of the files in the folder will be connected
but in the current repo we created a folder poem.txt with file `3.txt` so that file become a part of processing process
## More complex examples:
for more example check [tests/data](https://github.com/oduvan/acnestis/tree/master/tests/data) folder
* [009_git_subfolder](https://github.com/oduvan/acnestis/tree/master/tests/data/009_git_subfolder) - we don't need all of the files from the git repo, but only specific folder
* [010_subprocess_git](https://github.com/oduvan/acnestis/tree/master/tests/data/010_subprocess_git) - we declarate a specific acnestis folder for processing
* [011_sub_git_copy](https://github.com/oduvan/acnestis/tree/master/tests/data/011_sub_git_copy) - using of copy and rm steps
* [013_simple_aggregate](https://github.com/oduvan/acnestis/tree/master/tests/data/013_simple_aggregate) and [014_aggregate_two_poems](https://github.com/oduvan/acnestis/tree/master/tests/data/014_aggregate_two_poems) - more complex aggregation examples
Raw data
{
"_id": null,
"home_page": "https://github.com/oduvan/acnestis",
"name": "acnestis",
"maintainer": "",
"docs_url": null,
"requires_python": ">3.8",
"maintainer_email": "",
"keywords": "",
"author": "oduvan",
"author_email": "a.lyabah@checkio.org",
"download_url": "https://files.pythonhosted.org/packages/4d/c0/56c4a54b3b03d3d8bed73be29d6f5650911932210a27d7734cdb59e28a64/acnestis-0.1.1.tar.gz",
"platform": null,
"description": "[](https://pypi.python.org/pypi/acnestis/) \n[](https://pypi.python.org/pypi/acnestis/)\n[](https://github.com/psf/black)\n\n## Acnestis - collect, aggregate and convert\n\nA very simple tool that allows you collect data from different sources, change it ia different ways\n\n## Install\n\n```bash\n$ pip install acnestis\n```\n\n## You can do in declarative way\n\n* convert data in the current folder\n* aggregate data from different sources\n* inheritence - when you collect data from different source - you can replace any file and those files will be used in the child repo\n\n## Ok, this is confusing. Show me some example\n\nfirst of all, you can find some examples in the [tests/data](https://github.com/oduvan/acnestis/tree/master/tests/data) folder \n\n*(source folders, or from-folders, may contain both yaml and py files, that doesn't mean both are required. We just make alternatives for testing and illustration)*\n\n\n## Acnestis decloration\n\nin order to declarate folder as acnestis folder you should do one of the following \n\n* create `.acnestis.yaml` file in that folder\n* create `.acnestis.py` file in that folder. Py-file has an identical functionality, but some more power (of course)\n* create both `.asnestis.yaml` and `.acnestis.py` - in that case yaml-file will be used as the main one, but can use declared global variables from the py-file\n* declarate some of the sub-folders (even non-existed), in the current acnestis-decloration\n\nIn order to process acnestis folders - you should do\n\n```bash\n$ acnesting process from/folder to/folder\n```\n\nif no acnestis decloration in the `from/folder` it will just copy files to `to/folder`. if `from/folder` has at least one acnestis decloration - it will be used for processing that folder, the rest will be copied\n\n\n### Simple convertion\n\n*From: [tests/data/002_concat_poem](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem/) to: [tests/data/002_concat_poem_result](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem_result)*\n\nthis is a very simple example, where all files from folder poem.txt will be connected into a single file\n\n* we declarate [poem.txt](https://github.com/oduvan/acnestis/tree/master/tests/data/002_concat_poem/poem.txt) folder as acnestis folder\n\nPy version:\n\n```python\nfrom acnestis.processing import Processor\nfrom acnestis.steps import concat_files\n\nPROCESSOR = Processor([concat_files(\"poem.txt\")], as_file=\"poem.txt\")\n```\n\nYAML version:\n\n```yaml\nsteps:\n - name: concat_files\n into_file: poem.txt\nas_file: poem.txt\n```\n\nIt has one single step `concat_files` with attribute `into_file: poem.txt` - so it simply concat all the files into one\n\nAlso processing has attribute `as_file` means the folder will be replaced with one file `poem.txt`\n\n### Docker for more complex convertion\n\n*From [tests/data/006_docker_svgo](https://github.com/oduvan/acnestis/tree/master/tests/data/006_docker_svgo) to: [tests/data/006_docker_svgo_result](https://github.com/oduvan/acnestis/tree/master/tests/data/006_docker_svgo_result)\n\nyou can build own convertion tools using docker\n\nPy version:\n\n```python\nfrom acnestis.processing import Processor\nfrom acnestis.steps import docker\n\nPROCESSOR = Processor(\n [docker(\"acnestis_svgo\", skip_pull=True)],\n)\n```\n\nYAML version:\n\n```yaml\nsteps:\n - name: docker\n image: acnestis_svgo\n skip_pull: true\n```\n\none single step of using docker image `acnestis_svgo` for processing acnestis folder.\n\n[Dockerfile of acnestis_svgo](https://github.com/oduvan/acnestis/tree/master/tests/docker_svgo/Dockerfile) is very simple:\n\n```Dockerfile\n# Base image\nFROM node:20-alpine\n\n# Install SVGO\nRUN npm install -g svgo\n\n# Execute the SVGO command when the container starts\nENTRYPOINT [\"svgo\", \"-f\", \"/data_input\", \"-o\", \"/data_output\"]\n```\n\n### you can use simple python-code for convertion\n\nFrom [tests/data/004_code](https://github.com/oduvan/acnestis/tree/master/tests/data/004_code) to [tests/data/004_code_result](https://github.com/oduvan/acnestis/tree/master/tests/data/004_code_result)\n\n## Now let's play with aggregation.\n\nwe want not only convert data, but first collect it from different sources\n\nFor the testing and examples we will use [oduvan/acnestis-test-repo](https://github.com/oduvan/acnestis-test-repo.git) github repository and its branches.\n\n### simple use of git-repo\n\nFrom: [data/007_git](https://github.com/oduvan/acnestis/tree/master/tests/data/007_git) to [data/007_git_result](https://github.com/oduvan/acnestis/tree/master/tests/data/007_git_result)\n\nas any folders in the given repo are declarated as acnestis - we will see a simple copy files from the repo\n\nPy version:\n\n```python\nfrom acnestis.processing import Processor\nfrom acnestis.steps import git\n\nPROCESSOR = Processor(\n [git(\"https://github.com/oduvan/acnestis-test-repo.git\", branch=\"main\")],\n)\n```\n\nYAML version:\n\n```yaml\nsteps:\n - name: git\n url: https://github.com/oduvan/acnestis-test-repo.git\n branch: main\n```\n\n### injection in the child \n\nFrom: [tests/data/008_git_processing](https://github.com/oduvan/acnestis/tree/master/tests/data/008_git_processing) to: [tests/data/008_git_processing_result](https://github.com/oduvan/acnestis/tree/master/tests/data/008_git_processing_result)\n\nYAML version:\n\n```yaml\nsteps:\n - name: git\n url: https://github.com/oduvan/acnestis-test-repo.git\n```\n\nis very simple, but the master branch of the repo contains an acnestis-folder, which means it will be processed after checkout into the current one.\n\nDuring the checkout `poem.txt` folder will be created and all of the files in the folder will be connected\n\nbut in the current repo we created a folder poem.txt with file `3.txt` so that file become a part of processing process \n\n## More complex examples:\n\nfor more example check [tests/data](https://github.com/oduvan/acnestis/tree/master/tests/data) folder\n\n* [009_git_subfolder](https://github.com/oduvan/acnestis/tree/master/tests/data/009_git_subfolder) - we don't need all of the files from the git repo, but only specific folder\n* [010_subprocess_git](https://github.com/oduvan/acnestis/tree/master/tests/data/010_subprocess_git) - we declarate a specific acnestis folder for processing\n* [011_sub_git_copy](https://github.com/oduvan/acnestis/tree/master/tests/data/011_sub_git_copy) - using of copy and rm steps\n* [013_simple_aggregate](https://github.com/oduvan/acnestis/tree/master/tests/data/013_simple_aggregate) and [014_aggregate_two_poems](https://github.com/oduvan/acnestis/tree/master/tests/data/014_aggregate_two_poems) - more complex aggregation examples",
"bugtrack_url": null,
"license": "MIT",
"summary": "data collection, aggregation and convertion tool",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/oduvan/acnestis",
"Repository": "https://github.com/oduvan/acnestis"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "90e96ef0fd0d5aac9d90c502b688d485b4afeaabaa8017b677e4d4e648fc9c09",
"md5": "deeb12084d0ec8572bca73d053429ae0",
"sha256": "92601584dddfe5aac3ec44682bb29550113fec212b42c839963c5e91adaca5e0"
},
"downloads": -1,
"filename": "acnestis-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "deeb12084d0ec8572bca73d053429ae0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.8",
"size": 7324,
"upload_time": "2023-06-21T18:51:19",
"upload_time_iso_8601": "2023-06-21T18:51:19.114678Z",
"url": "https://files.pythonhosted.org/packages/90/e9/6ef0fd0d5aac9d90c502b688d485b4afeaabaa8017b677e4d4e648fc9c09/acnestis-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4dc056c4a54b3b03d3d8bed73be29d6f5650911932210a27d7734cdb59e28a64",
"md5": "0ad8dd40af4ba80ad9b00970cbdc692f",
"sha256": "23af907ef129a0d720440ea45e8e43152e2e7f54daf0693bea70e6b98755c4dc"
},
"downloads": -1,
"filename": "acnestis-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "0ad8dd40af4ba80ad9b00970cbdc692f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.8",
"size": 6009,
"upload_time": "2023-06-21T18:51:20",
"upload_time_iso_8601": "2023-06-21T18:51:20.800550Z",
"url": "https://files.pythonhosted.org/packages/4d/c0/56c4a54b3b03d3d8bed73be29d6f5650911932210a27d7734cdb59e28a64/acnestis-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-21 18:51:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "oduvan",
"github_project": "acnestis",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "acnestis"
}
