[](https://pypi.org/project/cliboa)
[](https://pypi.org/project/cliboa)
[](https://pypi.org/project/cliboa)
[](https://github.com/BrainPad/cliboa/actions/workflows/test.yaml)
[](https://github.com/psf/black)
[](https://github.com/BrainPad/cliboa/blob/master/CONTRIBUTING.md)
[](https://github.com/BrainPad/cliboa)
[](https://gitter.im/cliboa/users?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
# Table of Contents
* [Introduction](#introduction)
* [What is cliboa](#what-is-cliboa)
* [Features](#features)
* [Manual](#manual)
* [How to Contribute](#how-to-contribute)
* [Quick Start](#quick-start)
* [Install cliboa](#install-cliboa)
* [Configuration of a Simple ETL Processing](#configuration-of-a-simple-etl-processing)
* [Directory Tree](#directory-tree)
* [Install PyPI packages](#install-pypi-packages)
* [Write a Scenario of ETL Processing](#write-a-scenario-of-etl-processing)
* [Set an environment](#set-an-environment)
* [Execute a scenario of ETL Processing](#execute-a-scenario-of-etl-processing)
* [YAML Configuration](#yaml-configuration)
* [Default ETL Modules](#default-etl-modules)
* [How to Implement Additional ETL Modules](#how-to-implement-additional-etl-modules)
# Introduction
## What is cliboa
cliboa is an application framework which can implement ETL(ELT) pipeline. It eases the implementation of ETL(ELT) pipeline. In this case, ETL(ELT) pipeline means the processings like fetch, transform and transfer of data between various databases, storages, and other services.
## Features
- Python based framework.
- ETL(ELT) processing is executable by YAML based configuration.
- Additional modules for ETL(ELT) pipeline can be implemented by only a few steps if default modules not enough.
# Manual
See [MANUAL.md](/MANUAL.md)
# How to Contribute
See [CONTRIBUTING.md](/CONTRIBUTING.md)
# Quick Start
## Requirements
Available on macOS and any Linux distributions, like Debian, Ubuntu, CentOS, REL, or etc.
## Install cliboa
Python version 3.10 or later and poetry are required. In the environment which pip can be used, execute as below.
```
sudo pip3 install poetry
sudo pip3 install cliboa
```
## Configuration of a Simple ETL Processing
After installed cliboa, 'cliboadmin' can be used as an administrator command.
Create an executable environment of cliboa by using cliboadmin.
```
$ cd /usr/local
$ sudo cliboadmin init sample
$ cd sample
$ sudo cliboadmin create simple-etl
```
## Directory Tree
Directory tree which was created aforementioned commands is as below.
```
sample
├── pyproject.toml
├── bin
│ └── clibomanager.py
├── cliboa
│ └── conf
├── common
│ ├── __init__.py
│ ├── environment.py
│ └── scenario
├── conf
│ ├── cliboa.ini
│ └── logging.conf
├── logs
├── project
│ └── simple-etl
│ ├── scenario
│ └── scenario.yml
```
## Install PyPI packages
```
$ cd sample
$ poetry install
```
## Write a Scenario of ETL Processing
As a simple ETL processing, write scenario.yml in simple-etl as below.
The following example is just download a gzip file from the local sftp server, decompress it, and upload it to the local sftp server.
See [Examples](docs/yaml_configuration.md#examples)
## Set an Environment
To make the above scenario available, set a local machine as a sftp server according to respective environments. Also, put "test.csv.gz" under /usr/local.
## Execute a Scenario of ETL Processing
After wrote scenario.yml and set the environment, execute a scenario by as below command.
```
cd sample
poetry run python3 bin/clibomanager.py simple-etl
```
# YAML Configuration
see [yaml_configuration.md](/docs/yaml_configuration.md)
# Default ETL Modules
see [default_etl_modules.md](/docs/default_etl_modules.md)
# How to Implement Additional ETL Modules
see [additional_etl_modules.md](/docs/additional_etl_modules.md)
Raw data
{
"_id": null,
"home_page": null,
"name": "cliboa",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "etl, elt, data pipeline, workflow, automation, framework",
"author": "BrainPad",
"author_email": "cliboa@brainpad.co.jp",
"download_url": "https://files.pythonhosted.org/packages/0f/f9/17d611b866009eaea707136025e4bf113a05596edca6fc0e9da243129a34/cliboa-2.6.1b0.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/cliboa)\n[](https://pypi.org/project/cliboa)\n[](https://pypi.org/project/cliboa)\n[](https://github.com/BrainPad/cliboa/actions/workflows/test.yaml)\n[](https://github.com/psf/black)\n[](https://github.com/BrainPad/cliboa/blob/master/CONTRIBUTING.md)\n[](https://github.com/BrainPad/cliboa)\n[](https://gitter.im/cliboa/users?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)\n\n\n# Table of Contents\n* [Introduction](#introduction)\n * [What is cliboa](#what-is-cliboa)\n * [Features](#features)\n* [Manual](#manual)\n* [How to Contribute](#how-to-contribute)\n* [Quick Start](#quick-start)\n * [Install cliboa](#install-cliboa)\n * [Configuration of a Simple ETL Processing](#configuration-of-a-simple-etl-processing)\n * [Directory Tree](#directory-tree)\n * [Install PyPI packages](#install-pypi-packages)\n * [Write a Scenario of ETL Processing](#write-a-scenario-of-etl-processing)\n * [Set an environment](#set-an-environment)\n * [Execute a scenario of ETL Processing](#execute-a-scenario-of-etl-processing)\n* [YAML Configuration](#yaml-configuration)\n* [Default ETL Modules](#default-etl-modules)\n* [How to Implement Additional ETL Modules](#how-to-implement-additional-etl-modules)\n\n# Introduction\n## What is cliboa\ncliboa is an application framework which can implement ETL(ELT) pipeline. It eases the implementation of ETL(ELT) pipeline. In this case, ETL(ELT) pipeline means the processings like fetch, transform and transfer of data between various databases, storages, and other services.\n\n\n## Features\n- Python based framework.\n- ETL(ELT) processing is executable by YAML based configuration.\n- Additional modules for ETL(ELT) pipeline can be implemented by only a few steps if default modules not enough.\n\n# Manual\nSee [MANUAL.md](/MANUAL.md)\n\n# How to Contribute\nSee [CONTRIBUTING.md](/CONTRIBUTING.md)\n\n\n# Quick Start\n## Requirements\nAvailable on macOS and any Linux distributions, like Debian, Ubuntu, CentOS, REL, or etc.\n\n## Install cliboa\nPython version 3.10 or later and poetry are required. In the environment which pip can be used, execute as below.\n\n```\nsudo pip3 install poetry\nsudo pip3 install cliboa\n```\n\n## Configuration of a Simple ETL Processing\nAfter installed cliboa, 'cliboadmin' can be used as an administrator command. \n\nCreate an executable environment of cliboa by using cliboadmin.\n\n```\n$ cd /usr/local\n$ sudo cliboadmin init sample\n$ cd sample\n$ sudo cliboadmin create simple-etl\n```\n\n## Directory Tree\nDirectory tree which was created aforementioned commands is as below.\n\n```\nsample\n\u251c\u2500\u2500 pyproject.toml\n\u251c\u2500\u2500 bin\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 clibomanager.py\n\u251c\u2500\u2500 cliboa\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 conf\n\u251c\u2500\u2500 common\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 environment.py\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 scenario\n\u251c\u2500\u2500 conf\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cliboa.ini\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 logging.conf\n\u251c\u2500\u2500 logs\n\u251c\u2500\u2500 project\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 simple-etl\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 scenario\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 scenario.yml\n```\n\n## Install PyPI packages\n```\n$ cd sample\n$ poetry install\n```\n\n## Write a Scenario of ETL Processing\nAs a simple ETL processing, write scenario.yml in simple-etl as below.\n\nThe following example is just download a gzip file from the local sftp server, decompress it, and upload it to the local sftp server.\n\nSee [Examples](docs/yaml_configuration.md#examples)\n\n## Set an Environment\nTo make the above scenario available, set a local machine as a sftp server according to respective environments. Also, put \"test.csv.gz\" under /usr/local.\n\n## Execute a Scenario of ETL Processing\nAfter wrote scenario.yml and set the environment, execute a scenario by as below command.\n```\ncd sample\npoetry run python3 bin/clibomanager.py simple-etl\n```\n\n# YAML Configuration\nsee [yaml_configuration.md](/docs/yaml_configuration.md)\n\n# Default ETL Modules\nsee [default_etl_modules.md](/docs/default_etl_modules.md)\n\n# How to Implement Additional ETL Modules\nsee [additional_etl_modules.md](/docs/additional_etl_modules.md)\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "application framework for ETL(ELT) processing",
"version": "2.6.1b0",
"project_urls": {
"ChangeLog": "https://github.com/BrainPad/cliboa/releases",
"Documentation": "https://github.com/BrainPad/cliboa/wiki",
"Homepage": "https://github.com/BrainPad/cliboa",
"Issues": "https://github.com/BrainPad/cliboa/issues",
"Repository": "https://github.com/BrainPad/cliboa"
},
"split_keywords": [
"etl",
" elt",
" data pipeline",
" workflow",
" automation",
" framework"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "b428001d04af1961e0c85078004ac2b50a1164a2cebb4c17d2698d302ced0c55",
"md5": "dd9a5baccc48a316e168a1f693fe3808",
"sha256": "afc40539acc6677e6ee824729c9856242aae5f31ec51f642f2db5a432b34cf37"
},
"downloads": -1,
"filename": "cliboa-2.6.1b0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dd9a5baccc48a316e168a1f693fe3808",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 175534,
"upload_time": "2025-09-08T03:58:25",
"upload_time_iso_8601": "2025-09-08T03:58:25.294775Z",
"url": "https://files.pythonhosted.org/packages/b4/28/001d04af1961e0c85078004ac2b50a1164a2cebb4c17d2698d302ced0c55/cliboa-2.6.1b0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "0ff917d611b866009eaea707136025e4bf113a05596edca6fc0e9da243129a34",
"md5": "32cb8f41a9a9295b9fe834b3f5de8cb4",
"sha256": "15c01bbce50b8b64b54f3b7df2a02476f34c08c5a15739fe16f094a23b530d1f"
},
"downloads": -1,
"filename": "cliboa-2.6.1b0.tar.gz",
"has_sig": false,
"md5_digest": "32cb8f41a9a9295b9fe834b3f5de8cb4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 92523,
"upload_time": "2025-09-08T03:58:26",
"upload_time_iso_8601": "2025-09-08T03:58:26.841179Z",
"url": "https://files.pythonhosted.org/packages/0f/f9/17d611b866009eaea707136025e4bf113a05596edca6fc0e9da243129a34/cliboa-2.6.1b0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-08 03:58:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BrainPad",
"github_project": "cliboa",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "cliboa"
}
JSON
