# Kaiba
Kaiba is a data transformation tool written in Python that uses a DTL(Data Transformation Language) expressed in normal JSON to govern output structure, data fetching and data transformation.
___
[](https://codecov.io/gh/kaiba-tech/kaiba)
[](https://pypi.org/project/kaiba/)
[](https://github.com/wemake-services/wemake-python-styleguide)
___
**Documentation
([Stable](https://kaiba.readthedocs.io/) |
[Latest](https://kaiba.readthedocs.io/en/latest/)) |
[Source Code](https://github.com/kaiba-tech/kaiba) |
[Task Tracker](https://github.com/kaiba-tech/kaiba/issues)**
## What is Kaiba
Kaiba is a JSON to JSON mapper. That means that we read input JSON and create output JSON. How the output is created is based on instructions from a configuration file. The configuration file governs the the output structure and tells Kaiba where in the input to find data and where to place it in the output. In addition to this Kaiba supports data transformation with `data casting`, `regular expressions`, `if conditions`, `combination of data from multiple places` and of course setting `default` values.
__This enables you to change any input into the output you desire.__
## The Kaiba App
The kaiba App is currently in development
[app.kaiba.tech](https://app.kaiba.tech)
The app provides a user interface for creating Kaiba configurations. With the app you can map in real time easily and create the kaiba config.
## The Kaiba API
The kaiba api is open for anyone to try, you send your data and the configuration and get mapped data response.
[api.kaiba.tech/docs](https://api.kaiba.tech/docs)
## Typical usecases
* You `GET` data from api, but need to transform it for your backend system
* `POST`ing data to an api that needs data on a different format than what your system produces
* All your backends speak different language? pipe it through __Kaiba__
* Customer delivers weirdly formatted data? Use __Kaiba__ to make it sexy
* Have CSV but need nicely structured JSON? make CSV into a JSON list and transform it with __Kaiba__
* Have XML but need to change it? make it into JSON, transform it with __Kaiba__ and then dump it to XML again.
* Customers legacy system needs CSV. Use __Kaiba__ to transform your nicely structured JSON data into a JSON List that can be easily dumped to CSV
## Official Open kaiba Solutions
[kaiba-cli](https://github.com/kaiba-tech/kaiba-cli), commandline interface for file to file mapping.
[kaiba-api](https://github.com/kaiba-tech/kaiba-api), FastAPI driven rest server that maps data with kaiba
## Enterprise solutions
Coming...
## Goal
The goal of this library is to make JSON to JSON transformation/mapping easy, configurable and documentable. We achieve this by using a simple but feature-rich JSON configuration which then also acts as documentation and as a contract between parties.
## Why
Kaiba was born because we really dislike mapping. Documenting whatever decisions made in your code so that some product owner understands it is also _no me gusto_. Transforming data from one format to another is something software engineers do allmost daily... It should be easy! And documenting it shouldn't be something you have to worry about.
After the Worst POC in History we never wanted to do mapping by scripts and code again. This lead to the idea that it should be possible to create a file which governs how the structure should look and how the data should be transformed. This would then be the `single source of truth` and with Kaiba we have achieved this.
We believe that this will make collaboration between teams faster and easier. Use Kaiba to agree with data formats between Front-end and Back-end. Between the 3rd party system and your back-end. You can even use Kaiba for testing existing integrations ;-)
## Features
* Mapping with configuration File.
* [JSON Schema](https://json-schema.org/) validation of the config file.
* Structurally Transform JSON
* Combine multiple values to one.
* Default values
* If statements
* is, contains, in, not
* Casting
* integer, decimal, iso date
* Regular Expressions
## Contributing
Please see [contribute](https://kaiba.readthedocs.io/en/stable/contributing)
## Installation
Package is on pypi. Use pip or poetry to install
```sh
pip install kaiba
```
```sh
poetry add kaiba
```
## Introduction
Have a look at our introduction course [here](https://kaiba.readthedocs.io/en/stable/introduction)
## Quickstart
```python
import simplejson
from kaiba.process import process
my_config = {
'name': 'schema',
'array': False,
'objects': [
{
'name': 'invoices',
'array': True,
'iterators': [
{
'alias': 'invoice',
'path': ['root', 'invoices'],
},
],
'attributes': [
{
'name': 'amount',
'data_fetchers': [
{
'path': ['invoice', 'amount'],
},
],
'casting': {
'to': 'decimal',
'original_format': 'integer_containing_decimals',
},
'default': 0,
},
{
'name': 'debtor',
'data_fetchers': [
{
'path': ['root', 'customer', 'first_name'],
},
{
'path': ['root', 'customer', 'last_name'],
},
],
'separator': ' ',
},
],
'objects': [],
},
],
}
example_data = {
'root': {
'customer': {
'first_name': 'John',
'last_name': 'Smith',
},
'invoices': [
{
'amount': 10050,
},
{
'amount': 20050,
},
{
'amount': -15005,
},
],
},
}
mapped_data = process(example_data, my_config)
with open('resultfile.json', 'w') as output_file:
output_file.write(simplejson.dumps(mapped_data))
```
contents of resultfile.json
```json
{
"invoices": [
{
"amount": 100.5,
"debtor": "John Smith"
},
{
"amount": 200.5,
"debtor": "John Smith"
},
{
"amount": -150.05,
"debtor": "John Smith"
}
]
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/kaiba-tech/kaiba",
"name": "kaiba",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0.0",
"maintainer_email": "",
"keywords": "Json mapping,data transformation,json to json,dict to dict,configurable",
"author": "Thomas Borgen",
"author_email": "thomas.borgen@greenbird.com",
"download_url": "https://files.pythonhosted.org/packages/5d/ef/6e970276ff2f8f2844b0e38d03efe153b825d086996cda1e2c06f6f68f1c/kaiba-3.0.1.tar.gz",
"platform": null,
"description": "# Kaiba\n\nKaiba is a data transformation tool written in Python that uses a DTL(Data Transformation Language) expressed in normal JSON to govern output structure, data fetching and data transformation.\n___\n\n[](https://codecov.io/gh/kaiba-tech/kaiba)\n[](https://pypi.org/project/kaiba/)\n[](https://github.com/wemake-services/wemake-python-styleguide)\n___\n\n**Documentation\n([Stable](https://kaiba.readthedocs.io/) |\n[Latest](https://kaiba.readthedocs.io/en/latest/)) |\n[Source Code](https://github.com/kaiba-tech/kaiba) |\n[Task Tracker](https://github.com/kaiba-tech/kaiba/issues)**\n\n## What is Kaiba\n\nKaiba is a JSON to JSON mapper. That means that we read input JSON and create output JSON. How the output is created is based on instructions from a configuration file. The configuration file governs the the output structure and tells Kaiba where in the input to find data and where to place it in the output. In addition to this Kaiba supports data transformation with `data casting`, `regular expressions`, `if conditions`, `combination of data from multiple places` and of course setting `default` values.\n\n__This enables you to change any input into the output you desire.__\n\n## The Kaiba App\n\nThe kaiba App is currently in development\n\n[app.kaiba.tech](https://app.kaiba.tech)\n\nThe app provides a user interface for creating Kaiba configurations. With the app you can map in real time easily and create the kaiba config.\n\n## The Kaiba API\n\nThe kaiba api is open for anyone to try, you send your data and the configuration and get mapped data response.\n\n[api.kaiba.tech/docs](https://api.kaiba.tech/docs)\n\n## Typical usecases\n\n* You `GET` data from api, but need to transform it for your backend system\n* `POST`ing data to an api that needs data on a different format than what your system produces\n* All your backends speak different language? pipe it through __Kaiba__\n* Customer delivers weirdly formatted data? Use __Kaiba__ to make it sexy\n* Have CSV but need nicely structured JSON? make CSV into a JSON list and transform it with __Kaiba__\n* Have XML but need to change it? make it into JSON, transform it with __Kaiba__ and then dump it to XML again.\n* Customers legacy system needs CSV. Use __Kaiba__ to transform your nicely structured JSON data into a JSON List that can be easily dumped to CSV\n\n## Official Open kaiba Solutions\n\n[kaiba-cli](https://github.com/kaiba-tech/kaiba-cli), commandline interface for file to file mapping.\n\n[kaiba-api](https://github.com/kaiba-tech/kaiba-api), FastAPI driven rest server that maps data with kaiba\n\n## Enterprise solutions\n\nComing...\n\n## Goal\n\nThe goal of this library is to make JSON to JSON transformation/mapping easy, configurable and documentable. We achieve this by using a simple but feature-rich JSON configuration which then also acts as documentation and as a contract between parties.\n\n## Why\n\nKaiba was born because we really dislike mapping. Documenting whatever decisions made in your code so that some product owner understands it is also _no me gusto_. Transforming data from one format to another is something software engineers do allmost daily... It should be easy! And documenting it shouldn't be something you have to worry about.\n\nAfter the Worst POC in History we never wanted to do mapping by scripts and code again. This lead to the idea that it should be possible to create a file which governs how the structure should look and how the data should be transformed. This would then be the `single source of truth` and with Kaiba we have achieved this.\n\nWe believe that this will make collaboration between teams faster and easier. Use Kaiba to agree with data formats between Front-end and Back-end. Between the 3rd party system and your back-end. You can even use Kaiba for testing existing integrations ;-)\n\n## Features\n\n* Mapping with configuration File.\n* [JSON Schema](https://json-schema.org/) validation of the config file.\n* Structurally Transform JSON\n* Combine multiple values to one.\n* Default values\n* If statements\n * is, contains, in, not\n* Casting\n * integer, decimal, iso date\n* Regular Expressions\n\n## Contributing\nPlease see [contribute](https://kaiba.readthedocs.io/en/stable/contributing)\n\n## Installation\n\nPackage is on pypi. Use pip or poetry to install\n\n```sh\npip install kaiba\n```\n```sh\npoetry add kaiba\n```\n\n## Introduction\n\nHave a look at our introduction course [here](https://kaiba.readthedocs.io/en/stable/introduction)\n\n## Quickstart\n```python\nimport simplejson\n\nfrom kaiba.process import process\n\nmy_config = {\n 'name': 'schema',\n 'array': False,\n 'objects': [\n {\n 'name': 'invoices',\n 'array': True,\n 'iterators': [\n {\n 'alias': 'invoice',\n 'path': ['root', 'invoices'],\n },\n ],\n 'attributes': [\n {\n 'name': 'amount',\n 'data_fetchers': [\n {\n 'path': ['invoice', 'amount'],\n },\n ],\n 'casting': {\n 'to': 'decimal',\n 'original_format': 'integer_containing_decimals',\n },\n 'default': 0,\n },\n {\n 'name': 'debtor',\n 'data_fetchers': [\n {\n 'path': ['root', 'customer', 'first_name'],\n },\n {\n 'path': ['root', 'customer', 'last_name'],\n },\n ],\n 'separator': ' ',\n },\n ],\n 'objects': [],\n },\n ],\n}\n\nexample_data = {\n 'root': {\n 'customer': {\n 'first_name': 'John',\n 'last_name': 'Smith',\n },\n 'invoices': [\n {\n 'amount': 10050,\n },\n {\n 'amount': 20050,\n },\n {\n 'amount': -15005,\n },\n ],\n },\n}\n\nmapped_data = process(example_data, my_config)\n\nwith open('resultfile.json', 'w') as output_file:\n output_file.write(simplejson.dumps(mapped_data))\n\n```\n\ncontents of resultfile.json\n```json\n{\n \"invoices\": [\n {\n \"amount\": 100.5,\n \"debtor\": \"John Smith\"\n },\n {\n \"amount\": 200.5,\n \"debtor\": \"John Smith\"\n },\n {\n \"amount\": -150.05,\n \"debtor\": \"John Smith\"\n }\n ]\n}\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Configurable and documentable Json transformation and mapping",
"version": "3.0.1",
"project_urls": {
"Documentation": "https://kaiba-tech.github.io/kaiba/",
"Homepage": "https://github.com/kaiba-tech/kaiba",
"Repository": "https://github.com/kaiba-tech/kaiba"
},
"split_keywords": [
"json mapping",
"data transformation",
"json to json",
"dict to dict",
"configurable"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "31748d35851d60e61bfeeba6b75b87f44c6460da786cca962a857f1c148eda9c",
"md5": "7219265565309c9daef859429fb92fb6",
"sha256": "616f74fa5482259351d45e333e4075960d19dd3b56882d934a1e67bd543704fb"
},
"downloads": -1,
"filename": "kaiba-3.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7219265565309c9daef859429fb92fb6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0.0",
"size": 21675,
"upload_time": "2024-02-04T18:36:07",
"upload_time_iso_8601": "2024-02-04T18:36:07.103968Z",
"url": "https://files.pythonhosted.org/packages/31/74/8d35851d60e61bfeeba6b75b87f44c6460da786cca962a857f1c148eda9c/kaiba-3.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5def6e970276ff2f8f2844b0e38d03efe153b825d086996cda1e2c06f6f68f1c",
"md5": "c099ccad8366eacffe897dc1902cf413",
"sha256": "0616ce3562e361abb5d5c33bdf6d39d53388c650760b95b2d143e686df3c0f98"
},
"downloads": -1,
"filename": "kaiba-3.0.1.tar.gz",
"has_sig": false,
"md5_digest": "c099ccad8366eacffe897dc1902cf413",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0.0",
"size": 17443,
"upload_time": "2024-02-04T18:36:09",
"upload_time_iso_8601": "2024-02-04T18:36:09.089308Z",
"url": "https://files.pythonhosted.org/packages/5d/ef/6e970276ff2f8f2844b0e38d03efe153b825d086996cda1e2c06f6f68f1c/kaiba-3.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-04 18:36:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kaiba-tech",
"github_project": "kaiba",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kaiba"
}
JSON
