| Name | bomf JSON |
| Version |
0.11.3
JSON |
| download |
| home_page | None |
| Summary | BO4E Migration Framework |
| upload_time | 2024-09-26 06:46:04 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | MIT |
| keywords |
bo4e
data
migration
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# BO4E Migration Framework (bomf)
BOMF is the BO4E Migration Framework.
This repository contains the code of the Python package [bomf](https://pypi.org/project/bomf).
## Rationale
bomf is a framework, that allows its users to migrate data
- from source systems (starting with the raw data extracts)
- into an intermediate, common BO4E based data layer.
- From there map data to individual target system data models
- and finally create records in target systems (aka "loading").
The framework
- encourages user to program consistent data processing pipelines from any source to any target system
- enforces users to adapt to structured and consistent patterns
- and by doing so will lead to higher chances for maintainable and reusable code.
## Architecture / Overview
The overall setup for a migration from 1-n source systems (A, B, C...) to 1-m target systems (1,2, 3...) might look like this:
```mermaid
graph TD
A[Source System A] -->|System A DB Dump| A2[Source A Data Model: A JSON Extract]
B[Source System B] -->|System B CSV Export| B2[Source B Data Model: B CSV Files]
A2 -->|SourceAToBo4eDataSetMapper| C{Intermediate BO4E Layer aka DataSets}
B2 -->|SourceBToBo4eDataSetMapper| C
C -->|validations| C
C -->|Bo4eDataSetToTarget1Mapper| D1[Target 1 Data Model]
C -->|Bo4eDataSetToTarget2Mapper| D2[Target 2 Data Model]
C -->|Bo4eDataSetToTarget3Mapper| D3[Target 3 Data Model]
D1 -->L1[Target 1 Loader]
D2 -->L2[Target 2 Loader]
D3 -->L3[Target 3 Loader]
L1 -->M1[Target System 1]
L2 -->M2[Target System 2]
L3 -->M3[Target System 3]
```
The Intermediate BO4E Layer (that consists of different so called DataSets) is kind of a contract between the code that maps *from the source data model* and the code that maps *to the target data model*.
### Data Migration Flow
The migration of specific data from source to target is always the same:
```mermaid
graph TD
A1{Source Data 1} -->|Export| B1(All source data 1 extracts)
B1 -->C1[Filter on source data 1 model aka Pre-Select 1]
A2{Source Data 2} -->|Export| B2(All source data 2 extracts)
B2 -->C2[Filter on source data 2 model aka Pre-Select 2]
C1 -->|do not match filter predicate| Z{discarded data}
C1 -->|match filter criteria| M(Custom Logic: SourceDataSetToBo4EDataSetMapper)
C2 -->|do not match filter predicate| Z
C2 -->|match filter criteria| M
M -->|mapping| E(BO4E Data Sets)
E -->F[Validation]
F -->|obeys a validation rule|E
F -->|violate any validation rule|Z
F -->|passes all validations| G[BO4E to Target Mapper]
G -->|mapping| H(target data model)
H -->I[Target Loader]
I -->|load target model|L1[Loader: 1. load to target]
L1 -->|first: load to|T{Target System}
L1 -->|then|L2[Loader: 2 optionally poll until target has processed data]
L2 -->|second: poll until|T
L2 -->|then|L3[Loader: 3 optionally verify the data have been processed correctly]
L3 -->|finally: verify|T
L3 -->|verification failed|Z
L1 -->|loading failed|Z
L3 -->|verification successful|Y[The End.]
Z-->Z1[Monitoring and Logging]
Z1-->Z2[Human Analyst]
Z2 -.->|manually checks| T
Z2 -.->|feedback: heuristically define new rules for|F
Z2 -.->|feedback: heurisically define new filters for|C
```
## How to use this Repository on Your Machine (Development)
Please follow the [instructions in our Python Template Repository](https://github.com/Hochfrequenz/python_template_repository).
tl;dr: `tox`.
## Contribute
You are very welcome to contribute to this template repository by opening a pull request against the main branch.
Raw data
{
"_id": null,
"home_page": null,
"name": "bomf",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "BO4E, Data, Migration",
"author": null,
"author_email": "Hochfrequenz Unternehmensberatung GmbH <info@hochfrequenz.de>",
"download_url": "https://files.pythonhosted.org/packages/f0/3c/ebb491bd23f2c5d936f5ef480435034c512c181fa3c4c4f93eead61e827b/bomf-0.11.3.tar.gz",
"platform": null,
"description": "# BO4E Migration Framework (bomf)\n\nBOMF is the BO4E Migration Framework.\nThis repository contains the code of the Python package [bomf](https://pypi.org/project/bomf).\n\n\n\n\n\n\n\n## Rationale\nbomf is a framework, that allows its users to migrate data\n- from source systems (starting with the raw data extracts)\n- into an intermediate, common BO4E based data layer.\n- From there map data to individual target system data models\n- and finally create records in target systems (aka \"loading\").\n\nThe framework\n- encourages user to program consistent data processing pipelines from any source to any target system \n- enforces users to adapt to structured and consistent patterns\n- and by doing so will lead to higher chances for maintainable and reusable code.\n\n## Architecture / Overview\nThe overall setup for a migration from 1-n source systems (A, B, C...) to 1-m target systems (1,2, 3...) might look like this:\n\n```mermaid\ngraph TD\n A[Source System A] -->|System A DB Dump| A2[Source A Data Model: A JSON Extract]\n B[Source System B] -->|System B CSV Export| B2[Source B Data Model: B CSV Files]\n A2 -->|SourceAToBo4eDataSetMapper| C{Intermediate BO4E Layer aka DataSets}\n B2 -->|SourceBToBo4eDataSetMapper| C\n C -->|validations| C\n C -->|Bo4eDataSetToTarget1Mapper| D1[Target 1 Data Model]\n C -->|Bo4eDataSetToTarget2Mapper| D2[Target 2 Data Model]\n C -->|Bo4eDataSetToTarget3Mapper| D3[Target 3 Data Model]\n D1 -->L1[Target 1 Loader]\n D2 -->L2[Target 2 Loader]\n D3 -->L3[Target 3 Loader]\n L1 -->M1[Target System 1]\n L2 -->M2[Target System 2]\n L3 -->M3[Target System 3]\n```\nThe Intermediate BO4E Layer (that consists of different so called DataSets) is kind of a contract between the code that maps *from the source data model* and the code that maps *to the target data model*.\n\n### Data Migration Flow\nThe migration of specific data from source to target is always the same:\n```mermaid\ngraph TD\n A1{Source Data 1} -->|Export| B1(All source data 1 extracts)\n B1 -->C1[Filter on source data 1 model aka Pre-Select 1]\n A2{Source Data 2} -->|Export| B2(All source data 2 extracts)\n B2 -->C2[Filter on source data 2 model aka Pre-Select 2]\n C1 -->|do not match filter predicate| Z{discarded data}\n C1 -->|match filter criteria| M(Custom Logic: SourceDataSetToBo4EDataSetMapper) \n C2 -->|do not match filter predicate| Z\n C2 -->|match filter criteria| M\n M -->|mapping| E(BO4E Data Sets)\n E -->F[Validation]\n F -->|obeys a validation rule|E\n F -->|violate any validation rule|Z\n F -->|passes all validations| G[BO4E to Target Mapper]\n G -->|mapping| H(target data model)\n H -->I[Target Loader]\n I -->|load target model|L1[Loader: 1. load to target]\n L1 -->|first: load to|T{Target System}\n L1 -->|then|L2[Loader: 2 optionally poll until target has processed data]\n L2 -->|second: poll until|T\n L2 -->|then|L3[Loader: 3 optionally verify the data have been processed correctly]\n L3 -->|finally: verify|T\n L3 -->|verification failed|Z\n L1 -->|loading failed|Z\n L3 -->|verification successful|Y[The End.]\n Z-->Z1[Monitoring and Logging]\n Z1-->Z2[Human Analyst]\n Z2 -.->|manually checks| T\n Z2 -.->|feedback: heuristically define new rules for|F\n Z2 -.->|feedback: heurisically define new filters for|C\n```\n\n\n## How to use this Repository on Your Machine (Development)\n\nPlease follow the [instructions in our Python Template Repository](https://github.com/Hochfrequenz/python_template_repository).\ntl;dr: `tox`.\n\n## Contribute\n\nYou are very welcome to contribute to this template repository by opening a pull request against the main branch.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "BO4E Migration Framework",
"version": "0.11.3",
"project_urls": {
"Changelog": "https://github.com/Hochfrequenz/bo4e_migration_framework/releases",
"Homepage": "https://github.com/Hochfrequenz/bo4e_migration_framework"
},
"split_keywords": [
"bo4e",
" data",
" migration"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "67ca6921485d5667632c51d430df377d966d45f2ae38ff5de3eb858daf6836d7",
"md5": "495c7cefa0af314100c73b01deddb652",
"sha256": "acb7272fa8996582e2d9dc84ba35cff910229387538fb11c018e2c332554708a"
},
"downloads": -1,
"filename": "bomf-0.11.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "495c7cefa0af314100c73b01deddb652",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 18283,
"upload_time": "2024-09-26T06:46:03",
"upload_time_iso_8601": "2024-09-26T06:46:03.191664Z",
"url": "https://files.pythonhosted.org/packages/67/ca/6921485d5667632c51d430df377d966d45f2ae38ff5de3eb858daf6836d7/bomf-0.11.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f03cebb491bd23f2c5d936f5ef480435034c512c181fa3c4c4f93eead61e827b",
"md5": "3209daf4b72be6d01704b9c2e6d9e3d7",
"sha256": "d527a684d0c21c1145bc6318589030d6504b116a18c98002b604c5decf888699"
},
"downloads": -1,
"filename": "bomf-0.11.3.tar.gz",
"has_sig": false,
"md5_digest": "3209daf4b72be6d01704b9c2e6d9e3d7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 21117,
"upload_time": "2024-09-26T06:46:04",
"upload_time_iso_8601": "2024-09-26T06:46:04.562285Z",
"url": "https://files.pythonhosted.org/packages/f0/3c/ebb491bd23f2c5d936f5ef480435034c512c181fa3c4c4f93eead61e827b/bomf-0.11.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-26 06:46:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Hochfrequenz",
"github_project": "bo4e_migration_framework",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "bomf"
}
