## aiSSEMBLE™ Foundation Data Lineage Python
[](https://pypi.org/project/aissemble-foundation-data-lineage-python/)
This module serves as a generic wrapper for data lineage
types, objects, and functions. While it is designed to be
leveraged through the aiSSEMBLE™ ecosystem, it is not dependent
on aiSSEMBLE for execution. It is intentionally designed for
portability.
This Readme is intended to provide technical insight into the
implementation of this package. For consumption guidance,
please refer to the [aiSSEMBLE Github Pages](https://boozallen.github.io/aissemble/current/data-lineage.html)
### Core Functionality
This module presently provides three main capabilities:
* Generic types to represent Data Lineage metadata
* Convenience methods for emitting Data Lineage through various mediums
* Conversion functions to transform the generic types to and from popular Data Lineage formats.
Through these capabilities, this module fulfills the need for an easy-to-use, implementation-agnostic
Data Lineage interface.
### Developer Guidance
* This module should display little to no dependence on any other aiSSEMBLE module. It is intentionally generic.
* Any changes to method or function signatures must be reflected in any relevant template code within `foundation-mda`.
* Any change or addition in functionality must be accompanied by associated automated tests.
* As we are serving as an interface to third party libraries and services, any input parameters must be exhaustively validated.
### Use Custom Microprofile config properties
By default, foundation-data-lineage uses the below microprofile-config.properties for the messaging service configurations
```properties
kafka.bootstrap.servers=kafka-cluster:9093
mp.messaging.outgoing.lineage-event-out.cloud-events=false
mp.messaging.outgoing.lineage-event-out.connector=smallrye-kafka
mp.messaging.outgoing.lineage-event-out.topic=lineage-event-out
mp.messaging.outgoing.lineage-event-out.key.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.lineage-event-out.value.serializer=org.apache.kafka.common.serialization.StringSerializer
```
You can also use your own microprofile-config.properties file. To do that, you must provide your own property path to the code:
1. Set the path of the microprofile-config.properties on your file system
```python
# specify the new microprofile-config properties path
self.emitter.set_messaging_properties_path(string_path_to_the_property_file)
```
2. Set the new emitter topic
add below property to the data-lineage.properties file of the appropriate docker image (e.g. spark-worker)
```text
data.lineage.emission.topic=replace-with-your-topic
```
3. (Optional) If you are using the v1 kafka-cluster helm chart, add the new topic to kafka-cluster values.yaml file; e.g.: add `replace-with-your-topic:1:1` to the **KAFKA_CREATE_TOPICS** environment variable in the `-deploy/src/main/resources/apps/kafka-cluster/values.yaml`
## NOTE:
There is a known issue with the confluent-kafka dependency that is brought in by the openlineage-client library that prevents it from installing this requirement on some arm64 environments. The problem is that librdkafka does not provide a wheel for all operating systems in the arm64 architecture, requiring that it be built from source in order for confluent-kafka to be installed. If you run into this problem, you can install it from source using the below commands:
* apt-get update
* apt install -y python3-dev git
* git clone https://github.com/confluentinc/librdkafka.git
* cd librdkafka
* ./configure --arch=aarch64
* make
* make install
Raw data
{
"_id": null,
"home_page": null,
"name": "aissemble-foundation-data-lineage-python",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.11.4",
"maintainer_email": null,
"keywords": null,
"author": "aiSSEMBLE Baseline Community",
"author_email": "aissemble@bah.com",
"download_url": "https://files.pythonhosted.org/packages/3b/15/822916f9f7f9104202223cee6d15b5ed546f8b04c71b0c68156c0c124d85/aissemble_foundation_data_lineage_python-1.9.2.tar.gz",
"platform": null,
"description": "## aiSSEMBLE™ Foundation Data Lineage Python\n\n[](https://pypi.org/project/aissemble-foundation-data-lineage-python/)\n\n\n\nThis module serves as a generic wrapper for data lineage \ntypes, objects, and functions. While it is designed to be \nleveraged through the aiSSEMBLE™ ecosystem, it is not dependent\non aiSSEMBLE for execution. It is intentionally designed for \nportability.\n\nThis Readme is intended to provide technical insight into the\nimplementation of this package. For consumption guidance,\nplease refer to the [aiSSEMBLE Github Pages](https://boozallen.github.io/aissemble/current/data-lineage.html)\n\n### Core Functionality\n\nThis module presently provides three main capabilities:\n\n* Generic types to represent Data Lineage metadata\n* Convenience methods for emitting Data Lineage through various mediums\n* Conversion functions to transform the generic types to and from popular Data Lineage formats.\n\nThrough these capabilities, this module fulfills the need for an easy-to-use, implementation-agnostic \nData Lineage interface.\n\n### Developer Guidance\n\n* This module should display little to no dependence on any other aiSSEMBLE module. It is intentionally generic.\n* Any changes to method or function signatures must be reflected in any relevant template code within `foundation-mda`.\n* Any change or addition in functionality must be accompanied by associated automated tests.\n * As we are serving as an interface to third party libraries and services, any input parameters must be exhaustively validated.\n\n### Use Custom Microprofile config properties\nBy default, foundation-data-lineage uses the below microprofile-config.properties for the messaging service configurations\n```properties\nkafka.bootstrap.servers=kafka-cluster:9093\nmp.messaging.outgoing.lineage-event-out.cloud-events=false\nmp.messaging.outgoing.lineage-event-out.connector=smallrye-kafka\nmp.messaging.outgoing.lineage-event-out.topic=lineage-event-out\nmp.messaging.outgoing.lineage-event-out.key.serializer=org.apache.kafka.common.serialization.StringSerializer\nmp.messaging.outgoing.lineage-event-out.value.serializer=org.apache.kafka.common.serialization.StringSerializer\n```\n\nYou can also use your own microprofile-config.properties file. To do that, you must provide your own property path to the code:\n\n1. Set the path of the microprofile-config.properties on your file system\n```python\n # specify the new microprofile-config properties path\n self.emitter.set_messaging_properties_path(string_path_to_the_property_file)\n\n```\n2. Set the new emitter topic \n add below property to the data-lineage.properties file of the appropriate docker image (e.g. spark-worker)\n```text\ndata.lineage.emission.topic=replace-with-your-topic\n```\n\n3. (Optional) If you are using the v1 kafka-cluster helm chart, add the new topic to kafka-cluster values.yaml file; e.g.: add `replace-with-your-topic:1:1` to the **KAFKA_CREATE_TOPICS** environment variable in the `-deploy/src/main/resources/apps/kafka-cluster/values.yaml`\n\n## NOTE:\nThere is a known issue with the confluent-kafka dependency that is brought in by the openlineage-client library that prevents it from installing this requirement on some arm64 environments. The problem is that librdkafka does not provide a wheel for all operating systems in the arm64 architecture, requiring that it be built from source in order for confluent-kafka to be installed. If you run into this problem, you can install it from source using the below commands:\n\n* apt-get update\n* apt install -y python3-dev git\n* git clone https://github.com/confluentinc/librdkafka.git\n* cd librdkafka\n* ./configure --arch=aarch64\n* make\n* make install",
"bugtrack_url": null,
"license": null,
"summary": "Data lineage API for Python scripts",
"version": "1.9.2",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e035916a1c3f78ca38b45f634693eb78fed6da37aec166ba63ce62ece6f1c87b",
"md5": "a2e9f7864ba9347bf5a14d4aed3748f4",
"sha256": "bad6f40e0e06620f10104d3c3eeb252a985ce17434c1c54178ab7b7355f576e5"
},
"downloads": -1,
"filename": "aissemble_foundation_data_lineage_python-1.9.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a2e9f7864ba9347bf5a14d4aed3748f4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.11.4",
"size": 18090,
"upload_time": "2024-09-26T20:39:51",
"upload_time_iso_8601": "2024-09-26T20:39:51.058915Z",
"url": "https://files.pythonhosted.org/packages/e0/35/916a1c3f78ca38b45f634693eb78fed6da37aec166ba63ce62ece6f1c87b/aissemble_foundation_data_lineage_python-1.9.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3b15822916f9f7f9104202223cee6d15b5ed546f8b04c71b0c68156c0c124d85",
"md5": "a0666ec3bfe0c856dff5236abf2742e3",
"sha256": "7c2088c0741a8fcf33760d1a832dfbce729943197408fc9437ae10229c3d2026"
},
"downloads": -1,
"filename": "aissemble_foundation_data_lineage_python-1.9.2.tar.gz",
"has_sig": false,
"md5_digest": "a0666ec3bfe0c856dff5236abf2742e3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.11.4",
"size": 13393,
"upload_time": "2024-09-26T20:39:52",
"upload_time_iso_8601": "2024-09-26T20:39:52.029949Z",
"url": "https://files.pythonhosted.org/packages/3b/15/822916f9f7f9104202223cee6d15b5ed546f8b04c71b0c68156c0c124d85/aissemble_foundation_data_lineage_python-1.9.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-26 20:39:52",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "aissemble-foundation-data-lineage-python"
}
JSON