# datastew
 
Datastew is a python library for intelligent data harmonization using Large Language Model (LLM) vector embeddings.
## Installation
```bash
pip install datastew
```
## Usage
### Harmonizing excel/csv resources
You can directly import common data models, terminology sources or data dictionaries for harmonization directly from a
csv, tsv or excel file. An example how to match two separate variable descriptions is shown in
[datastew/scripts/mapping_excel_example.py](datastew/scripts/mapping_excel_example.py):
```python
from datastew.process.parsing import DataDictionarySource
from datastew.process.mapping import map_dictionary_to_dictionary
# Variable and description refer to the corresponding column names in your excel sheet
source = DataDictionarySource("source.xlxs", variable_field="var", description_field="desc")
target = DataDictionarySource("target.xlxs", variable_field="var", description_field="desc")
df = map_dictionary_to_dictionary(source, target)
df.to_excel("result.xlxs")
```
The resulting file contains the pairwise variable mapping based on the closest similarity for all possible matches
as well as a similarity measure per row.
Per default this will use the local MPNet model, which may not yield the optimal performance. If you got an OpenAI API
key it is possible to use their embedding API instead. To use your key, create an OpenAIAdapter model and pass it to the
function:
```python
from datastew.embedding import GPT4Adapter
embedding_model = GPT4Adapter(key="your_api_key")
df = map_dictionary_to_dictionary(source, target, embedding_model=embedding_model)
```
You can also retrieve embeddings from data dictionaries and visualize them in form of an interactive scatter plot to
explore sematic neighborhoods:
```python
from datastew.visualisation import plot_embeddings
# Get embedding vectors for your dictionaries
source_embeddings = source.get_embeddings()
# plot embedding neighborhoods for several dictionaries
plot_embeddings(data_dictionaries=[source, target])
```
### Creating and using stored mappings
A simple example how to initialize an in memory database and compute a similarity mapping is shown in
[datastew/scripts/mapping_db_example.py](datastew/scripts/mapping_db_example.py):
```python
from datastew.repository.sqllite import SQLLiteRepository
from datastew.repository.model import Terminology, Concept, Mapping
from datastew.embedding import MPNetAdapter
# omit mode to create a permanent db file instead
repository = SQLLiteRepository(mode="memory")
embedding_model = MPNetAdapter()
terminology = Terminology("snomed CT", "SNOMED")
text1 = "Diabetes mellitus (disorder)"
concept1 = Concept(terminology, text1, "Concept ID: 11893007")
mapping1 = Mapping(concept1, text1, embedding_model.get_embedding(text1))
text2 = "Hypertension (disorder)"
concept2 = Concept(terminology, text2, "Concept ID: 73211009")
mapping2 = Mapping(concept2, text2, embedding_model.get_embedding(text2))
repository.store_all([terminology, concept1, mapping1, concept2, mapping2])
text_to_map = "Sugar sickness"
embedding = embedding_model.get_embedding(text_to_map)
mappings, similarities = repository.get_closest_mappings(embedding, limit=2)
for mapping, similarity in zip(mappings, similarities):
print(f"Similarity: {similarity} -> {mapping}")
```
output:
```plaintext
Similarity: 0.47353370635583486 -> Concept ID: 11893007 : Diabetes mellitus (disorder) | Diabetes mellitus (disorder)
Similarity: 0.20031612264852067 -> Concept ID: 73211009 : Hypertension (disorder) | Hypertension (disorder)
```
You can also import data from file sources (csv, tsv, xlsx) or from a public API like OLS. An example script to
download & compute embeddings for SNOMED from ebi OLS can be found in
[datastew/scripts/ols_snomed_retrieval.py](datastew/scripts/ols_snomed_retrieval.py).
### Embedding visualization
You can visualize the embedding space of multiple data dictionary sources with t-SNE plots utilizing different
language models. An example how to generate a t-sne plot is shown in
[datastew/scripts/tsne_visualization.py](datastew/scripts/tsne_visualization.py):
```python
from datastew.embedding import MPNetAdapter
from datastew.process.parsing import DataDictionarySource
from datastew.visualisation import plot_embeddings
# Variable and description refer to the corresponding column names in your excel sheet
data_dictionary_source_1 = DataDictionarySource(
"source1.xlsx", variable_field="var", description_field="desc"
)
data_dictionary_source_2 = DataDictionarySource(
"source2.xlsx", variable_field="var", description_field="desc"
)
mpnet_adapter = MPNetAdapter()
plot_embeddings(
[data_dictionary_source_1, data_dictionary_source_2], embedding_model=mpnet_adapter
)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/SCAI-BIO/datastew",
"name": "datastew",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "data-harmonization LLM embeddings data-steward",
"author": "Tim Adams",
"author_email": "tim.adams@scai.fraunhofer.de",
"download_url": "https://files.pythonhosted.org/packages/13/8c/7d06a2a62de7e4631e9afcff36fd6c33102354669e3846eef7d98ba5b860/datastew-0.4.1.tar.gz",
"platform": null,
"description": "\n# datastew\n\n \n\nDatastew is a python library for intelligent data harmonization using Large Language Model (LLM) vector embeddings.\n\n## Installation\n\n```bash\npip install datastew\n```\n\n## Usage\n\n### Harmonizing excel/csv resources\n\nYou can directly import common data models, terminology sources or data dictionaries for harmonization directly from a\ncsv, tsv or excel file. An example how to match two separate variable descriptions is shown in\n[datastew/scripts/mapping_excel_example.py](datastew/scripts/mapping_excel_example.py):\n\n```python\nfrom datastew.process.parsing import DataDictionarySource\nfrom datastew.process.mapping import map_dictionary_to_dictionary\n\n# Variable and description refer to the corresponding column names in your excel sheet\nsource = DataDictionarySource(\"source.xlxs\", variable_field=\"var\", description_field=\"desc\")\ntarget = DataDictionarySource(\"target.xlxs\", variable_field=\"var\", description_field=\"desc\")\n\ndf = map_dictionary_to_dictionary(source, target)\ndf.to_excel(\"result.xlxs\")\n```\n\nThe resulting file contains the pairwise variable mapping based on the closest similarity for all possible matches\nas well as a similarity measure per row.\n\nPer default this will use the local MPNet model, which may not yield the optimal performance. If you got an OpenAI API\nkey it is possible to use their embedding API instead. To use your key, create an OpenAIAdapter model and pass it to the\nfunction:\n\n```python\nfrom datastew.embedding import GPT4Adapter\n\nembedding_model = GPT4Adapter(key=\"your_api_key\")\ndf = map_dictionary_to_dictionary(source, target, embedding_model=embedding_model)\n```\n\nYou can also retrieve embeddings from data dictionaries and visualize them in form of an interactive scatter plot to\nexplore sematic neighborhoods:\n\n```python\nfrom datastew.visualisation import plot_embeddings\n\n# Get embedding vectors for your dictionaries\nsource_embeddings = source.get_embeddings()\n\n# plot embedding neighborhoods for several dictionaries\nplot_embeddings(data_dictionaries=[source, target])\n\n```\n\n### Creating and using stored mappings\n\nA simple example how to initialize an in memory database and compute a similarity mapping is shown in\n[datastew/scripts/mapping_db_example.py](datastew/scripts/mapping_db_example.py):\n\n```python\nfrom datastew.repository.sqllite import SQLLiteRepository\nfrom datastew.repository.model import Terminology, Concept, Mapping\nfrom datastew.embedding import MPNetAdapter\n\n# omit mode to create a permanent db file instead\nrepository = SQLLiteRepository(mode=\"memory\")\nembedding_model = MPNetAdapter()\n\nterminology = Terminology(\"snomed CT\", \"SNOMED\")\n\ntext1 = \"Diabetes mellitus (disorder)\"\nconcept1 = Concept(terminology, text1, \"Concept ID: 11893007\")\nmapping1 = Mapping(concept1, text1, embedding_model.get_embedding(text1))\n\ntext2 = \"Hypertension (disorder)\"\nconcept2 = Concept(terminology, text2, \"Concept ID: 73211009\")\nmapping2 = Mapping(concept2, text2, embedding_model.get_embedding(text2))\n\nrepository.store_all([terminology, concept1, mapping1, concept2, mapping2])\n\ntext_to_map = \"Sugar sickness\"\nembedding = embedding_model.get_embedding(text_to_map)\nmappings, similarities = repository.get_closest_mappings(embedding, limit=2)\nfor mapping, similarity in zip(mappings, similarities):\n print(f\"Similarity: {similarity} -> {mapping}\")\n```\n\noutput:\n\n```plaintext\nSimilarity: 0.47353370635583486 -> Concept ID: 11893007 : Diabetes mellitus (disorder) | Diabetes mellitus (disorder)\nSimilarity: 0.20031612264852067 -> Concept ID: 73211009 : Hypertension (disorder) | Hypertension (disorder)\n```\n\nYou can also import data from file sources (csv, tsv, xlsx) or from a public API like OLS. An example script to\ndownload & compute embeddings for SNOMED from ebi OLS can be found in\n[datastew/scripts/ols_snomed_retrieval.py](datastew/scripts/ols_snomed_retrieval.py).\n\n### Embedding visualization\n\nYou can visualize the embedding space of multiple data dictionary sources with t-SNE plots utilizing different\nlanguage models. An example how to generate a t-sne plot is shown in\n[datastew/scripts/tsne_visualization.py](datastew/scripts/tsne_visualization.py):\n\n```python\nfrom datastew.embedding import MPNetAdapter\nfrom datastew.process.parsing import DataDictionarySource\nfrom datastew.visualisation import plot_embeddings\n\n# Variable and description refer to the corresponding column names in your excel sheet\ndata_dictionary_source_1 = DataDictionarySource(\n \"source1.xlsx\", variable_field=\"var\", description_field=\"desc\"\n)\ndata_dictionary_source_2 = DataDictionarySource(\n \"source2.xlsx\", variable_field=\"var\", description_field=\"desc\"\n)\n\nmpnet_adapter = MPNetAdapter()\nplot_embeddings(\n [data_dictionary_source_1, data_dictionary_source_2], embedding_model=mpnet_adapter\n)\n```\n\n",
"bugtrack_url": null,
"license": "Apache-2.0 license",
"summary": "Intelligent data steward toolbox using Large Language Model embeddings for automated Data-Harmonization.",
"version": "0.4.1",
"project_urls": {
"Documentation": "https://github.com/SCAI-BIO/datastew#readme",
"Homepage": "https://github.com/SCAI-BIO/datastew",
"Source": "https://github.com/SCAI-BIO/datastew",
"Tracker": "https://github.com/SCAI-BIO/datastew/issues"
},
"split_keywords": [
"data-harmonization",
"llm",
"embeddings",
"data-steward"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1cbb9a9c63cad13174f424fc7884d0b2721f28cc35ff66776f6749ffb1efe138",
"md5": "94b160983da5fe9279122cd7a0bc4dcf",
"sha256": "ffdee44ee499e54e3c80c8e5566fb27a25a7b1daa46695dd0b5db49a19240873"
},
"downloads": -1,
"filename": "datastew-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "94b160983da5fe9279122cd7a0bc4dcf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 39124,
"upload_time": "2024-11-13T12:53:15",
"upload_time_iso_8601": "2024-11-13T12:53:15.827728Z",
"url": "https://files.pythonhosted.org/packages/1c/bb/9a9c63cad13174f424fc7884d0b2721f28cc35ff66776f6749ffb1efe138/datastew-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "138c7d06a2a62de7e4631e9afcff36fd6c33102354669e3846eef7d98ba5b860",
"md5": "2b58f62b77923ddea6f916d977e4cc88",
"sha256": "9a5a696c965fa66bd4e8ebc083d0152b65c70dbc880f220503f16d2963b2d602"
},
"downloads": -1,
"filename": "datastew-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "2b58f62b77923ddea6f916d977e4cc88",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 32770,
"upload_time": "2024-11-13T12:53:17",
"upload_time_iso_8601": "2024-11-13T12:53:17.490338Z",
"url": "https://files.pythonhosted.org/packages/13/8c/7d06a2a62de7e4631e9afcff36fd6c33102354669e3846eef7d98ba5b860/datastew-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-13 12:53:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SCAI-BIO",
"github_project": "datastew",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "matplotlib",
"specs": [
[
"~=",
"3.8.1"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.25.2"
]
]
},
{
"name": "openai",
"specs": [
[
"~=",
"0.28.0"
]
]
},
{
"name": "openpyxl",
"specs": []
},
{
"name": "pandas",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "pip",
"specs": [
[
"==",
"21.3.1"
]
]
},
{
"name": "plotly",
"specs": [
[
"~=",
"5.17.0"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"~=",
"1.0.0"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2023.3"
]
]
},
{
"name": "seaborn",
"specs": [
[
"~=",
"0.13.0"
]
]
},
{
"name": "sentence-transformers",
"specs": [
[
"==",
"2.3.1"
]
]
},
{
"name": "setuptools",
"specs": [
[
"==",
"60.2.0"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
"==",
"1.3.2"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "thefuzz",
"specs": [
[
"~=",
"0.20.0"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2023.3"
]
]
},
{
"name": "wheel",
"specs": [
[
"==",
"0.37.1"
]
]
},
{
"name": "aiofiles",
"specs": [
[
"~=",
"0.7.0"
]
]
},
{
"name": "SQLAlchemy",
"specs": [
[
"~=",
"2.0.27"
]
]
},
{
"name": "scipy",
"specs": [
[
"~=",
"1.11.4"
]
]
},
{
"name": "pydantic",
"specs": [
[
"~=",
"2.5.0"
]
]
},
{
"name": "weaviate-client",
"specs": [
[
"~=",
"4.9.0"
]
]
}
],
"lcname": "datastew"
}
JSON
