# spacy-matching
## Background
In Germany's cancer registries, substances are reported in a free text field, e.g., "Interferon alpha-2a weekly i.v.".
The reported text might include additional information such as dosage or labels or typos.
For analysis, it is helpful to have substances reported in a harmonized way. The aim of this Python function is extracting the substance name from the free text field. The script uses the FuzzyMatcher from spaczz and spacy to scan each free text for potential matches.
## Usage
Call `add_substance()` with these arguments as Series:
- `col_with_substances` - **original** substance name (freetext)
- `col_with_ref_substances` - **reference** list of substances
Return:
- column with **recoded** value, i.e the substance name that is deemed most likely
- column with **similarity** score
```python
from spacy_matching import recoding as rec
# * get substance reference table from public repo
URL_V2 = "https://gitlab.opencode.de/robert-koch-institut/zentrum-fuer-krebsregisterdaten/cancerdata-references/-/raw/main/data/v2/Klassifikationen/substanz.csv?ref_type=heads"
reference_list = pd.read_csv(URL_V2, sep=";")
# * create a pandaSeries with some test data
fake_data = pd.Series(["Interferon alpha 2a", "Paclitaxel (nab)", "Filgrastim", "Leuprorelin; Tamoxifen"])
# * add a column with recoded substances
results = rec.add_substance(
col_with_substances=fake_data,
col_with_ref_substances=reference_list["substanz"],
only_first_match=True,
threshold=0.85
)
```
## Options and parameters
### threshold
The function features some parameters: The `threshold` parameter defines the accuracy. The lower the more matches but higher
values lead to more accurate matches. The graph below plots the number of substances extracted from free text fields against the corresponding threshold parameter. There is a tradeoff between the number of matches and their accuracy. A threshold value of 0.85 is set as default as it usually ensures sufficient accuracy.
### only_first_match
The option `only_first_match` = True should be used if the user wants to allow only one match per free text field.
Even if there are several substances in the free text field such as "Leuprorelin; Tamoxifen", the function will return only the first match.
If the option is set to "False", the function can return multiple hits. For the input "Interferon alpha-2a weekly i.v.", the function might return two substances "Interferon alpha-2a" and "Interferon alpha-2b" (assuming they are both on the reference list). Results based on both options is shown below.
## Credits
Thanks to @smeisegeier for helpful feedback on the code and for making the function available as a python package.
Raw data
{
"_id": null,
"home_page": null,
"name": "spacy-matching",
"maintainer": "smeisegeier",
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": "smeisegeier <meisegeiers@rki.de>",
"keywords": "matching, package, cancerdata",
"author": "msauerberg",
"author_email": "msauerberg <markus.sauerberg@bwfgb.hamburg.de>",
"download_url": "https://files.pythonhosted.org/packages/49/3d/b9111468ac539adc354eaf76bf6808c3833b72c8d8556230cc2814b7cad6/spacy_matching-0.5.0.tar.gz",
"platform": null,
"description": "# spacy-matching\n\n\n\n## Background\n\nIn Germany's cancer registries, substances are reported in a free text field, e.g., \"Interferon alpha-2a weekly i.v.\".\nThe reported text might include additional information such as dosage or labels or typos.\nFor analysis, it is helpful to have substances reported in a harmonized way. The aim of this Python function is extracting the substance name from the free text field. The script uses the FuzzyMatcher from spaczz and spacy to scan each free text for potential matches.\n\n## Usage\n\nCall `add_substance()` with these arguments as Series:\n\n- `col_with_substances` - **original** substance name (freetext)\n- `col_with_ref_substances` - **reference** list of substances\n\nReturn:\n\n- column with **recoded** value, i.e the substance name that is deemed most likely\n- column with **similarity** score\n\n```python\nfrom spacy_matching import recoding as rec\n\n# * get substance reference table from public repo\nURL_V2 = \"https://gitlab.opencode.de/robert-koch-institut/zentrum-fuer-krebsregisterdaten/cancerdata-references/-/raw/main/data/v2/Klassifikationen/substanz.csv?ref_type=heads\"\nreference_list = pd.read_csv(URL_V2, sep=\";\")\n\n# * create a pandaSeries with some test data\nfake_data = pd.Series([\"Interferon alpha 2a\", \"Paclitaxel (nab)\", \"Filgrastim\", \"Leuprorelin; Tamoxifen\"])\n\n# * add a column with recoded substances\nresults = rec.add_substance(\n col_with_substances=fake_data,\n col_with_ref_substances=reference_list[\"substanz\"],\n only_first_match=True,\n threshold=0.85\n)\n```\n\n## Options and parameters\n\n### threshold\n\nThe function features some parameters: The `threshold` parameter defines the accuracy. The lower the more matches but higher\nvalues lead to more accurate matches. The graph below plots the number of substances extracted from free text fields against the corresponding threshold parameter. There is a tradeoff between the number of matches and their accuracy. A threshold value of 0.85 is set as default as it usually ensures sufficient accuracy.\n\n\n\n### only_first_match\n\nThe option `only_first_match` = True should be used if the user wants to allow only one match per free text field.\nEven if there are several substances in the free text field such as \"Leuprorelin; Tamoxifen\", the function will return only the first match.\nIf the option is set to \"False\", the function can return multiple hits. For the input \"Interferon alpha-2a weekly i.v.\", the function might return two substances \"Interferon alpha-2a\" and \"Interferon alpha-2b\" (assuming they are both on the reference list). Results based on both options is shown below.\n\n\n\n## Credits\n\nThanks to @smeisegeier for helpful feedback on the code and for making the function available as a python package.\n",
"bugtrack_url": null,
"license": null,
"summary": "Add your description here",
"version": "0.5.0",
"project_urls": {
"Bug Tracker": "https://github.com/msauerberg/spacy_matching/issues",
"Homepage": "https://github.com/msauerberg/spacy_matching",
"Repository": "https://github.com/msauerberg/spacy_matching"
},
"split_keywords": [
"matching",
" package",
" cancerdata"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "59215d97375a9308923ebafbd62dff2057b573aa792fd4cf6ad7f39231715133",
"md5": "8027ee421cff0efd7fde713512b73418",
"sha256": "902f715caf36daa4759528db153461d1d6dbd04c854a6f2724a8f0c8118226ed"
},
"downloads": -1,
"filename": "spacy_matching-0.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8027ee421cff0efd7fde713512b73418",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 9230,
"upload_time": "2025-08-14T14:41:38",
"upload_time_iso_8601": "2025-08-14T14:41:38.055442Z",
"url": "https://files.pythonhosted.org/packages/59/21/5d97375a9308923ebafbd62dff2057b573aa792fd4cf6ad7f39231715133/spacy_matching-0.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "493db9111468ac539adc354eaf76bf6808c3833b72c8d8556230cc2814b7cad6",
"md5": "ee4f204ff276ac07f2a1f063db5fcca1",
"sha256": "c1feb4524740d3a9c26fd3f07647bf030b84e42628c89fe110b8f2adc820ccb6"
},
"downloads": -1,
"filename": "spacy_matching-0.5.0.tar.gz",
"has_sig": false,
"md5_digest": "ee4f204ff276ac07f2a1f063db5fcca1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 7553,
"upload_time": "2025-08-14T14:41:39",
"upload_time_iso_8601": "2025-08-14T14:41:39.097755Z",
"url": "https://files.pythonhosted.org/packages/49/3d/b9111468ac539adc354eaf76bf6808c3833b72c8d8556230cc2814b7cad6/spacy_matching-0.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-14 14:41:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "msauerberg",
"github_project": "spacy_matching",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "spacy-matching"
}
JSON
