# Concise Concepts
When wanting to apply NER to concise concepts, it is really easy to come up with examples, but pretty difficult to train an entire pipeline. Concise Concepts uses few-shot NER based on word embedding similarity to get you going
with easy! Now with entity scoring!
[](https://github.com/Pandora-Intelligence/concise-concepts/actions/workflows/python-package.yml)
[](https://github.com/pandora-intelligence/concise-concepts/releases)
[](https://pypi.org/project/concise-concepts/)
[](https://pypi.org/project/concise-concepts/)
[](https://github.com/ambv/black)
## Usage
This library defines matching patterns based on the most similar words found in each group, which are used to fill a [spaCy EntityRuler](https://spacy.io/api/entityruler). To better understand the rule definition, I recommend playing around with the [spaCy Rule-based Matcher Explorer](https://demos.explosion.ai/matcher).
### Tutorials
- [TechVizTheDataScienceGuy](https://www.youtube.com/c/TechVizTheDataScienceGuy) created a [nice tutorial](https://prakhar-mishra.medium.com/few-shot-named-entity-recognition-in-natural-language-processing-92d31f0d1143) on how to use it.
- [I](https://www.linkedin.com/in/david-berenstein-1bab11105/) created a [tutorial](https://www.rubrix.ml/blog/concise-concepts-rubrix/) in collaboration with Rubrix.
The section [Matching Pattern Rules](#matching-pattern-rules) expands on the construction, analysis and customization of these matching patterns.
# Install
```
pip install concise-concepts
```
# Quickstart
Take a look at the [configuration section](#configuration) for more info.
## Spacy Pipeline Component
Note that, [custom embedding models](#custom-embedding-models) are passed via `model_path`.
```python
import spacy
from spacy import displacy
import concise_concepts
data = {
"fruit": ["apple", "pear", "orange"],
"vegetable": ["broccoli", "spinach", "tomato"],
"meat": ['beef', 'pork', 'turkey', 'duck']
}
text = """
Heat the oil in a large pan and add the Onion, celery and carrots.
Then, cook over a medium–low heat for 10 minutes, or until softened.
Add the courgette, garlic, red peppers and oregano and cook for 2–3 minutes.
Later, add some oranges and chickens. """
nlp = spacy.load("en_core_web_md", disable=["ner"])
nlp.add_pipe(
"concise_concepts",
config={
"data": data,
"ent_score": True, # Entity Scoring section
"verbose": True,
"exclude_pos": ["VERB", "AUX"],
"exclude_dep": ["DOBJ", "PCOMP"],
"include_compound_words": False,
"json_path": "./fruitful_patterns.json",
"topn": (100,500,300)
},
)
doc = nlp(text)
options = {
"colors": {"fruit": "darkorange", "vegetable": "limegreen", "meat": "salmon"},
"ents": ["fruit", "vegetable", "meat"],
}
ents = doc.ents
for ent in ents:
new_label = f"{ent.label_} ({ent._.ent_score:.0%})"
options["colors"][new_label] = options["colors"].get(ent.label_.lower(), None)
options["ents"].append(new_label)
ent.label_ = new_label
doc.ents = ents
displacy.render(doc, style="ent", options=options)
```
## Standalone
This might be useful when iterating over few_shot training data when not wanting to reload larger models continuously.
Note that, [custom embedding models](#custom-embedding-models) are passed via `model`.
```python
import gensim
import spacy
from concise_concepts import Conceptualizer
model = gensim.downloader.load("fasttext-wiki-news-subwords-300")
nlp = spacy.load("en_core_web_sm")
data = {
"disease": ["cancer", "diabetes", "heart disease", "influenza", "pneumonia"],
"symptom": ["headache", "fever", "cough", "nausea", "vomiting", "diarrhea"],
}
conceptualizer = Conceptualizer(nlp, data, model)
conceptualizer.nlp("I have a headache and a fever.").ents
data = {
"disease": ["cancer", "diabetes"],
"symptom": ["headache", "fever"],
}
conceptualizer = Conceptualizer(nlp, data, model)
conceptualizer.nlp("I have a headache and a fever.").ents
```
# Configuration
## Matching Pattern Rules
A general introduction about the usage of matching patterns in the [usage section](#usage).
### Customizing Matching Pattern Rules
Even though the baseline parameters provide a decent result, the construction of these matching rules can be customized via the config passed to the spaCy pipeline.
- `exclude_pos`: A list of POS tags to be excluded from the rule-based match.
- `exclude_dep`: A list of dependencies to be excluded from the rule-based match.
- `include_compound_words`: If True, it will include compound words in the entity. For example, if the entity is "New York", it will also include "New York City" as an entity.
- `case_sensitive`: Whether to match the case of the words in the text.
### Analyze Matching Pattern Rules
To motivate actually looking at the data and support interpretability, the matching patterns that have been generated are stored as `./main_patterns.json`. This behavior can be changed by using the `json_path` variable via the config passed to the spaCy pipeline.
## Fuzzy matching using `spaczz`
- `fuzzy`: A boolean value that determines whether to use fuzzy matching
```python
data = {
"fruit": ["apple", "pear", "orange"],
"vegetable": ["broccoli", "spinach", "tomato"],
"meat": ["beef", "pork", "fish", "lamb"]
}
nlp.add_pipe("concise_concepts", config={"data": data, "fuzzy": True})
```
## Most Similar Word Expansion
- `topn`: Use a specific number of words to expand over.
```python
data = {
"fruit": ["apple", "pear", "orange"],
"vegetable": ["broccoli", "spinach", "tomato"],
"meat": ["beef", "pork", "fish", "lamb"]
}
topn = [50, 50, 150]
assert len(topn) == len
nlp.add_pipe("concise_concepts", config={"data": data, "topn": topn})
```
## Entity Scoring
- `ent_score`: Use embedding based word similarity to score entities against their groups
```python
import spacy
import concise_concepts
data = {
"ORG": ["Google", "Apple", "Amazon"],
"GPE": ["Netherlands", "France", "China"],
}
text = """Sony was founded in Japan."""
nlp = spacy.load("en_core_web_lg")
nlp.add_pipe("concise_concepts", config={"data": data, "ent_score": True, "case_sensitive": True})
doc = nlp(text)
print([(ent.text, ent.label_, ent._.ent_score) for ent in doc.ents])
# output
#
# [('Sony', 'ORG', 0.5207586), ('Japan', 'GPE', 0.7371268)]
```
## Custom Embedding Models
- `model_path`: Use custom `sense2vec.Sense2Vec`, `gensim.Word2vec` `gensim.FastText`, or `gensim.KeyedVectors`, or a pretrained model from [gensim](https://radimrehurek.com/gensim/downloader.html) library or a custom model path. For using a `sense2vec.Sense2Vec` take a look [here](https://github.com/explosion/sense2vec#pretrained-vectors).
- `model`: within [standalone usage](#standalone), it is possible to pass these models directly.
```python
data = {
"fruit": ["apple", "pear", "orange"],
"vegetable": ["broccoli", "spinach", "tomato"],
"meat": ["beef", "pork", "fish", "lamb"]
}
# model from https://radimrehurek.com/gensim/downloader.html or path to local file
model_path = "glove-wiki-gigaword-300"
nlp.add_pipe("concise_concepts", config={"data": data, "model_path": model_path})
````
Raw data
{
"_id": null,
"home_page": "https://github.com/pandora-intelligence/concise-concepts",
"name": "concise-concepts",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.12",
"maintainer_email": "",
"keywords": "spacy,NER,few-shot classification,nlu",
"author": "David Berenstein",
"author_email": "david.m.berenstein@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/3b/53/51e162b39416ede8847622343aac98c537822011c657e1ec63130dcfcbc4/concise-concepts-0.8.1.tar.gz",
"platform": null,
"description": "# Concise Concepts\nWhen wanting to apply NER to concise concepts, it is really easy to come up with examples, but pretty difficult to train an entire pipeline. Concise Concepts uses few-shot NER based on word embedding similarity to get you going\nwith easy! Now with entity scoring!\n\n\n[](https://github.com/Pandora-Intelligence/concise-concepts/actions/workflows/python-package.yml)\n[](https://github.com/pandora-intelligence/concise-concepts/releases)\n[](https://pypi.org/project/concise-concepts/)\n[](https://pypi.org/project/concise-concepts/)\n[](https://github.com/ambv/black)\n\n\n## Usage\nThis library defines matching patterns based on the most similar words found in each group, which are used to fill a [spaCy EntityRuler](https://spacy.io/api/entityruler). To better understand the rule definition, I recommend playing around with the [spaCy Rule-based Matcher Explorer](https://demos.explosion.ai/matcher).\n\n### Tutorials\n- [TechVizTheDataScienceGuy](https://www.youtube.com/c/TechVizTheDataScienceGuy) created a [nice tutorial](https://prakhar-mishra.medium.com/few-shot-named-entity-recognition-in-natural-language-processing-92d31f0d1143) on how to use it.\n\n- [I](https://www.linkedin.com/in/david-berenstein-1bab11105/) created a [tutorial](https://www.rubrix.ml/blog/concise-concepts-rubrix/) in collaboration with Rubrix.\n\nThe section [Matching Pattern Rules](#matching-pattern-rules) expands on the construction, analysis and customization of these matching patterns.\n\n\n# Install\n\n```\npip install concise-concepts\n```\n\n# Quickstart\n\nTake a look at the [configuration section](#configuration) for more info.\n\n## Spacy Pipeline Component\n\nNote that, [custom embedding models](#custom-embedding-models) are passed via `model_path`.\n\n```python\nimport spacy\nfrom spacy import displacy\n\nimport concise_concepts\n\ndata = {\n \"fruit\": [\"apple\", \"pear\", \"orange\"],\n \"vegetable\": [\"broccoli\", \"spinach\", \"tomato\"],\n \"meat\": ['beef', 'pork', 'turkey', 'duck']\n}\n\ntext = \"\"\"\n Heat the oil in a large pan and add the Onion, celery and carrots.\n Then, cook over a medium\u2013low heat for 10 minutes, or until softened.\n Add the courgette, garlic, red peppers and oregano and cook for 2\u20133 minutes.\n Later, add some oranges and chickens. \"\"\"\n\nnlp = spacy.load(\"en_core_web_md\", disable=[\"ner\"])\n\nnlp.add_pipe(\n \"concise_concepts\",\n config={\n \"data\": data,\n \"ent_score\": True, # Entity Scoring section\n \"verbose\": True,\n \"exclude_pos\": [\"VERB\", \"AUX\"],\n \"exclude_dep\": [\"DOBJ\", \"PCOMP\"],\n \"include_compound_words\": False,\n \"json_path\": \"./fruitful_patterns.json\",\n \"topn\": (100,500,300)\n },\n)\ndoc = nlp(text)\n\noptions = {\n \"colors\": {\"fruit\": \"darkorange\", \"vegetable\": \"limegreen\", \"meat\": \"salmon\"},\n \"ents\": [\"fruit\", \"vegetable\", \"meat\"],\n}\n\nents = doc.ents\nfor ent in ents:\n new_label = f\"{ent.label_} ({ent._.ent_score:.0%})\"\n options[\"colors\"][new_label] = options[\"colors\"].get(ent.label_.lower(), None)\n options[\"ents\"].append(new_label)\n ent.label_ = new_label\ndoc.ents = ents\n\ndisplacy.render(doc, style=\"ent\", options=options)\n```\n\n\n## Standalone\n\nThis might be useful when iterating over few_shot training data when not wanting to reload larger models continuously.\nNote that, [custom embedding models](#custom-embedding-models) are passed via `model`.\n\n```python\nimport gensim\nimport spacy\n\nfrom concise_concepts import Conceptualizer\n\nmodel = gensim.downloader.load(\"fasttext-wiki-news-subwords-300\")\nnlp = spacy.load(\"en_core_web_sm\")\ndata = {\n \"disease\": [\"cancer\", \"diabetes\", \"heart disease\", \"influenza\", \"pneumonia\"],\n \"symptom\": [\"headache\", \"fever\", \"cough\", \"nausea\", \"vomiting\", \"diarrhea\"],\n}\nconceptualizer = Conceptualizer(nlp, data, model)\nconceptualizer.nlp(\"I have a headache and a fever.\").ents\n\ndata = {\n \"disease\": [\"cancer\", \"diabetes\"],\n \"symptom\": [\"headache\", \"fever\"],\n}\nconceptualizer = Conceptualizer(nlp, data, model)\nconceptualizer.nlp(\"I have a headache and a fever.\").ents\n```\n\n# Configuration\n## Matching Pattern Rules\nA general introduction about the usage of matching patterns in the [usage section](#usage).\n### Customizing Matching Pattern Rules\nEven though the baseline parameters provide a decent result, the construction of these matching rules can be customized via the config passed to the spaCy pipeline.\n\n - `exclude_pos`: A list of POS tags to be excluded from the rule-based match.\n - `exclude_dep`: A list of dependencies to be excluded from the rule-based match.\n - `include_compound_words`: If True, it will include compound words in the entity. For example, if the entity is \"New York\", it will also include \"New York City\" as an entity.\n - `case_sensitive`: Whether to match the case of the words in the text.\n\n\n### Analyze Matching Pattern Rules\nTo motivate actually looking at the data and support interpretability, the matching patterns that have been generated are stored as `./main_patterns.json`. This behavior can be changed by using the `json_path` variable via the config passed to the spaCy pipeline.\n\n\n## Fuzzy matching using `spaczz`\n\n - `fuzzy`: A boolean value that determines whether to use fuzzy matching\n\n```python\ndata = {\n \"fruit\": [\"apple\", \"pear\", \"orange\"],\n \"vegetable\": [\"broccoli\", \"spinach\", \"tomato\"],\n \"meat\": [\"beef\", \"pork\", \"fish\", \"lamb\"]\n}\n\nnlp.add_pipe(\"concise_concepts\", config={\"data\": data, \"fuzzy\": True})\n```\n\n## Most Similar Word Expansion\n\n- `topn`: Use a specific number of words to expand over.\n\n```python\ndata = {\n \"fruit\": [\"apple\", \"pear\", \"orange\"],\n \"vegetable\": [\"broccoli\", \"spinach\", \"tomato\"],\n \"meat\": [\"beef\", \"pork\", \"fish\", \"lamb\"]\n}\n\ntopn = [50, 50, 150]\n\nassert len(topn) == len\n\nnlp.add_pipe(\"concise_concepts\", config={\"data\": data, \"topn\": topn})\n```\n\n## Entity Scoring\n\n- `ent_score`: Use embedding based word similarity to score entities against their groups\n\n```python\nimport spacy\nimport concise_concepts\n\ndata = {\n \"ORG\": [\"Google\", \"Apple\", \"Amazon\"],\n \"GPE\": [\"Netherlands\", \"France\", \"China\"],\n}\n\ntext = \"\"\"Sony was founded in Japan.\"\"\"\n\nnlp = spacy.load(\"en_core_web_lg\")\nnlp.add_pipe(\"concise_concepts\", config={\"data\": data, \"ent_score\": True, \"case_sensitive\": True})\ndoc = nlp(text)\n\nprint([(ent.text, ent.label_, ent._.ent_score) for ent in doc.ents])\n# output\n#\n# [('Sony', 'ORG', 0.5207586), ('Japan', 'GPE', 0.7371268)]\n```\n\n## Custom Embedding Models\n\n- `model_path`: Use custom `sense2vec.Sense2Vec`, `gensim.Word2vec` `gensim.FastText`, or `gensim.KeyedVectors`, or a pretrained model from [gensim](https://radimrehurek.com/gensim/downloader.html) library or a custom model path. For using a `sense2vec.Sense2Vec` take a look [here](https://github.com/explosion/sense2vec#pretrained-vectors).\n- `model`: within [standalone usage](#standalone), it is possible to pass these models directly.\n\n```python\ndata = {\n \"fruit\": [\"apple\", \"pear\", \"orange\"],\n \"vegetable\": [\"broccoli\", \"spinach\", \"tomato\"],\n \"meat\": [\"beef\", \"pork\", \"fish\", \"lamb\"]\n}\n\n# model from https://radimrehurek.com/gensim/downloader.html or path to local file\nmodel_path = \"glove-wiki-gigaword-300\"\n\nnlp.add_pipe(\"concise_concepts\", config={\"data\": data, \"model_path\": model_path})\n````\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "This repository contains an easy and intuitive approach to few-shot NER using most similar expansion over spaCy embeddings. Now with entity confidence scores!",
"version": "0.8.1",
"project_urls": {
"Documentation": "https://github.com/pandora-intelligence/concise-concepts",
"Homepage": "https://github.com/pandora-intelligence/concise-concepts",
"Repository": "https://github.com/pandora-intelligence/concise-concepts"
},
"split_keywords": [
"spacy",
"ner",
"few-shot classification",
"nlu"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4def3e226225eebbec7923b76e353baae875a29e9892f98681c25aacc0840596",
"md5": "27e848f17b0d89d3bbf7e39bcfe83706",
"sha256": "42fa5704b44c278da70258f689d92305e07cf4a40d0f7c7c54891eaac0d729bc"
},
"downloads": -1,
"filename": "concise_concepts-0.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "27e848f17b0d89d3bbf7e39bcfe83706",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<3.12",
"size": 15607,
"upload_time": "2023-06-19T13:01:23",
"upload_time_iso_8601": "2023-06-19T13:01:23.850910Z",
"url": "https://files.pythonhosted.org/packages/4d/ef/3e226225eebbec7923b76e353baae875a29e9892f98681c25aacc0840596/concise_concepts-0.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3b5351e162b39416ede8847622343aac98c537822011c657e1ec63130dcfcbc4",
"md5": "dcdbe3f956b607de77980854263b8cf2",
"sha256": "ada3d42efb94f58cc9dd0a12de81a5034a91bb030681ca641730ae5d9179fdae"
},
"downloads": -1,
"filename": "concise-concepts-0.8.1.tar.gz",
"has_sig": false,
"md5_digest": "dcdbe3f956b607de77980854263b8cf2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.12",
"size": 16216,
"upload_time": "2023-06-19T13:01:22",
"upload_time_iso_8601": "2023-06-19T13:01:22.263303Z",
"url": "https://files.pythonhosted.org/packages/3b/53/51e162b39416ede8847622343aac98c537822011c657e1ec63130dcfcbc4/concise-concepts-0.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-19 13:01:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pandora-intelligence",
"github_project": "concise-concepts",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "concise-concepts"
}
JSON
