# Costra
This is a tool for automatic evaluation of Czech sentence embeddings using Costra 1.1 dataset.
More information can be found in the following paper:
* Petra Barančíková and Ondřej Bojar: [*Costra 1.1: An Inquiry into Geometric Properties of
Sentence Spaces*](https://doi.org/10.1007/978-3-030-58323-1_14). In:
TSD 2020. Lecture Notes in Computer Science, vol 12284. Springer, Cham.
The presentation of the paper with the accompanying video can be found
[here](https://www.tsdconference.org/tsd2020/hall/paper_html/1075-omakox.php).
## Installation
```bash
$ pip install costra
```
## Usage
1. Get sentences from Costra:
```python
from costra import costra
CostraEvaluator = costra.CostraEvaluator()
sentences = CostraEvaluator.get_sentences()
```
2) Generate embeddings (example with SentenceTransformers):
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Seznam/retromae-small-cs")
embeddings = model.encode(sentences)
```
3) Evaluating the embeddings:
```python
results = CostraEvaluator.evaluate(embeddings)
```
4) Results have the following format, with `costra` being the overall score.
```python
{
'basic': 0.063,
'modality': 0.079,
'time': 0.692,
'style': 0.634,
'generalization': 0.695,
'opposite_meaning': 0.751,
'costra': 0.486
}
```
For more detail about Costra categories, refer to the original paper.
## Citation
If you use the tool, please consider citing
[the following paper](https://doi.org/10.1007/978-3-030-58323-1_14):
```bib
@inproceedings{Costra,
author = {Petra Baran{\v{\c}}{\'{\i}}kov{\'{a}} and Ond{\v{\r}}ej Bojar},
editor = {Petr Sojka and Ivan Kope{\v{\c}}ek and Karel Pala and Ales Hor{\'{a}}k},
title = {Costra 1.1: An Inquiry into Geometric Properties of Sentence Spaces},
booktitle = {Text, Speech, and Dialogue - 23rd International Conference, {TSD}
2020, Brno, Czech Republic, September 8-11, 2020, Proceedings},
series = {Lecture Notes in Computer Science},
volume = {12284},
pages = {135--143},
publisher = {Springer},
year = {2020},
url = {https://doi.org/10.1007/978-3-030-58323-1\_14},
doi = {10.1007/978-3-030-58323-1\_14},
}
```
## License
The data is distributed under the [Creative Commons 4.0 BY](https://creativecommons.org/licenses/by/4.0/).
Raw data
{
"_id": null,
"home_page": "https://github.com/barancik/costra",
"name": "costra",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.5",
"maintainer_email": null,
"keywords": "embeddings evaluation",
"author": "Petra Barancikova",
"author_email": "barancikova@ufal.mff.cuni.cz",
"download_url": "https://files.pythonhosted.org/packages/a2/0e/649809be4f4cea3646a1a7a97e3946f86cc576af7c718963f05272f86c3f/costra-1.1.tar.gz",
"platform": null,
"description": "# Costra\n\nThis is a tool for automatic evaluation of Czech sentence embeddings using Costra 1.1 dataset.\n\nMore information can be found in the following paper:\n\n* Petra Baran\u010d\u00edkov\u00e1 and Ond\u0159ej Bojar: [*Costra 1.1: An Inquiry into Geometric Properties of\n Sentence Spaces*](https://doi.org/10.1007/978-3-030-58323-1_14). In:\n TSD 2020. Lecture Notes in Computer Science, vol 12284. Springer, Cham.\n\nThe presentation of the paper with the accompanying video can be found\n [here](https://www.tsdconference.org/tsd2020/hall/paper_html/1075-omakox.php).\n\n\n## Installation\n\n ```bash\n $ pip install costra\n\n ```\n\n## Usage\n1. Get sentences from Costra:\n\n```python\nfrom costra import costra\nCostraEvaluator = costra.CostraEvaluator()\nsentences = CostraEvaluator.get_sentences()\n```\n\n2) Generate embeddings (example with SentenceTransformers):\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Seznam/retromae-small-cs\")\nembeddings = model.encode(sentences)\n```\n\n3) Evaluating the embeddings:\n\n```python\nresults = CostraEvaluator.evaluate(embeddings)\n```\n\n4) Results have the following format, with `costra` being the overall score.\n\n```python\n{\n 'basic': 0.063,\n 'modality': 0.079,\n 'time': 0.692,\n 'style': 0.634,\n 'generalization': 0.695,\n 'opposite_meaning': 0.751,\n 'costra': 0.486\n}\n```\n\nFor more detail about Costra categories, refer to the original paper.\n\n## Citation\n\nIf you use the tool, please consider citing\n[the following paper](https://doi.org/10.1007/978-3-030-58323-1_14):\n\n```bib\n@inproceedings{Costra,\n author = {Petra Baran{\\v{\\c}}{\\'{\\i}}kov{\\'{a}} and Ond{\\v{\\r}}ej Bojar},\n editor = {Petr Sojka and Ivan Kope{\\v{\\c}}ek and Karel Pala and Ales Hor{\\'{a}}k},\n title = {Costra 1.1: An Inquiry into Geometric Properties of Sentence Spaces},\n booktitle = {Text, Speech, and Dialogue - 23rd International Conference, {TSD}\n 2020, Brno, Czech Republic, September 8-11, 2020, Proceedings},\n series = {Lecture Notes in Computer Science},\n volume = {12284},\n pages = {135--143},\n publisher = {Springer},\n year = {2020},\n url = {https://doi.org/10.1007/978-3-030-58323-1\\_14},\n doi = {10.1007/978-3-030-58323-1\\_14},\n}\n```\n\n## License\n\nThe data is distributed under the [Creative Commons 4.0 BY](https://creativecommons.org/licenses/by/4.0/).\n",
"bugtrack_url": null,
"license": "Creative Commons Attribution-Noncommercial-Share Alike license",
"summary": null,
"version": "1.1",
"project_urls": {
"Homepage": "https://github.com/barancik/costra"
},
"split_keywords": [
"embeddings",
"evaluation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "184a24572aa7dbc4bb06435adec21440bec93f4e922f1893fffba579a53c2d87",
"md5": "49a4f656b2ccedae46edb465c24a6c69",
"sha256": "db5508904c66e8fe9e741ee6189514e77d4934ed6b63e913b239271f9f46730c"
},
"downloads": -1,
"filename": "costra-1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "49a4f656b2ccedae46edb465c24a6c69",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.5",
"size": 230034,
"upload_time": "2024-12-13T12:08:54",
"upload_time_iso_8601": "2024-12-13T12:08:54.586706Z",
"url": "https://files.pythonhosted.org/packages/18/4a/24572aa7dbc4bb06435adec21440bec93f4e922f1893fffba579a53c2d87/costra-1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a20e649809be4f4cea3646a1a7a97e3946f86cc576af7c718963f05272f86c3f",
"md5": "1cef502d6443e16db768d44feea79835",
"sha256": "29ed40e2b47654e4f5545b60ca8aa917e410b6904076626e4fa2641dcf486d94"
},
"downloads": -1,
"filename": "costra-1.1.tar.gz",
"has_sig": false,
"md5_digest": "1cef502d6443e16db768d44feea79835",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.5",
"size": 230432,
"upload_time": "2024-12-13T12:08:58",
"upload_time_iso_8601": "2024-12-13T12:08:58.367760Z",
"url": "https://files.pythonhosted.org/packages/a2/0e/649809be4f4cea3646a1a7a97e3946f86cc576af7c718963f05272f86c3f/costra-1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 12:08:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "barancik",
"github_project": "costra",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "costra"
}
JSON
