# ∞🧙🏼♂️AnyClassifier - One Line To Build Zero-Data Classifiers in Minutes, And A Step Towards The First AI ML Engineer
>Have you ever wanted/ been requested to build a classifier without any data? What it takes now is just one line 🤯.
**AnyClassifier** is a framework that empowers you to create high-performance classifiers without any labels or data, using minimal code.
It's designed to revolutionize the machine learning development process by eliminating the need for extensive data curation and labeling.
## 🚀 Key Features
- **Zero Data Required**: Build classifiers from scratch, even without a dataset
- **Competitive Result**: Achieving competitive results with synthetic data, comparable to using real data 🚀 - See [Benchmark](#benchmark)
- **Multilingual Synthetic Data Generation**: Supports generating whatever language as long as your LLM model can.
- **One-Line Implementation**: Democratizing AI for everyone, and agent (as a plugin for agentic flow)
- **LLM-Powered Synthetic Data and Annotations**: Leverage SOTA LLM for high-quality synthetic data generation and labeling - See [approach](docs/synthetic_data_generation.md)
- **Designed for Real-World Use**: Created by ML engineers, for ML and software engineers
## 🎯 Why AnyClassifier?
As machine learning engineers, we understand the challenges of building and curating high-quality labeled datasets.
The challenge amplifies when you want to build a multilingual dataset.
AnyClassifier eliminates this bottleneck, allowing you to focus on what matters most - solving real-world problems.
## 🏁 QuickStart in Apple Silicon - Train a model in 5 min!
<details>
<summary>Expand</summary>
```shell
# install (cp39 = python3.9, other valid values are cp310, cp311, cp312)
curl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-metal/llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
pip install llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
rm llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
pip install anyclassifier
# run
cd examples
python train_setfit_model_imdb.py
```
</details>
## 🏁 QuickStart in Colab
| Dataset | Colab Link |
|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| No Data! | [](https://colab.research.google.com/drive/1Wt_IlilfTqBbyn3gZQ3kITjObrVAypyi?usp=sharing) |
| imdb sentiment classification | [](https://colab.research.google.com/drive/1LB8PUTT9wM1Qb2cY-6Dx-RNiqmyCvRr1?usp=sharing) |
## 🛠️ Usage
### Download a small LLM (please accept the respective terms and condition of model license beforehand)
[meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
[google/gemma-2-9b](https://huggingface.co/google/gemma-2-9b)
```python
from huggingface_hub import hf_hub_download
# meta-llama/Meta-Llama-3.1-8B-Instruct
hf_hub_download("lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "Meta-Llama-3.1-8B-Instruct-Q8_0.gguf")
# google/gemma-2-9b
hf_hub_download("lmstudio-community/gemma-2-9b-it-GGUF", "gemma-2-9b-it-Q8_0.gguf")
```
### One Liner For No Data
```python
from huggingface_hub import hf_hub_download
from anyclassifier import build_anyclassifier
from anyclassifier.schema import Label
# Magic One Line!
trainer = build_anyclassifier(
"Classify a text's sentiment.",
[
Label(id=1, desc='positive sentiment'),
Label(id=0, desc='negative sentiment')
],
hf_hub_download("lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "Meta-Llama-3.1-8B-Instruct-Q8_0.gguf")
)
# Share Your Model!
trainer.push_to_hub("user_id/any_model")
```
### One Liner For Unlabeled Data
```python
from huggingface_hub import hf_hub_download
from anyclassifier import build_anyclassifier
from anyclassifier.schema import Label
unlabeled_dataset # a huggingface datasets.Dataset class can be from your local json/ csv, or from huggingface hub.
# Magic One Line!
trainer = build_anyclassifier(
"Classify a text's sentiment.",
[
Label(id=1, desc='positive sentiment'),
Label(id=0, desc='negative sentiment')
],
hf_hub_download("lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF", "Meta-Llama-3.1-8B-Instruct-Q8_0.gguf"), # as you like
unlabeled_dataset
)
# Share Your Model!
trainer.push_to_hub("user_id/any_model")
```
### To Use Model
```python
# SetFit
from setfit import SetFitModel
model = SetFitModel.from_pretrained("user_id/any_model")
preds = model.predict(["i loved the spiderman movie!", "pineapple on pizza is the worst 🤮"])
print(preds)
# FastText
from anyclassifier.fasttext_wrapper import FastTextForSequenceClassification
model = FastTextForSequenceClassification.from_pretrained("user_id/any_model")
preds = model.predict(["i loved the spiderman movie!", "pineapple on pizza is the worst 🤮"])
print(preds)
```
## 🔧 Installation
It is using llama.cpp as backend, and build wheel can take a lot of time (10min+), as such, we also provide an instruction to install with pre-built wheel.
<details>
<summary>Metal Backend (Apple's GPU) - cp39 = python3.9, other valid values are cp310, cp311, cp312</summary>
```shell
curl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-metal/llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
pip install llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
rm llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl
pip install anyclassifier
```
</details>
<details>
<summary>Colab (T4) Prebuilt Wheel</summary>
```shell
curl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-cu124/llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl
pip install llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl
rm llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl
pip install anyclassifier
```
</details>
<details>
<summary>CUDA Backend (Please read [llama-cpp-python](https://llama-cpp-python.readthedocs.io/en/latest/#installation))</summary>
```shell
CMAKE_ARGS="-DGGML_METAL=on" pip install anyclassifier
```
</details>
<details>
<summary>CPU</summary>
```shell
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install anyclassifier
```
</details>
<details>
<summary>Developer's installation</summary>
```shell
pip install -e .
```
</details>
## Other Usages
### To Label a Dataset
```python
from datasets import load_dataset
from anyclassifier.annotation.prompt import AnnotationPrompt
from anyclassifier.schema import Label
from anyclassifier.annotation.annotator import LlamaCppAnnotator
unlabeled_dataset = load_dataset("somepath")
prompt = AnnotationPrompt(
task_description="Classify a text's sentiment.",
label_definition=[
Label(id=0, desc='negative sentiment'),
Label(id=1, desc='positive sentiment')
]
)
annotator = LlamaCppAnnotator(prompt)
label_dataset = annotator.annotate_dataset(unlabeled_dataset, n_record=1000)
label_dataset.push_to_hub('user_id/any_data')
```
### To Generate a Synthetic Dataset
```python
from anyclassifier.schema import Label
from anyclassifier.synthetic_data_generation import SyntheticDataGeneratorForSequenceClassification
tree_constructor = SyntheticDataGeneratorForSequenceClassification()
dataset = tree_constructor.generate(
"Classify a text's sentiment.",
[
Label(id=0, desc='negative sentiment'),
Label(id=1, desc='positive sentiment')
]
)
dataset.push_to_hub('user_id/any_data')
```
See more examples in [examples](./examples)
| dataset | approach | model_type | example | resulting model | dataset |
|--------------------------------------------------|---------------------------|------------|------------------------------------------------------------------|-----------------------------------------------------------------------|--------------|
| "stanfordnlp/imdb" like | synthetic data generation | setfit | [link](examples/train_setfit_model_imdb_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_imdb) | [link](https://huggingface.co/datasets/kenhktsui/test_imdb_syn) |
| stanfordnlp/imdb | annotation | setfit | [link](examples/train_setfit_model_imdb.py) | [link](https://huggingface.co/kenhktsui/anyclassifier_setfit_demo) | [link](https://huggingface.co/datasets/kenhktsui/test_imdb) |
| "zeroshot/twitter-financial-news-sentiment" like | synthetic data generation | setfit | [link](examples/train_setfit_model_twitter_financial_news_sentiment_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_twitter_news_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_twitter_financial_news_syn) |
| zeroshot/twitter-financial-news-sentiment | annotation | setfit | [link](examples/train_setfit_model_twitter_financial_news_sentiment.py) | [link](https://huggingface.co/kenhktsui/setfit_test_twitter_news) | [link](https://huggingface.co/datasets/kenhktsui/test_twitter_financial_news) |
| "ccdv/arxiv-classification" like | synthetic data generation | setfit | [link](examples/train_setfit_model_arxiv_topic_synthetic.py) | [link](kenhktsui/setfit_test_arxiv_classification_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_arxiv_classification_syn) |
| ccdv/arxiv-classification | annotation | setfit | [link](examples/train_setfit_model_arxiv_topic.py) | [link](kenhktsui/setfit_test_arxiv_classification) | [link](https://huggingface.co/datasets/kenhktsui/test_arxiv_classification_syn) |
| "lmsys/toxic-chat, toxicchat0124" like | synthetic data generation | setfit | [link](examples/train_setfit_model_toxic_chat_synthetic.py) | [link](kenhktsui/setfit_test_toxic_chat_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_toxic_chat_syn) |
| lmsys/toxic-chat, toxicchat0124 | annotation | setfit | [link](examples/train_setfit_model_toxic_chat.py) | [link](kenhktsui/setfit_test_toxic_chat) | [link](https://huggingface.co/datasets/kenhktsui/test_toxic_chat) |
| "fancyzhx/ag_news" like | synthetic data generation | setfit | [link](examples/train_setfit_model_ag_news_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_ag_news_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_ag_news_syn) |
| fancyzhx/ag_news | annotation | setfit | [link](examples/train_setfit_model_ag_news.py) | [link](https://huggingface.co/kenhktsui/setfit_test_ag_news) | [link](https://huggingface.co/datasets/kenhktsui/test_ag_news) |
| chinese sentiment | synthetic data generation | N/A | [link](examples/generate_synthetic_chinese.py) | - | [link](https://huggingface.co/datasets/kenhktsui/chinese_sentiment_syn) |
## 🧪Benchmark
The objective is to see if synthetic data is performing as well as real data (annotation). Full training dataset indicates the upper limit of performance as more data is available.
Model performance of synthetic data is at par with/ close to that of real data, which is not bad because the testing data is usually by design more similar to training (real) data than synthetic data.
We also note that synthetic data is also advantageous when class is highly imbalanced like the toxic chat problem.
Our benchmark implies the synthetic data generated is close to the distribution of test data, showing the effectiveness of this synthetic data generation approach, without using any real data.
All models finetune on `sentence-transformers/paraphrase-mpnet-base-v2` (109M). The performance can be boosted by using a larger base model and generating more data.
| dataset | metric | synthetic data generation | annotation | full training dataset | full training reference|
|--|--------------------|---------------------------|------------|-----------------------|--|
|stanfordnlp/imdb| accuracy | 0.878 | 0.908 | 0.928 |[lvwerra/distilbert-imdb](https://huggingface.co/lvwerra/distilbert-imdb) |
|zeroshot/twitter-financial-news-sentiment| f1 (weighted) | 0.631 | 0.676 | 0.866 |[nickmuchi/finbert-tone-finetuned-fintwitter-classification](https://huggingface.co/nickmuchi/finbert-tone-finetuned-fintwitter-classification) |
|ccdv/arxiv-classification| acurracy | 0.618 | 0.566 | 0.805 | [paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8675939) |
|lmsys/toxic-chat, toxicchat0124 | f1 (binary) | 0.362 | 0.00* | 0.822 | [lmsys/toxicchat-t5-large-v1.0](https://huggingface.co/lmsys/toxicchat-t5-large-v1.0) |
|fancyzhx/ag_news| accuracy | 0.768 | 0.765 | 0.938 | [fabriceyhc/bert-base-uncased-ag_news](https://huggingface.co/fabriceyhc/bert-base-uncased-ag_news) |
\* Out of 42 annotations, only 2 labels is positive, making learning hard.
Codes to replicate is stored in [examples](examples/benchmark.py).
We will continue to add more benchmark on other datasets.
## 📜 Documentation
See [docs](./docs) on our detailed methodology and documentation.
## 🗺️ Roadmap
- High Quality Data:
- Prompt validation
- Label validation - inter-model annotation
- R&D in synthetic data generation
- High Quality Model
- Auto error analysis
- Auto model documentation
- Auto targeted synthetic data
- More Benchmarking
- Multilingual Support
# 👋 Contributing
- build models with AnyClassifier
- suggest features
- create issue/ PR
- benchmarking synthetic data generation vs annotation vs full training dataset
# 📄 License
AnyClassifier is released under the MIT License.
# 🤝🏻 Credits
Without these open source models and libraries, this project would not have been possible.
- powerful open source and consumer friendly LLMs e.g. Llama 3.1 8B and Gemma 2 9B
- [llama.cpp](https://github.com/ggerganov/llama.cpp) and [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
- [setfit](https://github.com/huggingface/setfit)
- huggingface ecosystem
# 📖 Citation
If you like this work, please cite:
```
@software{Tsui_AnyClassifier_2024,
author = {Tsui, Ken},
month = {8},
title = {{AnyClassifier}},
url = {https://github.com/kenhktsui/anyclassifier},
year = {2024}
}
```
# 📩 Follow Me For Update:
[X](https://x.com/kenhktsui)/ [huggingface](https://huggingface.co/kenhktsui)/ [github](https://github.com/kenhktsui)
Raw data
{
"_id": null,
"home_page": "https://github.com/kenhktsui/anyclassifier",
"name": "anyclassifier",
"maintainer": "Ken Tsui",
"docs_url": null,
"requires_python": null,
"maintainer_email": "kenhktsui@gmail.com",
"keywords": "nlp, machine learning, fewshot learning, transformers",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/2d/f8/55153517363634cddbbe7b724390573c39279e98056e531498e0cad10b42/anyclassifier-0.2.0.tar.gz",
"platform": null,
"description": "\n\n# \u221e\ud83e\uddd9\ud83c\udffc\u200d\u2642\ufe0fAnyClassifier - One Line To Build Zero-Data Classifiers in Minutes, And A Step Towards The First AI ML Engineer\n\n\n\n\n>Have you ever wanted/ been requested to build a classifier without any data? What it takes now is just one line \ud83e\udd2f. \n\n**AnyClassifier** is a framework that empowers you to create high-performance classifiers without any labels or data, using minimal code. \nIt's designed to revolutionize the machine learning development process by eliminating the need for extensive data curation and labeling.\n\n## \ud83d\ude80 Key Features\n- **Zero Data Required**: Build classifiers from scratch, even without a dataset\n- **Competitive Result**: Achieving competitive results with synthetic data, comparable to using real data \ud83d\ude80 - See [Benchmark](#benchmark)\n- **Multilingual Synthetic Data Generation**: Supports generating whatever language as long as your LLM model can. \n- **One-Line Implementation**: Democratizing AI for everyone, and agent (as a plugin for agentic flow)\n- **LLM-Powered Synthetic Data and Annotations**: Leverage SOTA LLM for high-quality synthetic data generation and labeling - See [approach](docs/synthetic_data_generation.md)\n- **Designed for Real-World Use**: Created by ML engineers, for ML and software engineers\n\n## \ud83c\udfaf Why AnyClassifier?\nAs machine learning engineers, we understand the challenges of building and curating high-quality labeled datasets. \nThe challenge amplifies when you want to build a multilingual dataset. \nAnyClassifier eliminates this bottleneck, allowing you to focus on what matters most - solving real-world problems.\n\n## \ud83c\udfc1 QuickStart in Apple Silicon - Train a model in 5 min!\n<details>\n<summary>Expand</summary>\n\n```shell\n# install (cp39 = python3.9, other valid values are cp310, cp311, cp312)\ncurl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-metal/llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\npip install llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\nrm llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\npip install anyclassifier\n# run\ncd examples\npython train_setfit_model_imdb.py\n```\n</details>\n\n\n## \ud83c\udfc1 QuickStart in Colab\n\n| Dataset | Colab Link |\n|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| No Data! | [](https://colab.research.google.com/drive/1Wt_IlilfTqBbyn3gZQ3kITjObrVAypyi?usp=sharing) |\n| imdb sentiment classification | [](https://colab.research.google.com/drive/1LB8PUTT9wM1Qb2cY-6Dx-RNiqmyCvRr1?usp=sharing) |\n\n\n## \ud83d\udee0\ufe0f Usage\n### Download a small LLM (please accept the respective terms and condition of model license beforehand)\n[meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) \n[google/gemma-2-9b](https://huggingface.co/google/gemma-2-9b)\n```python\nfrom huggingface_hub import hf_hub_download\n\n# meta-llama/Meta-Llama-3.1-8B-Instruct\nhf_hub_download(\"lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF\", \"Meta-Llama-3.1-8B-Instruct-Q8_0.gguf\")\n\n# google/gemma-2-9b\nhf_hub_download(\"lmstudio-community/gemma-2-9b-it-GGUF\", \"gemma-2-9b-it-Q8_0.gguf\")\n```\n\n### One Liner For No Data\n\n```python\nfrom huggingface_hub import hf_hub_download\nfrom anyclassifier import build_anyclassifier\nfrom anyclassifier.schema import Label\n\n# Magic One Line!\ntrainer = build_anyclassifier(\n \"Classify a text's sentiment.\",\n [\n Label(id=1, desc='positive sentiment'),\n Label(id=0, desc='negative sentiment')\n ],\n hf_hub_download(\"lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF\", \"Meta-Llama-3.1-8B-Instruct-Q8_0.gguf\")\n)\n# Share Your Model!\ntrainer.push_to_hub(\"user_id/any_model\")\n```\n\n### One Liner For Unlabeled Data\n```python\nfrom huggingface_hub import hf_hub_download\nfrom anyclassifier import build_anyclassifier\nfrom anyclassifier.schema import Label\n\nunlabeled_dataset # a huggingface datasets.Dataset class can be from your local json/ csv, or from huggingface hub.\n\n# Magic One Line!\ntrainer = build_anyclassifier(\n \"Classify a text's sentiment.\",\n [\n Label(id=1, desc='positive sentiment'),\n Label(id=0, desc='negative sentiment')\n ],\n hf_hub_download(\"lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF\", \"Meta-Llama-3.1-8B-Instruct-Q8_0.gguf\"), # as you like\n unlabeled_dataset\n)\n# Share Your Model!\ntrainer.push_to_hub(\"user_id/any_model\")\n```\n\n### To Use Model\n\n```python\n# SetFit\nfrom setfit import SetFitModel\n\nmodel = SetFitModel.from_pretrained(\"user_id/any_model\")\npreds = model.predict([\"i loved the spiderman movie!\", \"pineapple on pizza is the worst \ud83e\udd2e\"])\nprint(preds)\n\n# FastText\nfrom anyclassifier.fasttext_wrapper import FastTextForSequenceClassification\n\nmodel = FastTextForSequenceClassification.from_pretrained(\"user_id/any_model\")\npreds = model.predict([\"i loved the spiderman movie!\", \"pineapple on pizza is the worst \ud83e\udd2e\"])\nprint(preds)\n```\n\n## \ud83d\udd27 Installation\nIt is using llama.cpp as backend, and build wheel can take a lot of time (10min+), as such, we also provide an instruction to install with pre-built wheel.\n<details>\n<summary>Metal Backend (Apple's GPU) - cp39 = python3.9, other valid values are cp310, cp311, cp312</summary>\n\n```shell\ncurl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-metal/llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\npip install llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\nrm llama_cpp_python-0.2.88-cp39-cp39-macosx_11_0_arm64.whl\npip install anyclassifier\n```\n</details>\n\n<details>\n<summary>Colab (T4) Prebuilt Wheel</summary>\n\n```shell\ncurl -L -O https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.88-cu124/llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl\npip install llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl\nrm llama_cpp_python-0.2.88-cp310-cp310-linux_x86_64.whl\npip install anyclassifier\n```\n</details>\n\n<details>\n<summary>CUDA Backend (Please read [llama-cpp-python](https://llama-cpp-python.readthedocs.io/en/latest/#installation))</summary>\n\n```shell\nCMAKE_ARGS=\"-DGGML_METAL=on\" pip install anyclassifier\n```\n</details>\n\n<details>\n<summary>CPU</summary>\n\n```shell\nCMAKE_ARGS=\"-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS\" pip install anyclassifier\n```\n</details>\n\n<details>\n<summary>Developer's installation</summary>\n\n```shell\npip install -e .\n```\n</details>\n\n## Other Usages\n### To Label a Dataset\n\n```python\nfrom datasets import load_dataset\nfrom anyclassifier.annotation.prompt import AnnotationPrompt\nfrom anyclassifier.schema import Label\nfrom anyclassifier.annotation.annotator import LlamaCppAnnotator\n\nunlabeled_dataset = load_dataset(\"somepath\")\nprompt = AnnotationPrompt(\n task_description=\"Classify a text's sentiment.\",\n label_definition=[\n Label(id=0, desc='negative sentiment'),\n Label(id=1, desc='positive sentiment')\n ] \n)\nannotator = LlamaCppAnnotator(prompt)\nlabel_dataset = annotator.annotate_dataset(unlabeled_dataset, n_record=1000)\nlabel_dataset.push_to_hub('user_id/any_data')\n```\n\n### To Generate a Synthetic Dataset\n```python\nfrom anyclassifier.schema import Label\nfrom anyclassifier.synthetic_data_generation import SyntheticDataGeneratorForSequenceClassification\n\ntree_constructor = SyntheticDataGeneratorForSequenceClassification()\ndataset = tree_constructor.generate(\n \"Classify a text's sentiment.\",\n [\n Label(id=0, desc='negative sentiment'),\n Label(id=1, desc='positive sentiment')\n ]\n)\ndataset.push_to_hub('user_id/any_data')\n```\n\nSee more examples in [examples](./examples) \n\n| dataset | approach | model_type | example | resulting model | dataset |\n|--------------------------------------------------|---------------------------|------------|------------------------------------------------------------------|-----------------------------------------------------------------------|--------------|\n| \"stanfordnlp/imdb\" like | synthetic data generation | setfit | [link](examples/train_setfit_model_imdb_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_imdb) | [link](https://huggingface.co/datasets/kenhktsui/test_imdb_syn) |\n| stanfordnlp/imdb | annotation | setfit | [link](examples/train_setfit_model_imdb.py) | [link](https://huggingface.co/kenhktsui/anyclassifier_setfit_demo) | [link](https://huggingface.co/datasets/kenhktsui/test_imdb) |\n| \"zeroshot/twitter-financial-news-sentiment\" like | synthetic data generation | setfit | [link](examples/train_setfit_model_twitter_financial_news_sentiment_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_twitter_news_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_twitter_financial_news_syn) |\n| zeroshot/twitter-financial-news-sentiment | annotation | setfit | [link](examples/train_setfit_model_twitter_financial_news_sentiment.py) | [link](https://huggingface.co/kenhktsui/setfit_test_twitter_news) | [link](https://huggingface.co/datasets/kenhktsui/test_twitter_financial_news) |\n| \"ccdv/arxiv-classification\" like | synthetic data generation | setfit | [link](examples/train_setfit_model_arxiv_topic_synthetic.py) | [link](kenhktsui/setfit_test_arxiv_classification_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_arxiv_classification_syn) |\n| ccdv/arxiv-classification | annotation | setfit | [link](examples/train_setfit_model_arxiv_topic.py) | [link](kenhktsui/setfit_test_arxiv_classification) | [link](https://huggingface.co/datasets/kenhktsui/test_arxiv_classification_syn) |\n| \"lmsys/toxic-chat, toxicchat0124\" like | synthetic data generation | setfit | [link](examples/train_setfit_model_toxic_chat_synthetic.py) | [link](kenhktsui/setfit_test_toxic_chat_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_toxic_chat_syn) |\n| lmsys/toxic-chat, toxicchat0124 | annotation | setfit | [link](examples/train_setfit_model_toxic_chat.py) | [link](kenhktsui/setfit_test_toxic_chat) | [link](https://huggingface.co/datasets/kenhktsui/test_toxic_chat) |\n| \"fancyzhx/ag_news\" like | synthetic data generation | setfit | [link](examples/train_setfit_model_ag_news_synthetic.py) | [link](https://huggingface.co/kenhktsui/setfit_test_ag_news_syn) | [link](https://huggingface.co/datasets/kenhktsui/test_ag_news_syn) |\n| fancyzhx/ag_news | annotation | setfit | [link](examples/train_setfit_model_ag_news.py) | [link](https://huggingface.co/kenhktsui/setfit_test_ag_news) | [link](https://huggingface.co/datasets/kenhktsui/test_ag_news) |\n| chinese sentiment | synthetic data generation | N/A | [link](examples/generate_synthetic_chinese.py) | - | [link](https://huggingface.co/datasets/kenhktsui/chinese_sentiment_syn) |\n\n\n## \ud83e\uddeaBenchmark\nThe objective is to see if synthetic data is performing as well as real data (annotation). Full training dataset indicates the upper limit of performance as more data is available. \nModel performance of synthetic data is at par with/ close to that of real data, which is not bad because the testing data is usually by design more similar to training (real) data than synthetic data.\nWe also note that synthetic data is also advantageous when class is highly imbalanced like the toxic chat problem. \nOur benchmark implies the synthetic data generated is close to the distribution of test data, showing the effectiveness of this synthetic data generation approach, without using any real data. \nAll models finetune on `sentence-transformers/paraphrase-mpnet-base-v2` (109M). The performance can be boosted by using a larger base model and generating more data.\n\n| dataset | metric | synthetic data generation | annotation | full training dataset | full training reference|\n|--|--------------------|---------------------------|------------|-----------------------|--|\n|stanfordnlp/imdb| accuracy | 0.878 | 0.908 | 0.928 |[lvwerra/distilbert-imdb](https://huggingface.co/lvwerra/distilbert-imdb) |\n|zeroshot/twitter-financial-news-sentiment| f1 (weighted) | 0.631 | 0.676 | 0.866 |[nickmuchi/finbert-tone-finetuned-fintwitter-classification](https://huggingface.co/nickmuchi/finbert-tone-finetuned-fintwitter-classification) |\n|ccdv/arxiv-classification| acurracy | 0.618 | 0.566 | 0.805 | [paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8675939) |\n|lmsys/toxic-chat, toxicchat0124 | f1 (binary) | 0.362 | 0.00* | 0.822 | [lmsys/toxicchat-t5-large-v1.0](https://huggingface.co/lmsys/toxicchat-t5-large-v1.0) |\n|fancyzhx/ag_news| accuracy | 0.768 | 0.765 | 0.938 | [fabriceyhc/bert-base-uncased-ag_news](https://huggingface.co/fabriceyhc/bert-base-uncased-ag_news) |\n\n\\* Out of 42 annotations, only 2 labels is positive, making learning hard.\n\nCodes to replicate is stored in [examples](examples/benchmark.py).\nWe will continue to add more benchmark on other datasets.\n\n\n## \ud83d\udcdc Documentation\nSee [docs](./docs) on our detailed methodology and documentation.\n\n## \ud83d\uddfa\ufe0f Roadmap\n- High Quality Data:\n - Prompt validation\n - Label validation - inter-model annotation\n - R&D in synthetic data generation\n- High Quality Model\n - Auto error analysis\n - Auto model documentation\n - Auto targeted synthetic data\n- More Benchmarking\n- Multilingual Support\n\n# \ud83d\udc4b Contributing\n- build models with AnyClassifier\n- suggest features\n- create issue/ PR\n- benchmarking synthetic data generation vs annotation vs full training dataset\n\n# \ud83d\udcc4 License\nAnyClassifier is released under the MIT License.\n\n# \ud83e\udd1d\ud83c\udffb Credits\nWithout these open source models and libraries, this project would not have been possible. \n- powerful open source and consumer friendly LLMs e.g. Llama 3.1 8B and Gemma 2 9B\n- [llama.cpp](https://github.com/ggerganov/llama.cpp) and [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)\n- [setfit](https://github.com/huggingface/setfit)\n- huggingface ecosystem\n\n# \ud83d\udcd6 Citation\nIf you like this work, please cite:\n```\n@software{Tsui_AnyClassifier_2024,\nauthor = {Tsui, Ken},\nmonth = {8},\ntitle = {{AnyClassifier}},\nurl = {https://github.com/kenhktsui/anyclassifier},\nyear = {2024}\n}\n```\n\n# \ud83d\udce9 Follow Me For Update:\n[X](https://x.com/kenhktsui)/ [huggingface](https://huggingface.co/kenhktsui)/ [github](https://github.com/kenhktsui)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "One Line To Build Any Classifier Without Data",
"version": "0.2.0",
"project_urls": {
"Download": "https://github.com/kenhktsui/anyclassifier/tags",
"Homepage": "https://github.com/kenhktsui/anyclassifier"
},
"split_keywords": [
"nlp",
" machine learning",
" fewshot learning",
" transformers"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "88ff116d90c9c6ae32e3624f9b17b9e8beababef414a4e762f46384bb08d1e60",
"md5": "e63ac4b0b069f62e761beafd45d5f28b",
"sha256": "6666218070ad7dc141bb5fd77822922f2d56bbd31baaaf074aa20896fdb60520"
},
"downloads": -1,
"filename": "anyclassifier-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e63ac4b0b069f62e761beafd45d5f28b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 23065,
"upload_time": "2024-08-19T10:17:25",
"upload_time_iso_8601": "2024-08-19T10:17:25.073374Z",
"url": "https://files.pythonhosted.org/packages/88/ff/116d90c9c6ae32e3624f9b17b9e8beababef414a4e762f46384bb08d1e60/anyclassifier-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2df855153517363634cddbbe7b724390573c39279e98056e531498e0cad10b42",
"md5": "b5dd4c029c4b4b53548ecf6e4e7e04f1",
"sha256": "c51c3b2bc96b0e89fb73c8dd9b9cdec3983bc1b973a7e21d8fad48cd78ae62eb"
},
"downloads": -1,
"filename": "anyclassifier-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "b5dd4c029c4b4b53548ecf6e4e7e04f1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 23783,
"upload_time": "2024-08-19T10:17:27",
"upload_time_iso_8601": "2024-08-19T10:17:27.018889Z",
"url": "https://files.pythonhosted.org/packages/2d/f8/55153517363634cddbbe7b724390573c39279e98056e531498e0cad10b42/anyclassifier-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-19 10:17:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kenhktsui",
"github_project": "anyclassifier",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "anyclassifier"
}
JSON
