# SimilarityText
[](https://badge.fury.io/py/SimilarityText)
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
**Advanced text similarity and classification using AI and Machine Learning**
SimilarityText is a powerful Python library that leverages state-of-the-art AI and traditional NLP techniques to measure semantic similarity between texts and classify documents. With support for **transformer models**, **machine learning classifiers**, and **50+ languages**, it's the perfect tool for modern NLP applications.
---
## 🌟 Key Features
### 🎯 Text Similarity
- **Classic TF-IDF**: Fast and efficient lexical similarity
- **Neural Transformers**: State-of-the-art semantic understanding using BERT-based models
- **Cross-lingual**: Compare texts across different languages
- **Auto-method Selection**: Automatically chooses the best available method
### 🏷️ Text Classification
- **Word Frequency**: Simple baseline method
- **Machine Learning**: SVM and Naive Bayes classifiers with TF-IDF features
- **Deep Learning**: Transformer-based classification for maximum accuracy
- **Confidence Scores**: Get prediction probabilities for all methods
### 🌍 Multilingual Support
- **50+ languages** supported out of the box
- **17 languages** with advanced stemming (Arabic, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish)
- **Automatic language detection** with graceful fallbacks
- **Cross-lingual transformers** for multilingual tasks
### 🚀 Easy to Use
- Simple, intuitive API
- sklearn-compatible interface (`predict`, `predict_proba`)
- Extensive documentation and examples
- Backward compatible (v0.2.0 code still works)
---
## 📦 Installation
### Basic Installation
```bash
pip install SimilarityText
```
This installs the core library with TF-IDF and ML classification support.
### Advanced Installation (with Transformers)
For state-of-the-art neural network support:
```bash
pip install SimilarityText[transformers]
```
Or install dependencies separately:
```bash
pip install sentence-transformers torch transformers
```
### From Source
```bash
git clone https://github.com/fabiocax/SimilarityText.git
cd SimilarityText
pip install -e .
```
---
## 🚀 Quick Start
### Text Similarity
```python
from similarity import Similarity
# Initialize (downloads required NLTK data on first run)
sim = Similarity()
# Calculate similarity between two texts
score = sim.similarity(
'The cat is sleeping on the couch',
'A feline is resting on the sofa'
)
print(f"Similarity: {score:.2f}") # Output: ~0.75
```
### Text Classification
```python
from similarity import Classification
# Prepare training data
training_data = [
{"class": "positive", "word": "I love this product! Amazing quality."},
{"class": "positive", "word": "Excellent service, highly recommend!"},
{"class": "negative", "word": "Terrible experience, very disappointed."},
{"class": "negative", "word": "Poor quality, waste of money."},
]
# Train classifier
classifier = Classification(use_ml=True) # Use ML for better accuracy
classifier.learning(training_data)
# Classify new text
text = "This is absolutely wonderful! Best purchase ever."
predicted_class, confidence = classifier.calculate_score(
text,
return_confidence=True
)
print(f"Class: {predicted_class}, Confidence: {confidence:.2f}")
# Output: Class: positive, Confidence: 0.89
```
---
## 📚 Comprehensive Guide
### Similarity Methods
#### 1. TF-IDF Method (Default - Fast)
```python
from similarity import Similarity
sim = Similarity(
language='english', # Target language
langdetect=False, # Auto-detect language
quiet=True # Suppress output
)
# Compare texts
score = sim.similarity('Python programming', 'Java programming')
print(f"TF-IDF Score: {score:.4f}")
```
**Best for**: Quick comparisons, large-scale batch processing, production systems with latency constraints
#### 2. Transformer Method (Most Accurate)
```python
from similarity import Similarity
sim = Similarity(
use_transformers=True,
model_name='paraphrase-multilingual-MiniLM-L12-v2' # Default model
)
# Compare texts with deep semantic understanding
score = sim.similarity(
'The quick brown fox jumps over the lazy dog',
'A fast auburn fox leaps above an idle canine'
)
print(f"Transformer Score: {score:.4f}")
# Cross-lingual comparison
score = sim.similarity(
'I love artificial intelligence',
'Eu amo inteligência artificial' # Portuguese
)
print(f"Cross-lingual Score: {score:.4f}")
```
**Best for**: Semantic understanding, cross-lingual tasks, when accuracy is critical
#### 3. Method Selection
```python
sim = Similarity(use_transformers=True)
# Auto: Uses transformers if available, falls back to TF-IDF
score = sim.similarity(text1, text2, method='auto')
# Force TF-IDF
score = sim.similarity(text1, text2, method='tfidf')
# Force transformers
score = sim.similarity(text1, text2, method='transformer')
```
### Similarity Parameters
```python
Similarity(
update=True, # Download NLTK data on initialization
language='english', # Default language for processing
langdetect=False, # Enable automatic language detection
nltk_downloads=[], # Additional NLTK packages to download
quiet=True, # Suppress informational messages
use_transformers=False, # Enable transformer models
model_name='paraphrase-multilingual-MiniLM-L12-v2' # Transformer model
)
```
### Classification Methods
#### 1. Word Frequency Method (Baseline)
```python
from similarity import Classification
classifier = Classification(
language='english',
use_ml=False # Disable ML, use word frequency
)
classifier.learning(training_data)
predicted_class = classifier.calculate_score("Sample text")
```
**Best for**: Simple categorization, understanding, baseline comparisons
#### 2. Machine Learning Method (Recommended)
```python
classifier = Classification(
language='english',
use_ml=True # Enable SVM/Naive Bayes
)
classifier.learning(training_data)
# Get prediction with confidence
predicted_class, confidence = classifier.calculate_score(
"Sample text",
return_confidence=True
)
# sklearn-like interface
predicted = classifier.predict("Sample text")
probabilities = classifier.predict_proba("Sample text")
print(f"Probabilities: {probabilities}")
```
**Best for**: Production systems, when you have training data, balanced accuracy/speed
#### 3. Transformer Method (Highest Accuracy)
```python
classifier = Classification(
language='english',
use_transformers=True,
model_name='paraphrase-multilingual-MiniLM-L12-v2'
)
classifier.learning(training_data)
predicted_class, confidence = classifier.calculate_score(
"Sample text",
return_confidence=True
)
```
**Best for**: Maximum accuracy, semantic understanding, sufficient compute resources
### Classification Parameters
```python
Classification(
language='english', # Language for text processing
use_ml=True, # Enable ML classifiers (SVM/Naive Bayes)
use_transformers=False, # Enable transformer-based classification
model_name='paraphrase-multilingual-MiniLM-L12-v2' # Model name
)
```
---
## 🎯 Complete Examples
### Example 1: Semantic Similarity Comparison
```python
from similarity import Similarity
# Initialize both methods
sim_classic = Similarity()
sim_neural = Similarity(use_transformers=True)
# Test pairs
pairs = [
("The car is red", "The automobile is crimson"),
("Python is a programming language", "Java is used for coding"),
("I love machine learning", "Machine learning is fascinating"),
]
print("Method Comparison:")
print("-" * 60)
for text1, text2 in pairs:
score_tfidf = sim_classic.similarity(text1, text2)
score_neural = sim_neural.similarity(text1, text2)
print(f"\nText A: {text1}")
print(f"Text B: {text2}")
print(f"TF-IDF: {score_tfidf:.4f}")
print(f"Transformer: {score_neural:.4f}")
print(f"Difference: {abs(score_neural - score_tfidf):.4f}")
```
### Example 2: Sentiment Analysis
```python
from similarity import Classification
# Training data
training_data = [
{"class": "positive", "word": "excellent product quality amazing"},
{"class": "positive", "word": "love it best purchase ever"},
{"class": "positive", "word": "highly recommend great service"},
{"class": "negative", "word": "terrible waste of money disappointed"},
{"class": "negative", "word": "poor quality broke immediately"},
{"class": "negative", "word": "awful experience never again"},
{"class": "neutral", "word": "okay average nothing special"},
{"class": "neutral", "word": "it works as expected"},
]
# Train classifier
classifier = Classification(use_ml=True)
classifier.learning(training_data)
# Test reviews
reviews = [
"This is the best thing I've ever bought!",
"Complete disaster, total waste of money.",
"It's fine, does what it says.",
"Absolutely fantastic, exceeded expectations!",
]
print("Sentiment Analysis Results:")
print("-" * 60)
for review in reviews:
sentiment, confidence = classifier.calculate_score(
review,
return_confidence=True
)
print(f"\nReview: {review}")
print(f"Sentiment: {sentiment.upper()}")
print(f"Confidence: {confidence:.2f}")
```
### Example 3: Multilingual Document Classification
```python
from similarity import Classification
# Multilingual training data
training_data = [
{"class": "technology", "word": "artificial intelligence machine learning"},
{"class": "technology", "word": "inteligência artificial aprendizado de máquina"},
{"class": "technology", "word": "intelligence artificielle apprentissage automatique"},
{"class": "sports", "word": "football soccer championship tournament"},
{"class": "sports", "word": "futebol campeonato torneio"},
{"class": "sports", "word": "football championnat tournoi"},
]
# Use transformer for multilingual understanding
classifier = Classification(use_transformers=True)
classifier.learning(training_data)
# Test in different languages
test_texts = [
"Deep learning neural networks are fascinating", # English
"O campeonato de futebol foi emocionante", # Portuguese
"L'intelligence artificielle change le monde", # French
]
print("Multilingual Classification:")
print("-" * 60)
for text in test_texts:
category, confidence = classifier.calculate_score(
text,
return_confidence=True
)
print(f"\nText: {text}")
print(f"Category: {category}")
print(f"Confidence: {confidence:.2f}")
```
---
## 🔬 Performance Comparison
### Similarity Methods
| Method | Speed | Accuracy | Cross-lingual | Memory | Best Use Case |
|--------|-------|----------|---------------|--------|---------------|
| **TF-IDF** | ⚡⚡⚡ Very Fast | ⭐⭐⭐ Good | ❌ No | Low | Quick comparisons, batch processing |
| **Transformers** | ⚡ Slow | ⭐⭐⭐⭐⭐ Excellent | ✅ Yes | High | Semantic understanding, cross-lingual |
### Classification Methods
| Method | Speed | Accuracy | Training Time | Memory | Best Use Case |
|--------|-------|----------|---------------|--------|---------------|
| **Word Frequency** | ⚡⚡⚡ Very Fast | ⭐⭐ Fair | Instant | Very Low | Baseline, simple tasks |
| **ML (SVM)** | ⚡⚡ Fast | ⭐⭐⭐⭐ Very Good | Fast | Low | Production systems |
| **Transformers** | ⚡ Slow | ⭐⭐⭐⭐⭐ Excellent | Medium | High | Maximum accuracy |
### Benchmark Results
Tested on Intel i7, 16GB RAM, using 1000 text pairs:
```
Similarity Benchmarks:
├── TF-IDF: 0.05s (20,000 pairs/sec)
├── Transformers: 2.30s (435 pairs/sec)
Classification Benchmarks (100 documents):
├── Word Frequency: 0.02s
├── ML (SVM): 0.15s
├── Transformers: 1.80s
```
---
## 📖 Available Transformer Models
### Recommended Models
| Model | Size | Speed | Languages | Best For |
|-------|------|-------|-----------|----------|
| `paraphrase-multilingual-MiniLM-L12-v2` | 418MB | Fast | 50+ | General purpose (default) |
| `all-MiniLM-L6-v2` | 80MB | Very Fast | EN | English-only, speed critical |
| `paraphrase-mpnet-base-v2` | 420MB | Medium | EN | English, highest accuracy |
| `distiluse-base-multilingual-cased-v2` | 480MB | Medium | 50+ | Multilingual, good balance |
| `all-mpnet-base-v2` | 420MB | Medium | EN | English, semantic search |
### Usage
```python
# Use a specific model
sim = Similarity(
use_transformers=True,
model_name='all-MiniLM-L6-v2' # Fast English model
)
```
Browse all models: [https://www.sbert.net/docs/pretrained_models.html](https://www.sbert.net/docs/pretrained_models.html)
---
## 🌐 Supported Languages
Full language support includes:
**European**: English, Portuguese, Spanish, French, German, Italian, Dutch, Russian, Polish, Romanian, Hungarian, Czech, Swedish, Danish, Finnish, Norwegian, Turkish, Greek
**Asian**: Chinese, Japanese, Korean, Arabic, Hebrew, Thai, Vietnamese, Indonesian
**Others**: Hindi, Bengali, Tamil, Urdu, Persian, and 30+ more
**Advanced stemming** available for: Arabic, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish
---
## 📊 API Reference
See [API.md](API.md) for complete API documentation.
### Similarity Class
```python
class Similarity:
def __init__(self, update=True, language='english', langdetect=False,
nltk_downloads=[], quiet=True, use_transformers=False,
model_name='paraphrase-multilingual-MiniLM-L12-v2'):
"""Initialize Similarity analyzer"""
def similarity(self, text_a, text_b, method='auto'):
"""Calculate similarity between two texts (returns float 0.0-1.0)"""
def detectlang(self, text):
"""Detect language of text (returns language name)"""
```
### Classification Class
```python
class Classification:
def __init__(self, language='english', use_ml=True, use_transformers=False,
model_name='paraphrase-multilingual-MiniLM-L12-v2'):
"""Initialize classifier"""
def learning(self, training_data):
"""Train classifier with list of {"class": str, "word": str} dicts"""
def calculate_score(self, sentence, return_confidence=False):
"""Classify sentence, optionally return confidence"""
def predict(self, sentence):
"""Predict class (sklearn-compatible)"""
def predict_proba(self, sentence):
"""Get class probabilities (sklearn-compatible)"""
```
---
## 🆕 What's New in v0.3.0
### 🎯 Major Features
- ✨ **Transformer support**: State-of-the-art neural models via sentence-transformers
- 🧠 **ML classifiers**: SVM and Naive Bayes with TF-IDF
- 🌍 **Better multilingual**: Improved language handling with 17 stemmers
- 📊 **Confidence scores**: Get prediction probabilities
- 🔧 **Flexible API**: sklearn-like interface with `predict()` and `predict_proba()`
### 🐛 Critical Bug Fixes
- Fixed typo: `requeriments.txt` → `requirements.txt`
- Fixed RSLPStemmer being used for all languages (now language-aware)
- Fixed crashes when stopwords unavailable for languages
- Fixed language detection failures on short texts
- Fixed exception messages for better debugging
- Added `punkt_tab` to NLTK downloads for compatibility
### 🔄 Backwards Compatibility
All v0.2.0 code continues to work without modifications. New features are opt-in.
See [CHANGELOG.md](CHANGELOG.md) for complete version history.
---
## 📝 Examples
Explore the `example/` directory:
- **`example.py`**: Basic TF-IDF similarity examples
- **`exemplo2.py`**: Classification examples
- **`example_advanced.py`**: Advanced AI features with transformers and comparisons
Run examples:
```bash
python example/example.py
python example/example_advanced.py
```
---
## 🤝 Contributing
Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
### Quick Start for Contributors
```bash
# Clone repository
git clone https://github.com/fabiocax/SimilarityText.git
cd SimilarityText
# Install in development mode
pip install -e .[transformers]
# Run examples
python example/example_advanced.py
```
---
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---
## 👤 Author
**Fabio Alberti**
- Email: fabiocax@gmail.com
- GitHub: [@fabiocax](https://github.com/fabiocax)
---
## 🔗 Links
- **GitHub**: [https://github.com/fabiocax/SimilarityText](https://github.com/fabiocax/SimilarityText)
- **PyPI**: [https://pypi.org/project/SimilarityText/](https://pypi.org/project/SimilarityText/)
- **Documentation**: [https://github.com/fabiocax/SimilarityText/blob/main/README.md](https://github.com/fabiocax/SimilarityText/blob/main/README.md)
- **Issues**: [https://github.com/fabiocax/SimilarityText/issues](https://github.com/fabiocax/SimilarityText/issues)
---
## 🙏 Acknowledgments
- **sentence-transformers**: For providing excellent pre-trained models
- **scikit-learn**: For robust ML algorithms
- **NLTK**: For comprehensive NLP tools
- All contributors and users of this library
---
## ⭐ Star History
If you find this project useful, please consider giving it a star on GitHub!
[](https://star-history.com/#fabiocax/SimilarityText&Date)
---
## 📈 Roadmap
- [ ] Add more pre-trained models
- [ ] Batch processing API
- [ ] GPU acceleration support
- [ ] REST API server
- [ ] Caching mechanisms
- [ ] More language-specific optimizations
- [ ] Integration with popular frameworks (FastAPI, Flask)
---
**Made with ❤️ using Python and AI**
Raw data
{
"_id": null,
"home_page": "https://github.com/fabiocax/SimilarityText",
"name": "SimilarityText",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "nlp, natural language processing, text similarity, semantic similarity, text classification, machine learning, deep learning, transformers, bert, sentence transformers, tf-idf, cosine similarity, sentiment analysis, multilingual, ai, artificial intelligence",
"author": "Fabio Alberti",
"author_email": "fabiocax@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/4a/c8/451e807ee10c0eebb32ee406d4dfeb0e684ba4c84f6045d0916bfdca38ab/SimilarityText-0.3.0.tar.gz",
"platform": null,
"description": "# SimilarityText\n\n[](https://badge.fury.io/py/SimilarityText)\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n\n**Advanced text similarity and classification using AI and Machine Learning**\n\nSimilarityText is a powerful Python library that leverages state-of-the-art AI and traditional NLP techniques to measure semantic similarity between texts and classify documents. With support for **transformer models**, **machine learning classifiers**, and **50+ languages**, it's the perfect tool for modern NLP applications.\n\n---\n\n## \ud83c\udf1f Key Features\n\n### \ud83c\udfaf Text Similarity\n- **Classic TF-IDF**: Fast and efficient lexical similarity\n- **Neural Transformers**: State-of-the-art semantic understanding using BERT-based models\n- **Cross-lingual**: Compare texts across different languages\n- **Auto-method Selection**: Automatically chooses the best available method\n\n### \ud83c\udff7\ufe0f Text Classification\n- **Word Frequency**: Simple baseline method\n- **Machine Learning**: SVM and Naive Bayes classifiers with TF-IDF features\n- **Deep Learning**: Transformer-based classification for maximum accuracy\n- **Confidence Scores**: Get prediction probabilities for all methods\n\n### \ud83c\udf0d Multilingual Support\n- **50+ languages** supported out of the box\n- **17 languages** with advanced stemming (Arabic, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish)\n- **Automatic language detection** with graceful fallbacks\n- **Cross-lingual transformers** for multilingual tasks\n\n### \ud83d\ude80 Easy to Use\n- Simple, intuitive API\n- sklearn-compatible interface (`predict`, `predict_proba`)\n- Extensive documentation and examples\n- Backward compatible (v0.2.0 code still works)\n\n---\n\n## \ud83d\udce6 Installation\n\n### Basic Installation\n\n```bash\npip install SimilarityText\n```\n\nThis installs the core library with TF-IDF and ML classification support.\n\n### Advanced Installation (with Transformers)\n\nFor state-of-the-art neural network support:\n\n```bash\npip install SimilarityText[transformers]\n```\n\nOr install dependencies separately:\n\n```bash\npip install sentence-transformers torch transformers\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/fabiocax/SimilarityText.git\ncd SimilarityText\npip install -e .\n```\n\n---\n\n## \ud83d\ude80 Quick Start\n\n### Text Similarity\n\n```python\nfrom similarity import Similarity\n\n# Initialize (downloads required NLTK data on first run)\nsim = Similarity()\n\n# Calculate similarity between two texts\nscore = sim.similarity(\n 'The cat is sleeping on the couch',\n 'A feline is resting on the sofa'\n)\nprint(f\"Similarity: {score:.2f}\") # Output: ~0.75\n```\n\n### Text Classification\n\n```python\nfrom similarity import Classification\n\n# Prepare training data\ntraining_data = [\n {\"class\": \"positive\", \"word\": \"I love this product! Amazing quality.\"},\n {\"class\": \"positive\", \"word\": \"Excellent service, highly recommend!\"},\n {\"class\": \"negative\", \"word\": \"Terrible experience, very disappointed.\"},\n {\"class\": \"negative\", \"word\": \"Poor quality, waste of money.\"},\n]\n\n# Train classifier\nclassifier = Classification(use_ml=True) # Use ML for better accuracy\nclassifier.learning(training_data)\n\n# Classify new text\ntext = \"This is absolutely wonderful! Best purchase ever.\"\npredicted_class, confidence = classifier.calculate_score(\n text,\n return_confidence=True\n)\nprint(f\"Class: {predicted_class}, Confidence: {confidence:.2f}\")\n# Output: Class: positive, Confidence: 0.89\n```\n\n---\n\n## \ud83d\udcda Comprehensive Guide\n\n### Similarity Methods\n\n#### 1. TF-IDF Method (Default - Fast)\n\n```python\nfrom similarity import Similarity\n\nsim = Similarity(\n language='english', # Target language\n langdetect=False, # Auto-detect language\n quiet=True # Suppress output\n)\n\n# Compare texts\nscore = sim.similarity('Python programming', 'Java programming')\nprint(f\"TF-IDF Score: {score:.4f}\")\n```\n\n**Best for**: Quick comparisons, large-scale batch processing, production systems with latency constraints\n\n#### 2. Transformer Method (Most Accurate)\n\n```python\nfrom similarity import Similarity\n\nsim = Similarity(\n use_transformers=True,\n model_name='paraphrase-multilingual-MiniLM-L12-v2' # Default model\n)\n\n# Compare texts with deep semantic understanding\nscore = sim.similarity(\n 'The quick brown fox jumps over the lazy dog',\n 'A fast auburn fox leaps above an idle canine'\n)\nprint(f\"Transformer Score: {score:.4f}\")\n\n# Cross-lingual comparison\nscore = sim.similarity(\n 'I love artificial intelligence',\n 'Eu amo intelig\u00eancia artificial' # Portuguese\n)\nprint(f\"Cross-lingual Score: {score:.4f}\")\n```\n\n**Best for**: Semantic understanding, cross-lingual tasks, when accuracy is critical\n\n#### 3. Method Selection\n\n```python\nsim = Similarity(use_transformers=True)\n\n# Auto: Uses transformers if available, falls back to TF-IDF\nscore = sim.similarity(text1, text2, method='auto')\n\n# Force TF-IDF\nscore = sim.similarity(text1, text2, method='tfidf')\n\n# Force transformers\nscore = sim.similarity(text1, text2, method='transformer')\n```\n\n### Similarity Parameters\n\n```python\nSimilarity(\n update=True, # Download NLTK data on initialization\n language='english', # Default language for processing\n langdetect=False, # Enable automatic language detection\n nltk_downloads=[], # Additional NLTK packages to download\n quiet=True, # Suppress informational messages\n use_transformers=False, # Enable transformer models\n model_name='paraphrase-multilingual-MiniLM-L12-v2' # Transformer model\n)\n```\n\n### Classification Methods\n\n#### 1. Word Frequency Method (Baseline)\n\n```python\nfrom similarity import Classification\n\nclassifier = Classification(\n language='english',\n use_ml=False # Disable ML, use word frequency\n)\n\nclassifier.learning(training_data)\npredicted_class = classifier.calculate_score(\"Sample text\")\n```\n\n**Best for**: Simple categorization, understanding, baseline comparisons\n\n#### 2. Machine Learning Method (Recommended)\n\n```python\nclassifier = Classification(\n language='english',\n use_ml=True # Enable SVM/Naive Bayes\n)\n\nclassifier.learning(training_data)\n\n# Get prediction with confidence\npredicted_class, confidence = classifier.calculate_score(\n \"Sample text\",\n return_confidence=True\n)\n\n# sklearn-like interface\npredicted = classifier.predict(\"Sample text\")\nprobabilities = classifier.predict_proba(\"Sample text\")\nprint(f\"Probabilities: {probabilities}\")\n```\n\n**Best for**: Production systems, when you have training data, balanced accuracy/speed\n\n#### 3. Transformer Method (Highest Accuracy)\n\n```python\nclassifier = Classification(\n language='english',\n use_transformers=True,\n model_name='paraphrase-multilingual-MiniLM-L12-v2'\n)\n\nclassifier.learning(training_data)\npredicted_class, confidence = classifier.calculate_score(\n \"Sample text\",\n return_confidence=True\n)\n```\n\n**Best for**: Maximum accuracy, semantic understanding, sufficient compute resources\n\n### Classification Parameters\n\n```python\nClassification(\n language='english', # Language for text processing\n use_ml=True, # Enable ML classifiers (SVM/Naive Bayes)\n use_transformers=False, # Enable transformer-based classification\n model_name='paraphrase-multilingual-MiniLM-L12-v2' # Model name\n)\n```\n\n---\n\n## \ud83c\udfaf Complete Examples\n\n### Example 1: Semantic Similarity Comparison\n\n```python\nfrom similarity import Similarity\n\n# Initialize both methods\nsim_classic = Similarity()\nsim_neural = Similarity(use_transformers=True)\n\n# Test pairs\npairs = [\n (\"The car is red\", \"The automobile is crimson\"),\n (\"Python is a programming language\", \"Java is used for coding\"),\n (\"I love machine learning\", \"Machine learning is fascinating\"),\n]\n\nprint(\"Method Comparison:\")\nprint(\"-\" * 60)\nfor text1, text2 in pairs:\n score_tfidf = sim_classic.similarity(text1, text2)\n score_neural = sim_neural.similarity(text1, text2)\n\n print(f\"\\nText A: {text1}\")\n print(f\"Text B: {text2}\")\n print(f\"TF-IDF: {score_tfidf:.4f}\")\n print(f\"Transformer: {score_neural:.4f}\")\n print(f\"Difference: {abs(score_neural - score_tfidf):.4f}\")\n```\n\n### Example 2: Sentiment Analysis\n\n```python\nfrom similarity import Classification\n\n# Training data\ntraining_data = [\n {\"class\": \"positive\", \"word\": \"excellent product quality amazing\"},\n {\"class\": \"positive\", \"word\": \"love it best purchase ever\"},\n {\"class\": \"positive\", \"word\": \"highly recommend great service\"},\n {\"class\": \"negative\", \"word\": \"terrible waste of money disappointed\"},\n {\"class\": \"negative\", \"word\": \"poor quality broke immediately\"},\n {\"class\": \"negative\", \"word\": \"awful experience never again\"},\n {\"class\": \"neutral\", \"word\": \"okay average nothing special\"},\n {\"class\": \"neutral\", \"word\": \"it works as expected\"},\n]\n\n# Train classifier\nclassifier = Classification(use_ml=True)\nclassifier.learning(training_data)\n\n# Test reviews\nreviews = [\n \"This is the best thing I've ever bought!\",\n \"Complete disaster, total waste of money.\",\n \"It's fine, does what it says.\",\n \"Absolutely fantastic, exceeded expectations!\",\n]\n\nprint(\"Sentiment Analysis Results:\")\nprint(\"-\" * 60)\nfor review in reviews:\n sentiment, confidence = classifier.calculate_score(\n review,\n return_confidence=True\n )\n print(f\"\\nReview: {review}\")\n print(f\"Sentiment: {sentiment.upper()}\")\n print(f\"Confidence: {confidence:.2f}\")\n```\n\n### Example 3: Multilingual Document Classification\n\n```python\nfrom similarity import Classification\n\n# Multilingual training data\ntraining_data = [\n {\"class\": \"technology\", \"word\": \"artificial intelligence machine learning\"},\n {\"class\": \"technology\", \"word\": \"intelig\u00eancia artificial aprendizado de m\u00e1quina\"},\n {\"class\": \"technology\", \"word\": \"intelligence artificielle apprentissage automatique\"},\n {\"class\": \"sports\", \"word\": \"football soccer championship tournament\"},\n {\"class\": \"sports\", \"word\": \"futebol campeonato torneio\"},\n {\"class\": \"sports\", \"word\": \"football championnat tournoi\"},\n]\n\n# Use transformer for multilingual understanding\nclassifier = Classification(use_transformers=True)\nclassifier.learning(training_data)\n\n# Test in different languages\ntest_texts = [\n \"Deep learning neural networks are fascinating\", # English\n \"O campeonato de futebol foi emocionante\", # Portuguese\n \"L'intelligence artificielle change le monde\", # French\n]\n\nprint(\"Multilingual Classification:\")\nprint(\"-\" * 60)\nfor text in test_texts:\n category, confidence = classifier.calculate_score(\n text,\n return_confidence=True\n )\n print(f\"\\nText: {text}\")\n print(f\"Category: {category}\")\n print(f\"Confidence: {confidence:.2f}\")\n```\n\n---\n\n## \ud83d\udd2c Performance Comparison\n\n### Similarity Methods\n\n| Method | Speed | Accuracy | Cross-lingual | Memory | Best Use Case |\n|--------|-------|----------|---------------|--------|---------------|\n| **TF-IDF** | \u26a1\u26a1\u26a1 Very Fast | \u2b50\u2b50\u2b50 Good | \u274c No | Low | Quick comparisons, batch processing |\n| **Transformers** | \u26a1 Slow | \u2b50\u2b50\u2b50\u2b50\u2b50 Excellent | \u2705 Yes | High | Semantic understanding, cross-lingual |\n\n### Classification Methods\n\n| Method | Speed | Accuracy | Training Time | Memory | Best Use Case |\n|--------|-------|----------|---------------|--------|---------------|\n| **Word Frequency** | \u26a1\u26a1\u26a1 Very Fast | \u2b50\u2b50 Fair | Instant | Very Low | Baseline, simple tasks |\n| **ML (SVM)** | \u26a1\u26a1 Fast | \u2b50\u2b50\u2b50\u2b50 Very Good | Fast | Low | Production systems |\n| **Transformers** | \u26a1 Slow | \u2b50\u2b50\u2b50\u2b50\u2b50 Excellent | Medium | High | Maximum accuracy |\n\n### Benchmark Results\n\nTested on Intel i7, 16GB RAM, using 1000 text pairs:\n\n```\nSimilarity Benchmarks:\n\u251c\u2500\u2500 TF-IDF: 0.05s (20,000 pairs/sec)\n\u251c\u2500\u2500 Transformers: 2.30s (435 pairs/sec)\n\nClassification Benchmarks (100 documents):\n\u251c\u2500\u2500 Word Frequency: 0.02s\n\u251c\u2500\u2500 ML (SVM): 0.15s\n\u251c\u2500\u2500 Transformers: 1.80s\n```\n\n---\n\n## \ud83d\udcd6 Available Transformer Models\n\n### Recommended Models\n\n| Model | Size | Speed | Languages | Best For |\n|-------|------|-------|-----------|----------|\n| `paraphrase-multilingual-MiniLM-L12-v2` | 418MB | Fast | 50+ | General purpose (default) |\n| `all-MiniLM-L6-v2` | 80MB | Very Fast | EN | English-only, speed critical |\n| `paraphrase-mpnet-base-v2` | 420MB | Medium | EN | English, highest accuracy |\n| `distiluse-base-multilingual-cased-v2` | 480MB | Medium | 50+ | Multilingual, good balance |\n| `all-mpnet-base-v2` | 420MB | Medium | EN | English, semantic search |\n\n### Usage\n\n```python\n# Use a specific model\nsim = Similarity(\n use_transformers=True,\n model_name='all-MiniLM-L6-v2' # Fast English model\n)\n```\n\nBrowse all models: [https://www.sbert.net/docs/pretrained_models.html](https://www.sbert.net/docs/pretrained_models.html)\n\n---\n\n## \ud83c\udf10 Supported Languages\n\nFull language support includes:\n\n**European**: English, Portuguese, Spanish, French, German, Italian, Dutch, Russian, Polish, Romanian, Hungarian, Czech, Swedish, Danish, Finnish, Norwegian, Turkish, Greek\n\n**Asian**: Chinese, Japanese, Korean, Arabic, Hebrew, Thai, Vietnamese, Indonesian\n\n**Others**: Hindi, Bengali, Tamil, Urdu, Persian, and 30+ more\n\n**Advanced stemming** available for: Arabic, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish\n\n---\n\n## \ud83d\udcca API Reference\n\nSee [API.md](API.md) for complete API documentation.\n\n### Similarity Class\n\n```python\nclass Similarity:\n def __init__(self, update=True, language='english', langdetect=False,\n nltk_downloads=[], quiet=True, use_transformers=False,\n model_name='paraphrase-multilingual-MiniLM-L12-v2'):\n \"\"\"Initialize Similarity analyzer\"\"\"\n\n def similarity(self, text_a, text_b, method='auto'):\n \"\"\"Calculate similarity between two texts (returns float 0.0-1.0)\"\"\"\n\n def detectlang(self, text):\n \"\"\"Detect language of text (returns language name)\"\"\"\n```\n\n### Classification Class\n\n```python\nclass Classification:\n def __init__(self, language='english', use_ml=True, use_transformers=False,\n model_name='paraphrase-multilingual-MiniLM-L12-v2'):\n \"\"\"Initialize classifier\"\"\"\n\n def learning(self, training_data):\n \"\"\"Train classifier with list of {\"class\": str, \"word\": str} dicts\"\"\"\n\n def calculate_score(self, sentence, return_confidence=False):\n \"\"\"Classify sentence, optionally return confidence\"\"\"\n\n def predict(self, sentence):\n \"\"\"Predict class (sklearn-compatible)\"\"\"\n\n def predict_proba(self, sentence):\n \"\"\"Get class probabilities (sklearn-compatible)\"\"\"\n```\n\n---\n\n## \ud83c\udd95 What's New in v0.3.0\n\n### \ud83c\udfaf Major Features\n- \u2728 **Transformer support**: State-of-the-art neural models via sentence-transformers\n- \ud83e\udde0 **ML classifiers**: SVM and Naive Bayes with TF-IDF\n- \ud83c\udf0d **Better multilingual**: Improved language handling with 17 stemmers\n- \ud83d\udcca **Confidence scores**: Get prediction probabilities\n- \ud83d\udd27 **Flexible API**: sklearn-like interface with `predict()` and `predict_proba()`\n\n### \ud83d\udc1b Critical Bug Fixes\n- Fixed typo: `requeriments.txt` \u2192 `requirements.txt`\n- Fixed RSLPStemmer being used for all languages (now language-aware)\n- Fixed crashes when stopwords unavailable for languages\n- Fixed language detection failures on short texts\n- Fixed exception messages for better debugging\n- Added `punkt_tab` to NLTK downloads for compatibility\n\n### \ud83d\udd04 Backwards Compatibility\nAll v0.2.0 code continues to work without modifications. New features are opt-in.\n\nSee [CHANGELOG.md](CHANGELOG.md) for complete version history.\n\n---\n\n## \ud83d\udcdd Examples\n\nExplore the `example/` directory:\n\n- **`example.py`**: Basic TF-IDF similarity examples\n- **`exemplo2.py`**: Classification examples\n- **`example_advanced.py`**: Advanced AI features with transformers and comparisons\n\nRun examples:\n```bash\npython example/example.py\npython example/example_advanced.py\n```\n\n---\n\n## \ud83e\udd1d Contributing\n\nContributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Quick Start for Contributors\n\n```bash\n# Clone repository\ngit clone https://github.com/fabiocax/SimilarityText.git\ncd SimilarityText\n\n# Install in development mode\npip install -e .[transformers]\n\n# Run examples\npython example/example_advanced.py\n```\n\n---\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n## \ud83d\udc64 Author\n\n**Fabio Alberti**\n\n- Email: fabiocax@gmail.com\n- GitHub: [@fabiocax](https://github.com/fabiocax)\n\n---\n\n## \ud83d\udd17 Links\n\n- **GitHub**: [https://github.com/fabiocax/SimilarityText](https://github.com/fabiocax/SimilarityText)\n- **PyPI**: [https://pypi.org/project/SimilarityText/](https://pypi.org/project/SimilarityText/)\n- **Documentation**: [https://github.com/fabiocax/SimilarityText/blob/main/README.md](https://github.com/fabiocax/SimilarityText/blob/main/README.md)\n- **Issues**: [https://github.com/fabiocax/SimilarityText/issues](https://github.com/fabiocax/SimilarityText/issues)\n\n---\n\n## \ud83d\ude4f Acknowledgments\n\n- **sentence-transformers**: For providing excellent pre-trained models\n- **scikit-learn**: For robust ML algorithms\n- **NLTK**: For comprehensive NLP tools\n- All contributors and users of this library\n\n---\n\n## \u2b50 Star History\n\nIf you find this project useful, please consider giving it a star on GitHub!\n\n[](https://star-history.com/#fabiocax/SimilarityText&Date)\n\n---\n\n## \ud83d\udcc8 Roadmap\n\n- [ ] Add more pre-trained models\n- [ ] Batch processing API\n- [ ] GPU acceleration support\n- [ ] REST API server\n- [ ] Caching mechanisms\n- [ ] More language-specific optimizations\n- [ ] Integration with popular frameworks (FastAPI, Flask)\n\n---\n\n**Made with \u2764\ufe0f using Python and AI**\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Advanced text similarity and classification using AI and Machine Learning",
"version": "0.3.0",
"project_urls": {
"Bug Reports": "https://github.com/fabiocax/SimilarityText/issues",
"Changelog": "https://github.com/fabiocax/SimilarityText/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/fabiocax/SimilarityText/blob/main/README.md",
"Homepage": "https://github.com/fabiocax/SimilarityText",
"Source": "https://github.com/fabiocax/SimilarityText"
},
"split_keywords": [
"nlp",
" natural language processing",
" text similarity",
" semantic similarity",
" text classification",
" machine learning",
" deep learning",
" transformers",
" bert",
" sentence transformers",
" tf-idf",
" cosine similarity",
" sentiment analysis",
" multilingual",
" ai",
" artificial intelligence"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f9e88025fdad431a16467b1f1f9faa9695197c38f20ec62f16b928b5ee79f85e",
"md5": "b5dd9e332dcf5506c1b6e1227bcff8fd",
"sha256": "13df0c5d595678f794aa1946ba96f5980f9cb481525c08014ac750d13a62dca6"
},
"downloads": -1,
"filename": "SimilarityText-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b5dd9e332dcf5506c1b6e1227bcff8fd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 13741,
"upload_time": "2025-10-06T18:30:09",
"upload_time_iso_8601": "2025-10-06T18:30:09.714703Z",
"url": "https://files.pythonhosted.org/packages/f9/e8/8025fdad431a16467b1f1f9faa9695197c38f20ec62f16b928b5ee79f85e/SimilarityText-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4ac8451e807ee10c0eebb32ee406d4dfeb0e684ba4c84f6045d0916bfdca38ab",
"md5": "509995c1d80ab45aca7b54063f849257",
"sha256": "0504c9f17b97746ff19d20412fe806ffb1b978063c1c7e46c26d88488e7b1030"
},
"downloads": -1,
"filename": "SimilarityText-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "509995c1d80ab45aca7b54063f849257",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 38541,
"upload_time": "2025-10-06T18:30:11",
"upload_time_iso_8601": "2025-10-06T18:30:11.612523Z",
"url": "https://files.pythonhosted.org/packages/4a/c8/451e807ee10c0eebb32ee406d4dfeb0e684ba4c84f6045d0916bfdca38ab/SimilarityText-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-06 18:30:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fabiocax",
"github_project": "SimilarityText",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "click",
"specs": []
},
{
"name": "joblib",
"specs": []
},
{
"name": "nltk",
"specs": []
},
{
"name": "regex",
"specs": []
},
{
"name": "scikit-learn",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "threadpoolctl",
"specs": []
},
{
"name": "tqdm",
"specs": []
},
{
"name": "langdetect",
"specs": []
},
{
"name": "numpy",
"specs": []
}
],
"lcname": "similaritytext"
}
JSON
