[](https://opensource.org/licenses/Apache-2.0) [](https://pepy.tech/project/caribe) [](https://huggingface.co/docs/transformers/model_doc/t5) [](https://pandas.pydata.org/) []() [](https://huggingface.co/) [](https://happytransformer.com/) []() []() [](https://huggingface.co/KES/T5-KES) [](https://huggingface.co/KES/T5-TTParser) [](https://huggingface.co/KES/caribe-capitalise) [](https://huggingface.co/KES/TEC-English)
<p align="center">
<img src="https://github.com/KestonSmith/CaribeLogo/raw/master/CaribeLOGO.png" title="Caribe">
# Caribe
>This is a natural processing python library takes Caribbean English Creoles and converts it to Standard English.
Future updates would include the conversion of other Caribbean English Creole languages to Standard English and additional natural language processing methods.
____
## Installation
Use the below command to install package/library
```
pip install Caribe
```
____
## Visit the website: <a href="https://www.thecaribe.org/">here</a>
---
## Main Usage
- ### Trinidad English Creole to English
```python
import Caribe as cb
text= "Dem men doh kno wat dey doing wid d money bai"
output= cb.tec_translator(text)
print(output.tec_translate()) #Output: These men do not know what they are doing with the money.
```
- ### English to Trinidad English Creole
```python
import Caribe as cb
text= "Where are you going now?"
output= cb.english_to_tec(text)
print(output.translate()) #Output: Weh yuh going now
```
- ### Guyanese English Creole to English
```python
from Caribe import guyanese as gy
text= "Me and meh kozn waan ah job"
output= gy.gec_translator(text)
print(output.gec_translate()) #Output: Me and my cousin want a job.
```
____
## Other Usages
> Sample 1: Checks the english creole input against existing known creole phrases before decoding the sentence into a more standardized version of English language. A corrector is used to check and fix small grammatical errors.
```python
# Sample 1
import Caribe as cb
sentence = "Dey have dey reasons"
standard = cb.phrase_decode(sentence)
standard = cb.trinidad_decode(standard)
fixed = cb.caribe_corrector(standard)
print(fixed) #Output: They have their reasons.
```
>Sample 2: Checks the trinidad english creole input against existing known phrases
```python
# Sample 2
import Caribe as cb
sentence = "Waz de scene"
standard = cb.phrase_decode(sentence)
print(standard) # Outputs: How are you
```
>Sample 3: Checks the sentence for any grammatical errors or incomplete words and corrects it.
```python
#Sample 3
import Caribe as cb
sentence = "I am playin fotball outsde"
standard = cb.caribe_corrector(sentence)
print(standard) # Outputs: I am playing football outside
```
>Sample 4: Makes parts of speech tagging on creole words.
```python
#Sample 4
import Caribe as cb
sentence = "wat iz de time there"
analyse = cb.nlp()
output = analyse.caribe_pos(sentence)
print(output) # Outputs: ["('wat', 'PRON')", "('iz', 'VERB')", "('de', 'DET')", "('time', 'NOUN')", "('there', 'ADV')"]
```
>Sample 5: Remove punctuation marks.
```python
#Sample 5
import Caribe as cb
sentence = "My aunt, Shelly is a lawyer!"
analyse = cb.remove_signs(sentence)
print(analyse) # Outputs: My aunt Shelly is a lawyer
```
>Sample 6: Sentence Correction using T5-KES.
```python
#Sample 6 Using t5_kes_corrector
import Caribe as cb
sentence = "Wat you doin for d the christmas"
correction = cb.t5_kes_corrector(sentence)
print(correction) # Output: What are you doing for christmas?
```
>Sample 7: Sentence Correction using Decoder and T5-KES.
```python
#Sample 7 Using t5_kes_corrector and decoder
import Caribe as cb
sentence = "Ah want ah phone for d christmas"
decoded= cb.trinidad_decode(sentence)
correction = cb.t5_kes_corrector(decoded)
print(correction) # Output: I want a phone for christmas.
```
>Sample 8: Sentence Capitalisation.
```python
#Sample 7 Using Caribe sentence capitalization model
import Caribe as cb
sentence = "john is a boy. he is 12 years old. his sister's name is Joy."
capitalized_text= cb.capitalize(sentence)
print(capitalized_text) # Output: John is a boy. He is 12 years old. His sister's name is Joy.
```
---
- ## Additional Information
- `trinidad_decode()` : Decodes the sentence as a whole string.
- `guyanese_decode()`: Decodes the sentence as a whole string.
- `trinidad_decode_split()`: Decodes the sentence word by word.
- `phrase_decode()`: Decodes the sentence against known creole phrases.
- `caribe_corrector()`: Corrects grammatical errors in a sentence using a trained NLP model.
- `t5_kes_corrector()`: Corrects grammatical errors in a sentence using a trained NLP model.
- `trinidad_encode()`: Encodes a sentence to Trinidadian English Creole.
- `guyanese_encode()`: Encodes a sentence to Guyanese English Creole.
- `trinidad_direct_translation()`: Translates Trinidad English Creole to English.
- `capitalize()`: Capitalize groups of sentences using an NLP model.
- `caribe_pos()`: Generates parts of speech tagging on creole words.
- `pos_report()`: Generates parts of speech tagging on english words.
- `remove_signs()`: Takes any sentence and remove punctuation marks.
---
- ## File Encodings on NLP datasets
Caribe introduces file encoding (Beta) in version 0.1.0. This allows a dataset of any supported filetype to be translated into Trinidad English Creole. The file encoding feature only supports txt, json or csv files only.
- ### Usage of File Encodings:
```python
import Caribe as cb
convert = cb.file_encode("test.txt", "text")
# Generates a translated text file
convert = cb.file_encode("test.json", "json")
# Generates a translated json file
convert = cb.file_encode("test.csv", "csv")
# Generates a translated csv file
```
---
- ## First Parser for the Trinidad English Creole Language
This model utilises T5-base pre-trained model. It was fine tuned using a combination of a custom dataset and creolised JFLEG dataset. JFLEG dataset was translated using the file encoding feature of the library.
Within the Creole continuum, there exists different levels of lects. These include:
- Acrolect: The version of a language closest to standard international english.
- Mesolect: The version that consists of a mixture of arcolectal and basilectal features.
- Basilect: The version closest to a Creole language.
**This NLP task was difficult because the structure of local dialect is not standardised but often rely on the structure of its lexifier (English). Spelling also varies from speaker to speaker. Additionally, creole words/spelling are not initial present in the vector space which made training times and optimization longer .**
## Results
Initial results have been mixed.
| Original Text | Parsed Text | Expected or Correctly Parsed Text |
|------------------------------------------|-------------------------------------|-------------------------------------|
| Ah have live with mi paremnts en London. | Ah live with meh parents in London. | Ah live with meh parents in London. |
| Ah can get me fone? | Ah cud get meh fone? | Ah cud get meh fone? |
| muh moda an fada is nt relly home | muh moda an fada is nt relly home. | muh moda an fada not really home. |
| Me ah go market | Ah going tuh d market. | Ah going tuh d market. / I going tuh de market. |
| Ah waz a going tu school. | Ah going to school. | Ah going to school. |
| Ah don't like her. | Ah doh like she. | Ah doh like she. / I doh like she. |
| Ah waz thinking bout goeng tuh d d Mall. | Ah thinking bout going tuh d Mall. | Ah thinking bout going tuh d mall. |
### Usage of the TrinEC Parser
```python
import Caribe as cb
text= "Ah have live with mi paremnts en London"
s= cb.Parser(text)
print(s.TT_Parser()) #Output: Ah live with meh parents in London.
```
---
## Trinidad English Creole to English Translator using the T5 model
A model was fine-tuned(supervised) on a custom dataset to translate from Trinidad English Creole to English. This task was done as an alternative method to the decoded dictionary- sentence correction method. Future Testing will illustrate a comparison between both methods.
```python
import Caribe as cb
text= "Dem men doh kno wat dey doing wid d money bai"
output= cb.tec_translator(text)
print(output.tec_translate()) #Output: These men do not know what they are doing with the money.
```
---
## Dictionary Data
The encoder and decoder utilises a dictionary data structure. The data for the dictionary was gathered from web-scapping social media sites among other websites and using Lise Winer Dictionary of the English Creole of Trinidad and Tobago among other scholarly resources.
____
## Fine-tune a T5 model on custom datasets easier using Caribe built on HuggingFace Transformers APIs
### Training Section:
Caribe allows any user to fine-tune a T5 model on a custom dataset. The snippet below trains and generates a model in the "model/" folder.
Please ensure that your training and evaluation datasets are in the recommended format before training. For more info checkout [T5 documentation.](https://huggingface.co/docs/transformers/main/en/model_doc/t5#t5)
```python
from Caribe import T5_Caribe as t5
model = t5.T5_Trainer("train_dataset.csv", "eval_dataset.csv")
connect = model.connect_datasets("csv")
train = model.caribe_training(output_path="./content", epochs=10, eval_strategy="steps", decay=0.01, l_rate=2e-5, train_batch_size=8, eval_batch_size=8, checkpoints=2)
```
### Parameters:
- Epochs : Num of training iterations.
- eval_strategy: Displays training in 'steps' or 'epoch'
- decay: Regularization of Training weights.
- l_rate: Learning rate.
- train_batch_size: Number of samples from the training data per iteration.
- eval_batch_size: Number of samples from the evaluation data per iteration.
- checkpoints: Produces and saves preset amount of model versions during training.
### Generating Text Section:
Generates text from the output of the model. Please note that if you have not train and generate the model folder or have a pre-existing model folder with the required files, the below code will generate an error.
```python
from Caribe import caribe_generate
g = caribe_generate.generate_text("eng:How are you", temperature=1.7, num_beams=10)
output=g.output()
print(output)
```
### Parameters:
- min_length: Minimum number of generated tokens.
- max_length:Max number of generated tokens.
- do_sample: When True, picks words based on their conditional probability.
- early_stopping: If true,stops the beam search when the least amount of num beams sentences have been completed each batch.
- num_beams: Number of steps for each search path.
- temperature: The value utilized to calculate the likelihood of the next token.
- top_k:The tokens with the greatest likelihood should be retained for top-k sampling.
- top_p: Most tokens with the highest probabilities that add up to top_p or higher.
- no_repeat_ngram_size: The amount of times an n-gram that size can only occur once.
---
## Caribe_Corrector (T5-KES) vs Gingerit Corrector
Initial Tests were carried out with performance factors to measure the accuracy of the correction and the degree of sentence distortion from the correct sentence. Initial tests showed that the T5 corrector performed better in terms of accuracy with a lower sentence distortion and attained higher MT scores. The T5 corrector also outperforms Gingerit on positional translations as shown in the table below.
| **Original Creole Text** | **Decoded Sentence** | **Caribe Corrector (T5-KES)** | **Gingerit Corrector** | **Correct Output** |
|----------------------------------|------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------|
| Ah need ah car fuh meh birthday | I need I car for me birthday | I need a car for my birthday. | I need my car for my birthday | I need a car for my birthday |
| Wah iz d time on yuh side? | want is the time on you side? | What is the time on your side? | Want is the time on your side? | What is the time on your side? |
| Ah man is d provider of de house | I man is the provider of the house | A man is the provider of the house. | My man is the provider of the house | A man is the provider of the house. |
| Ah orange issa fruit | I orange is a fruit | An orange is a fruit. | My orange is a fruit | An orange is a fruit. |
| Wah time yuh wah come over? | want time you want come over? | What time do you want to come over? | Want time you want to come over? | What time do you want to come over? |
|Dey make dey own choices at the end of de day |their make their own choices at the end of the day |They make their own choices at the end of the day. |they make their own choices at the end of the day. |They make their own choices at the end of the day. |
---
# Guyanese English Creole Features
Caribe introduces decoding, encoding and file encoding using Guyanese English Creole and translating Guyanese dialect.
## Guyanese English Creole to English
```python
from Caribe import guyanese as gy
text= "Me and meh kozn waan ah job"
output= gy.gec_translator(text)
print(output.gec_translate()) #Output: Me and my cousin want a job.
```
- ### Decoding a sentence
```python
# Decoding a creole sentence
from Caribe import guyanese as gy
sentence = "waam star, me ga fu go di markit"
output = gy.guyanese_decode(sentence)
print(output) # Output: what going on brother, me have to go the market
```
- ### Encoding a sentence
```python
# Encoding a sentence
from Caribe import guyanese as gy
sentence = "I do not want nothing to do with him"
output = gy.guyanese_encode(sentence)
print(output) # Output: ah du not waan notn tuh du wid him
```
- ### File Encodings on NLP datasets using Guyanese English Creole:
```python
from Caribe import guyanese as gy
convert = gy.guyanese_file_encode("test.txt", "text")
# Generates a translated text file
convert = gy.guyanese_file_encode("test.json", "json")
# Generates a translated json file
convert = gy.guyanese_file_encode("test.csv", "csv")
# Generates a translated csv file
```
---
# News, Issues and Future Plans (14/08/2022)
- Datasets are continuously being updated.
- NLP Models and Dictionaries are continuously updated.
- Future plans to create translations, models and datasets for Caribbean French and Spanish Creoles to their respective lexifers (Requires extensive research).
- Some users complained of problems importing some of the dependencies. This is currently being monitored (10/06/2022).
- New model introduced for sentence capitalization (09/06/2022) !!!
- NEW model introduced for direct translation from Trinidad English Creole(TEC) to English(26/06/2022).
- NEW model introduced for direct translation from Guyanese English Creole(GEC) to English(26/09/2022).
- The gingerit_corrector function is deprecated.
---
- ## Contact
For any concerns or issues with this library.
Email: keston.smith@my.uwi.edu
Website: https://www.thecaribe.org/
___
CHANGELOG =======================================
Version 0.0.1 (16/09/2021)
- Initial Release
Version 0.0.2 (16/09/2021)
- Minor bugs fixed
- More words added
Version 0.0.3 (16/09/2021)
- Minor bugs fixed
- More words added
- phase decode method created
Version 0.0.4 (17/09/2021)
- More words added
- caribe corrector method created
Version 0.0.5 (17/09/2021)
- Minor Dependency issues resolved
Version 0.0.6 (17/09/2021)
- More Words and phrases added
Version 0.0.7 (21/09/2021)
- Major bug fixed where individual letters in words were translated randomly
- More words added to the corpus.
Version 0.0.8 (30/09/2021)
- caribe_pos tagging method introduced.
- pos_report method introduced.
- remove_signs method introduced.
Version 0.1.0 (14/10/2021)
- trinidad_encode method converts standard english sentence to a creolised form.
- Caribe introduces dialect file encoding on text, json and csv files. This has the ability to creolised nlp datasets.
Version 0.1.1 (20/10/2021)
- More words added to the corpus.
Version 0.1.2 (27/10/2021)
- caribe_pos members converted from string to tuple.
- More words added to the corpus.
Version 0.1.5 (13/11/2021)
- More words added to the corpus.
Version 0.1.8 (10/12/2021)
- Introducing a new trained NLP grammar correction model "T5-KES".
- Major Update in version 0.2.0 coming January.
Version 0.2.0 (06/02/2022)
- New Sentence Corrector introduced: t5_kes_corrector()
Version 0.2.1 (07/02/2022)
- More Words added to the corpus
Version 0.2.2 (13/02/2022)
- Removal of a dependency that causes slower functionality
Version 0.2.4 (27/02/2022)
- Introduced a new api function to the corrector: t5_kes_api_corrector(sentence) for faster performance
Version 0.2.9 (28/02/2022)
- More Words added
- Fixed an issue where certain words were not being encoded on NLP datasets.
Version 0.3.0 (03/03/2022)
- Introducing the first ever parser for Trinidad and Tobago Creole English.
- Model was trained on a combination of a custom dataset and a creolised JFLEG dataset using the file encoding feature.
Version 0.3.1 (06/03/2022)
- Updated TTEC Parser model (Reduced training loss and trained on more custom data).
- New api function for parser for faster performance TTparser_api().
- Warnings/Issues: Api function calls may result in contradicting results from the main function or may sometimes result in Key Errors. Use non-api functions to achieve better results.
Version 0.3.4 (11/03/2022)
- All api functions will disabled in V-0.3.6 until further notice.
Version 0.3.6 (14/03/2022)
- More words added to the corpus.
- Api functions issues resolved.
Version 0.4.7 (20/03/2022)
- NLP model retrain and improved.
- Official Logo Added.
Version 0.5.0 (27/03/2022)
- More Words Added.
Version 0.5.1 (21/04/2022)
- More Words Added.
- trinidad_direct_translation function added to directly translate Trinidad English Creole to English.
- caribe_corrector function will not longer use gingerit and will be replaced with the T5 model.
- gingerit_corrector function added to facilitate the gingerit model.
Version 0.5.3 (05/06/2022)
- More Words Added.
- Trinidad English Creole Parser Model updated.
Version 0.5.5 (09/06/2022)
- New NLP model introdcued for sentence capitalization.
- T5-KES sentence correction model updated.
Version 0.5.9 (10/06/2022)
- Resolving minor dependency issues.
Version 0.6.4 (26/06/2022)
- Introducing a NEW NLP model to directly translate Trinidad English Creole to English.
Version 0.6.6 (29/06/2022)
- More Words Added.
Version 0.6.8 (02/07/2022)
- More Words Added
- Model Updated
- Entries Changed or Removed
Version 0.7.5 (31/07/2022)
- Introducing an easier method to Fine-tuning a T5-model on custom datasets.
Version 0.7.7 (14/08/2022)
- Introducing Guyanese English Creole
Version 0.7.8 (04/09/2022)
- More Words Added
Version 0.7.9 (26/09/2022)
- Introducing a NEW NLP model to directly translate Guyanese English Creole to English.
Version 0.8.1 (02/10/2022)
- Introducing a NEW NLP model to directly translate Standard English to Trinidad English Creole.
Version 0.9.0 (08/08/2023)
- Resolved dependency issues with cloud computing services where the service refused to install the latest version of package because of a missing/deleted dependency.
- The gingerit_corrector function is deprecated from version 0.9.0 and above.
- Gingerit package no longer exists. Dependency removed.
Version 0.9.2 (02/12/2023)
- More words added to Guyanese creole corpus.
- More words added to Trinidad Creole corpus.
Version 0.9.4 (15/07/2024)
- More words added to Trinidad Creole corpus.
- Fixed minor dependency issues.
Raw data
{
"_id": null,
"home_page": null,
"name": "Caribe",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "Translator",
"author": "Keston Smith",
"author_email": "keston.smith@my.uwi.edu",
"download_url": "https://files.pythonhosted.org/packages/0f/84/a09e041652ddfdc30fdc270aaa1f36a3444742fe401073e8b6bd90cc3637/Caribe-0.9.4.tar.gz",
"platform": null,
"description": "\r\n[](https://opensource.org/licenses/Apache-2.0) [](https://pepy.tech/project/caribe) [](https://huggingface.co/docs/transformers/model_doc/t5) [](https://pandas.pydata.org/) []() [](https://huggingface.co/) [](https://happytransformer.com/) []() []() [](https://huggingface.co/KES/T5-KES) [](https://huggingface.co/KES/T5-TTParser) [](https://huggingface.co/KES/caribe-capitalise) [](https://huggingface.co/KES/TEC-English)\r\n\r\n<p align=\"center\">\r\n<img src=\"https://github.com/KestonSmith/CaribeLogo/raw/master/CaribeLOGO.png\" title=\"Caribe\">\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n# Caribe \r\n\r\n\r\n>This is a natural processing python library takes Caribbean English Creoles and converts it to Standard English.\r\nFuture updates would include the conversion of other Caribbean English Creole languages to Standard English and additional natural language processing methods.\r\n\r\n____\r\n\r\n## Installation\r\n \r\nUse the below command to install package/library\r\n```\r\npip install Caribe \r\n\r\n```\r\n____\r\n## Visit the website: <a href=\"https://www.thecaribe.org/\">here</a> \r\n---\r\n## Main Usage\r\n- ### Trinidad English Creole to English\r\n```python\r\n\r\nimport Caribe as cb\r\n\r\ntext= \"Dem men doh kno wat dey doing wid d money bai\"\r\n\r\noutput= cb.tec_translator(text)\r\n\r\nprint(output.tec_translate()) #Output: These men do not know what they are doing with the money.\r\n\r\n\r\n```\r\n- ### English to Trinidad English Creole \r\n```python\r\n\r\nimport Caribe as cb\r\n\r\ntext= \"Where are you going now?\"\r\n\r\noutput= cb.english_to_tec(text)\r\n\r\nprint(output.translate()) #Output: Weh yuh going now\r\n\r\n\r\n```\r\n- ### Guyanese English Creole to English\r\n```python\r\nfrom Caribe import guyanese as gy\r\n\r\ntext= \"Me and meh kozn waan ah job\"\r\n\r\noutput= gy.gec_translator(text)\r\n\r\nprint(output.gec_translate()) #Output: Me and my cousin want a job.\r\n\r\n\r\n```\r\n____\r\n ## Other Usages\r\n > Sample 1: Checks the english creole input against existing known creole phrases before decoding the sentence into a more standardized version of English language. A corrector is used to check and fix small grammatical errors.\r\n```python\r\n# Sample 1\r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"Dey have dey reasons\"\r\nstandard = cb.phrase_decode(sentence)\r\nstandard = cb.trinidad_decode(standard)\r\nfixed = cb.caribe_corrector(standard)\r\nprint(fixed) #Output: They have their reasons.\r\n\r\n```\r\n>Sample 2: Checks the trinidad english creole input against existing known phrases\r\n```python\r\n# Sample 2 \r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"Waz de scene\"\r\nstandard = cb.phrase_decode(sentence)\r\n\r\nprint(standard) # Outputs: How are you\r\n\r\n```\r\n>Sample 3: Checks the sentence for any grammatical errors or incomplete words and corrects it.\r\n```python\r\n#Sample 3\r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"I am playin fotball outsde\"\r\nstandard = cb.caribe_corrector(sentence)\r\n\r\nprint(standard) # Outputs: I am playing football outside\r\n\r\n```\r\n>Sample 4: Makes parts of speech tagging on creole words.\r\n```python\r\n#Sample 4\r\nimport Caribe as cb\r\n\r\nsentence = \"wat iz de time there\"\r\nanalyse = cb.nlp()\r\noutput = analyse.caribe_pos(sentence)\r\n\r\nprint(output) # Outputs: [\"('wat', 'PRON')\", \"('iz', 'VERB')\", \"('de', 'DET')\", \"('time', 'NOUN')\", \"('there', 'ADV')\"]\r\n\r\n```\r\n>Sample 5: Remove punctuation marks.\r\n```python\r\n#Sample 5\r\nimport Caribe as cb\r\n\r\nsentence = \"My aunt, Shelly is a lawyer!\"\r\nanalyse = cb.remove_signs(sentence)\r\n\r\n\r\nprint(analyse) # Outputs: My aunt Shelly is a lawyer\r\n\r\n```\r\n\r\n>Sample 6: Sentence Correction using T5-KES.\r\n```python\r\n#Sample 6 Using t5_kes_corrector\r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"Wat you doin for d the christmas\"\r\ncorrection = cb.t5_kes_corrector(sentence)\r\n\r\n\r\nprint(correction) # Output: What are you doing for christmas?\r\n\r\n```\r\n>Sample 7: Sentence Correction using Decoder and T5-KES.\r\n```python\r\n#Sample 7 Using t5_kes_corrector and decoder\r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"Ah want ah phone for d christmas\"\r\ndecoded= cb.trinidad_decode(sentence)\r\ncorrection = cb.t5_kes_corrector(decoded)\r\n\r\n\r\nprint(correction) # Output: I want a phone for christmas.\r\n\r\n```\r\n\r\n>Sample 8: Sentence Capitalisation.\r\n```python\r\n#Sample 7 Using Caribe sentence capitalization model\r\nimport Caribe as cb\r\n\r\n\r\nsentence = \"john is a boy. he is 12 years old. his sister's name is Joy.\"\r\n\r\ncapitalized_text= cb.capitalize(sentence)\r\n\r\nprint(capitalized_text) # Output: John is a boy. He is 12 years old. His sister's name is Joy.\r\n\r\n```\r\n\r\n---\r\n- ## Additional Information\r\n - `trinidad_decode()` : Decodes the sentence as a whole string.\r\n - `guyanese_decode()`: Decodes the sentence as a whole string.\r\n - `trinidad_decode_split()`: Decodes the sentence word by word.\r\n - `phrase_decode()`: Decodes the sentence against known creole phrases.\r\n - `caribe_corrector()`: Corrects grammatical errors in a sentence using a trained NLP model.\r\n - `t5_kes_corrector()`: Corrects grammatical errors in a sentence using a trained NLP model.\r\n - `trinidad_encode()`: Encodes a sentence to Trinidadian English Creole.\r\n - `guyanese_encode()`: Encodes a sentence to Guyanese English Creole.\r\n - `trinidad_direct_translation()`: Translates Trinidad English Creole to English.\r\n - `capitalize()`: Capitalize groups of sentences using an NLP model.\r\n - `caribe_pos()`: Generates parts of speech tagging on creole words.\r\n - `pos_report()`: Generates parts of speech tagging on english words.\r\n - `remove_signs()`: Takes any sentence and remove punctuation marks. \r\n\r\n---\r\n- ## File Encodings on NLP datasets\r\nCaribe introduces file encoding (Beta) in version 0.1.0. This allows a dataset of any supported filetype to be translated into Trinidad English Creole. The file encoding feature only supports txt, json or csv files only.\r\n\r\n- ### Usage of File Encodings:\r\n```python\r\nimport Caribe as cb\r\n\r\nconvert = cb.file_encode(\"test.txt\", \"text\")\r\n# Generates a translated text file\r\nconvert = cb.file_encode(\"test.json\", \"json\")\r\n# Generates a translated json file\r\nconvert = cb.file_encode(\"test.csv\", \"csv\")\r\n# Generates a translated csv file\r\n\r\n\r\n```\r\n---\r\n- ## First Parser for the Trinidad English Creole Language\r\n\r\nThis model utilises T5-base pre-trained model. It was fine tuned using a combination of a custom dataset and creolised JFLEG dataset. JFLEG dataset was translated using the file encoding feature of the library. \r\n\r\nWithin the Creole continuum, there exists different levels of lects. These include: \r\n\r\n- Acrolect: The version of a language closest to standard international english.\r\n- Mesolect: The version that consists of a mixture of arcolectal and basilectal features.\r\n- Basilect: The version closest to a Creole language.\r\n\r\n**This NLP task was difficult because the structure of local dialect is not standardised but often rely on the structure of its lexifier (English). Spelling also varies from speaker to speaker. Additionally, creole words/spelling are not initial present in the vector space which made training times and optimization longer .**\r\n\r\n## Results\r\nInitial results have been mixed.\r\n\r\n| Original Text | Parsed Text | Expected or Correctly Parsed Text |\r\n|------------------------------------------|-------------------------------------|-------------------------------------|\r\n| Ah have live with mi paremnts en London. | Ah live with meh parents in London. | Ah live with meh parents in London. |\r\n| Ah can get me fone? | Ah cud get meh fone? | Ah cud get meh fone? |\r\n| muh moda an fada is nt relly home | muh moda an fada is nt relly home. | muh moda an fada not really home. |\r\n| Me ah go market | Ah going tuh d market. | Ah going tuh d market. / I going tuh de market. |\r\n| Ah waz a going tu school. | Ah going to school. | Ah going to school. |\r\n| Ah don't like her. | Ah doh like she. | Ah doh like she. / I doh like she. |\r\n| Ah waz thinking bout goeng tuh d d Mall. | Ah thinking bout going tuh d Mall. | Ah thinking bout going tuh d mall. |\r\n\r\n\r\n### Usage of the TrinEC Parser\r\n\r\n```python\r\nimport Caribe as cb\r\n\r\ntext= \"Ah have live with mi paremnts en London\"\r\n\r\ns= cb.Parser(text)\r\n\r\nprint(s.TT_Parser()) #Output: Ah live with meh parents in London.\r\n\r\n\r\n```\r\n---\r\n## Trinidad English Creole to English Translator using the T5 model\r\nA model was fine-tuned(supervised) on a custom dataset to translate from Trinidad English Creole to English. This task was done as an alternative method to the decoded dictionary- sentence correction method. Future Testing will illustrate a comparison between both methods.\r\n\r\n```python\r\nimport Caribe as cb\r\n\r\ntext= \"Dem men doh kno wat dey doing wid d money bai\"\r\n\r\noutput= cb.tec_translator(text)\r\n\r\nprint(output.tec_translate()) #Output: These men do not know what they are doing with the money.\r\n\r\n\r\n```\r\n\r\n---\r\n## Dictionary Data\r\nThe encoder and decoder utilises a dictionary data structure. The data for the dictionary was gathered from web-scapping social media sites among other websites and using Lise Winer Dictionary of the English Creole of Trinidad and Tobago among other scholarly resources.\r\n____\r\n## Fine-tune a T5 model on custom datasets easier using Caribe built on HuggingFace Transformers APIs \r\n\r\n### Training Section:\r\nCaribe allows any user to fine-tune a T5 model on a custom dataset. The snippet below trains and generates a model in the \"model/\" folder. \r\nPlease ensure that your training and evaluation datasets are in the recommended format before training. For more info checkout [T5 documentation.](https://huggingface.co/docs/transformers/main/en/model_doc/t5#t5) \r\n```python\r\nfrom Caribe import T5_Caribe as t5\r\n\r\nmodel = t5.T5_Trainer(\"train_dataset.csv\", \"eval_dataset.csv\")\r\nconnect = model.connect_datasets(\"csv\")\r\ntrain = model.caribe_training(output_path=\"./content\", epochs=10, eval_strategy=\"steps\", decay=0.01, l_rate=2e-5, train_batch_size=8, eval_batch_size=8, checkpoints=2)\r\n\r\n```\r\n### Parameters:\r\n - Epochs : Num of training iterations.\r\n\r\n - eval_strategy: Displays training in 'steps' or 'epoch'\r\n - decay: Regularization of Training weights.\r\n\r\n - l_rate: Learning rate.\r\n\r\n - train_batch_size: Number of samples from the training data per iteration.\r\n\r\n - eval_batch_size: Number of samples from the evaluation data per iteration.\r\n\r\n - checkpoints: Produces and saves preset amount of model versions during training.\r\n\r\n ### Generating Text Section:\r\n Generates text from the output of the model. Please note that if you have not train and generate the model folder or have a pre-existing model folder with the required files, the below code will generate an error.\r\n ```python\r\nfrom Caribe import caribe_generate\r\n\r\ng = caribe_generate.generate_text(\"eng:How are you\", temperature=1.7, num_beams=10)\r\noutput=g.output()\r\nprint(output)\r\n\r\n ```\r\n### Parameters:\r\n- min_length: Minimum number of generated tokens.\r\n\r\n- max_length:Max number of generated tokens.\r\n\r\n- do_sample: When True, picks words based on their conditional probability.\r\n\r\n- early_stopping: If true,stops the beam search when the least amount of num beams sentences have been completed each batch.\r\n\r\n- num_beams: Number of steps for each search path.\r\n\r\n- temperature: The value utilized to calculate the likelihood of the next token.\r\n\r\n- top_k:The tokens with the greatest likelihood should be retained for top-k sampling.\r\n\r\n- top_p: Most tokens with the highest probabilities that add up to top_p or higher.\r\n\r\n- no_repeat_ngram_size: The amount of times an n-gram that size can only occur once. \r\n\r\n---\r\n## Caribe_Corrector (T5-KES) vs Gingerit Corrector\r\nInitial Tests were carried out with performance factors to measure the accuracy of the correction and the degree of sentence distortion from the correct sentence. Initial tests showed that the T5 corrector performed better in terms of accuracy with a lower sentence distortion and attained higher MT scores. The T5 corrector also outperforms Gingerit on positional translations as shown in the table below.\r\n\r\n| **Original Creole Text** | **Decoded Sentence** | **Caribe Corrector (T5-KES)** | **Gingerit Corrector** | **Correct Output** |\r\n|----------------------------------|------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------|\r\n| Ah need ah car fuh meh birthday | I need I car for me birthday | I need a car for my birthday. | I need my car for my birthday | I need a car for my birthday |\r\n| Wah iz d time on yuh side? | want is the time on you side? | What is the time on your side? | Want is the time on your side? | What is the time on your side? |\r\n| Ah man is d provider of de house | I man is the provider of the house | A man is the provider of the house. | My man is the provider of the house | A man is the provider of the house. |\r\n| Ah orange issa fruit | I orange is a fruit | An orange is a fruit. | My orange is a fruit | An orange is a fruit. |\r\n| Wah time yuh wah come over? | want time you want come over? | What time do you want to come over? | Want time you want to come over? | What time do you want to come over? |\r\n|Dey make dey own choices at the end of de day |their make their own choices at the end of the day |They make their own choices at the end of the day. |they make their own choices at the end of the day. |They make their own choices at the end of the day. |\r\n\r\n---\r\n# Guyanese English Creole Features\r\nCaribe introduces decoding, encoding and file encoding using Guyanese English Creole and translating Guyanese dialect. \r\n\r\n\r\n## Guyanese English Creole to English\r\n```python\r\nfrom Caribe import guyanese as gy\r\n\r\ntext= \"Me and meh kozn waan ah job\"\r\n\r\noutput= gy.gec_translator(text)\r\n\r\nprint(output.gec_translate()) #Output: Me and my cousin want a job.\r\n\r\n\r\n```\r\n\r\n- ### Decoding a sentence\r\n```python\r\n# Decoding a creole sentence\r\n\r\nfrom Caribe import guyanese as gy\r\n\r\nsentence = \"waam star, me ga fu go di markit\"\r\noutput = gy.guyanese_decode(sentence)\r\nprint(output) # Output: what going on brother, me have to go the market\r\n\r\n```\r\n- ### Encoding a sentence\r\n```python\r\n# Encoding a sentence\r\n\r\nfrom Caribe import guyanese as gy\r\n\r\nsentence = \"I do not want nothing to do with him\"\r\noutput = gy.guyanese_encode(sentence)\r\nprint(output) # Output: ah du not waan notn tuh du wid him\r\n```\r\n\r\n- ### File Encodings on NLP datasets using Guyanese English Creole:\r\n```python\r\nfrom Caribe import guyanese as gy\r\n\r\nconvert = gy.guyanese_file_encode(\"test.txt\", \"text\")\r\n# Generates a translated text file\r\nconvert = gy.guyanese_file_encode(\"test.json\", \"json\")\r\n# Generates a translated json file\r\nconvert = gy.guyanese_file_encode(\"test.csv\", \"csv\")\r\n# Generates a translated csv file\r\n```\r\n\r\n---\r\n# News, Issues and Future Plans (14/08/2022)\r\n\r\n- Datasets are continuously being updated.\r\n- NLP Models and Dictionaries are continuously updated.\r\n- Future plans to create translations, models and datasets for Caribbean French and Spanish Creoles to their respective lexifers (Requires extensive research).\r\n- Some users complained of problems importing some of the dependencies. This is currently being monitored (10/06/2022).\r\n- New model introduced for sentence capitalization (09/06/2022) !!!\r\n- NEW model introduced for direct translation from Trinidad English Creole(TEC) to English(26/06/2022).\r\n- NEW model introduced for direct translation from Guyanese English Creole(GEC) to English(26/09/2022).\r\n\r\n- The gingerit_corrector function is deprecated. \r\n\r\n---\r\n- ## Contact \r\nFor any concerns or issues with this library.\r\n\r\nEmail: keston.smith@my.uwi.edu \r\n\r\nWebsite: https://www.thecaribe.org/\r\n___\r\n\r\nCHANGELOG =======================================\r\n\r\nVersion 0.0.1 (16/09/2021)\r\n\r\n- Initial Release\r\n\r\nVersion 0.0.2 (16/09/2021)\r\n\r\n- Minor bugs fixed\r\n- More words added\r\n\r\nVersion 0.0.3 (16/09/2021)\r\n\r\n- Minor bugs fixed\r\n- More words added\r\n- phase decode method created\r\n\r\nVersion 0.0.4 (17/09/2021)\r\n\r\n- More words added\r\n- caribe corrector method created\r\n\r\nVersion 0.0.5 (17/09/2021)\r\n\r\n- Minor Dependency issues resolved\r\n\r\nVersion 0.0.6 (17/09/2021)\r\n\r\n- More Words and phrases added\r\n\r\nVersion 0.0.7 (21/09/2021)\r\n\r\n- Major bug fixed where individual letters in words were translated randomly\r\n- More words added to the corpus.\r\n\r\nVersion 0.0.8 (30/09/2021)\r\n\r\n- caribe_pos tagging method introduced.\r\n- pos_report method introduced.\r\n- remove_signs method introduced.\r\n\r\nVersion 0.1.0 (14/10/2021)\r\n\r\n- trinidad_encode method converts standard english sentence to a creolised form.\r\n- Caribe introduces dialect file encoding on text, json and csv files. This has the ability to creolised nlp datasets.\r\n\r\nVersion 0.1.1 (20/10/2021)\r\n\r\n- More words added to the corpus.\r\n\r\nVersion 0.1.2 (27/10/2021)\r\n\r\n- caribe_pos members converted from string to tuple.\r\n- More words added to the corpus.\r\n\r\nVersion 0.1.5 (13/11/2021)\r\n\r\n- More words added to the corpus.\r\n\r\nVersion 0.1.8 (10/12/2021)\r\n\r\n- Introducing a new trained NLP grammar correction model \"T5-KES\".\r\n- Major Update in version 0.2.0 coming January.\r\n\r\nVersion 0.2.0 (06/02/2022)\r\n- New Sentence Corrector introduced: t5_kes_corrector()\r\n\r\nVersion 0.2.1 (07/02/2022)\r\n- More Words added to the corpus\r\n\r\nVersion 0.2.2 (13/02/2022)\r\n- Removal of a dependency that causes slower functionality\r\n\r\nVersion 0.2.4 (27/02/2022)\r\n- Introduced a new api function to the corrector: t5_kes_api_corrector(sentence) for faster performance\r\n\r\nVersion 0.2.9 (28/02/2022)\r\n- More Words added\r\n- Fixed an issue where certain words were not being encoded on NLP datasets.\r\n\r\nVersion 0.3.0 (03/03/2022)\r\n- Introducing the first ever parser for Trinidad and Tobago Creole English.\r\n- Model was trained on a combination of a custom dataset and a creolised JFLEG dataset using the file encoding feature.\r\n\r\nVersion 0.3.1 (06/03/2022)\r\n- Updated TTEC Parser model (Reduced training loss and trained on more custom data).\r\n- New api function for parser for faster performance TTparser_api().\r\n- Warnings/Issues: Api function calls may result in contradicting results from the main function or may sometimes result in Key Errors. Use non-api functions to achieve better results.\r\n\r\nVersion 0.3.4 (11/03/2022)\r\n- All api functions will disabled in V-0.3.6 until further notice.\r\n\r\nVersion 0.3.6 (14/03/2022)\r\n- More words added to the corpus.\r\n- Api functions issues resolved. \r\n\r\nVersion 0.4.7 (20/03/2022)\r\n- NLP model retrain and improved.\r\n- Official Logo Added.\r\n\r\nVersion 0.5.0 (27/03/2022)\r\n- More Words Added.\r\n\r\nVersion 0.5.1 (21/04/2022)\r\n- More Words Added. \r\n- trinidad_direct_translation function added to directly translate Trinidad English Creole to English.\r\n- caribe_corrector function will not longer use gingerit and will be replaced with the T5 model.\r\n- gingerit_corrector function added to facilitate the gingerit model.\r\n\r\nVersion 0.5.3 (05/06/2022)\r\n- More Words Added.\r\n- Trinidad English Creole Parser Model updated.\r\n\r\nVersion 0.5.5 (09/06/2022)\r\n- New NLP model introdcued for sentence capitalization.\r\n- T5-KES sentence correction model updated.\r\n\r\nVersion 0.5.9 (10/06/2022)\r\n- Resolving minor dependency issues.\r\n\r\nVersion 0.6.4 (26/06/2022)\r\n- Introducing a NEW NLP model to directly translate Trinidad English Creole to English.\r\n\r\nVersion 0.6.6 (29/06/2022)\r\n- More Words Added.\r\n\r\nVersion 0.6.8 (02/07/2022)\r\n- More Words Added\r\n- Model Updated\r\n- Entries Changed or Removed\r\n\r\nVersion 0.7.5 (31/07/2022)\r\n- Introducing an easier method to Fine-tuning a T5-model on custom datasets.\r\n\r\nVersion 0.7.7 (14/08/2022)\r\n- Introducing Guyanese English Creole\r\n\r\nVersion 0.7.8 (04/09/2022)\r\n- More Words Added\r\n\r\nVersion 0.7.9 (26/09/2022)\r\n- Introducing a NEW NLP model to directly translate Guyanese English Creole to English.\r\n\r\nVersion 0.8.1 (02/10/2022)\r\n- Introducing a NEW NLP model to directly translate Standard English to Trinidad English Creole.\r\n\r\nVersion 0.9.0 (08/08/2023)\r\n- Resolved dependency issues with cloud computing services where the service refused to install the latest version of package because of a missing/deleted dependency.\r\n- The gingerit_corrector function is deprecated from version 0.9.0 and above. \r\n- Gingerit package no longer exists. Dependency removed.\r\n\r\nVersion 0.9.2 (02/12/2023)\r\n- More words added to Guyanese creole corpus.\r\n- More words added to Trinidad Creole corpus.\r\n\r\nVersion 0.9.4 (15/07/2024)\r\n- More words added to Trinidad Creole corpus.\r\n- Fixed minor dependency issues.\r\n\r\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Caribbean English Creoles to Standard English",
"version": "0.9.4",
"project_urls": null,
"split_keywords": [
"translator"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0f84a09e041652ddfdc30fdc270aaa1f36a3444742fe401073e8b6bd90cc3637",
"md5": "a1e5e6c8565b2daffdf0361e6624477c",
"sha256": "659475a52fce78101901c5edfe5544063e033038c9091b2b1895de4b34b042df"
},
"downloads": -1,
"filename": "Caribe-0.9.4.tar.gz",
"has_sig": false,
"md5_digest": "a1e5e6c8565b2daffdf0361e6624477c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 27383,
"upload_time": "2024-07-15T18:32:46",
"upload_time_iso_8601": "2024-07-15T18:32:46.079663Z",
"url": "https://files.pythonhosted.org/packages/0f/84/a09e041652ddfdc30fdc270aaa1f36a3444742fe401073e8b6bd90cc3637/Caribe-0.9.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-15 18:32:46",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "caribe"
}
JSON