# HuLU evaluate
`hulu_evaluate` is a library for evaluating and training language models on Hungarian tasks within the HuLU benchmark. It includes support for fine-tuning with LoRA, official evaluation scripts, and provides a leaderboard for benchmarking.
## Features
- **Training on multiple tasks**: Supports tasks like "CoLA", "RTE", "WNLI", "CB", "SST", and "COPA".
- **LoRA (Low-Rank Adaptation)**: Fine-tune models with reduced compute requirements.
- **Official training scripts**: Provided for training and evaluation.
- **HuLU Leaderboard**: Submit your results to the HuLU leaderboard.
## Installation
To install the `hulu_evaluate` library, clone the repository and install dependencies:
```bash
pip install git+https://git.nlp.nytud.hu/DLT/HuLU-evaluate.git
```
## Usage
Command-Line Interface (CLI)
The CLI provides commands for configuration, training, and submission.
Setting Up the CLI
The CLI entry point can be called with:
```bash
hulu-evaluate <command> [<args>]
```
### Commands include:
## Train a Model: Train on HuLU tasks using the train command:
```bash
hulu-evaluate train --model_name <MODEL_NAME> --output_dir <OUTPUT_DIR> --train_epochs 6 --train_batch 8
```
You can submit your results on the hulu.nytud.hu webpage. The results are created by default in the "HuluFinetune" directory.
## Programmatic Usage
You can also integrate hulu_evaluate functionality directly in Python scripts.
Training
```python
from hulu_evaluate.train.hulu_main import HuluMain
from hulu_evaluate.hulu_arguments.train_arguments import HuluArgument
# Define training arguments
arguments = HuluArgument(
model_name="my_model",
train_epochs=6,
tasks=["cola", "rte"],
)
# Train the model
HuluMain(arguments)
```
## Command Reference
### Training Command
train - Trains a model on HuLU tasks with specified parameters.
```bash
hulu-evaluate train --model_name <MODEL_NAME> --output_dir <OUTPUT_DIR> --train_epochs <EPOCHS>
--model_name: Name of the model to train.
--train_epochs: Number of epochs.
--use_lora: Enable LoRA fine-tuning (optional).
```
### Contributing
Contributions are welcome! Please submit issues or pull requests to improve hulu_evaluate.
### License
This project is licensed under the Apache License.
Raw data
{
"_id": null,
"home_page": "https://hulu.nytud.hu/",
"name": "hulu-evaluate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": null,
"keywords": "machine-learning models natural-language-processing deep-learning evaluation benchmark",
"author": "HUN-REN Research Center for Linguistics",
"author_email": "varga.kristof@nytud.hun-ren.hu",
"download_url": "https://files.pythonhosted.org/packages/73/92/046c6514c588cdbcb0bc24d62b1fbc80119c8d5fffb387826ccaea57324b/hulu_evaluate-0.0.2.tar.gz",
"platform": null,
"description": "# HuLU evaluate\n\n`hulu_evaluate` is a library for evaluating and training language models on Hungarian tasks within the HuLU benchmark. It includes support for fine-tuning with LoRA, official evaluation scripts, and provides a leaderboard for benchmarking.\n\n## Features\n\n- **Training on multiple tasks**: Supports tasks like \"CoLA\", \"RTE\", \"WNLI\", \"CB\", \"SST\", and \"COPA\".\n- **LoRA (Low-Rank Adaptation)**: Fine-tune models with reduced compute requirements.\n- **Official training scripts**: Provided for training and evaluation.\n- **HuLU Leaderboard**: Submit your results to the HuLU leaderboard.\n\n## Installation\n\nTo install the `hulu_evaluate` library, clone the repository and install dependencies:\n\n```bash\npip install git+https://git.nlp.nytud.hu/DLT/HuLU-evaluate.git\n```\n\n## Usage\nCommand-Line Interface (CLI)\nThe CLI provides commands for configuration, training, and submission.\n\nSetting Up the CLI\nThe CLI entry point can be called with:\n\n```bash\nhulu-evaluate <command> [<args>]\n```\n### Commands include:\n\n## Train a Model: Train on HuLU tasks using the train command:\n\n```bash\nhulu-evaluate train --model_name <MODEL_NAME> --output_dir <OUTPUT_DIR> --train_epochs 6 --train_batch 8\n```\n\nYou can submit your results on the hulu.nytud.hu webpage. The results are created by default in the \"HuluFinetune\" directory. \n\n## Programmatic Usage\nYou can also integrate hulu_evaluate functionality directly in Python scripts.\n\nTraining\n```python\nfrom hulu_evaluate.train.hulu_main import HuluMain\nfrom hulu_evaluate.hulu_arguments.train_arguments import HuluArgument\n\n# Define training arguments\narguments = HuluArgument(\n model_name=\"my_model\",\n train_epochs=6,\n tasks=[\"cola\", \"rte\"],\n)\n\n# Train the model\nHuluMain(arguments)\n```\n\n## Command Reference\n### Training Command\ntrain - Trains a model on HuLU tasks with specified parameters.\n\n```bash\nhulu-evaluate train --model_name <MODEL_NAME> --output_dir <OUTPUT_DIR> --train_epochs <EPOCHS>\n--model_name: Name of the model to train.\n--train_epochs: Number of epochs.\n--use_lora: Enable LoRA fine-tuning (optional).\n```\n\n### Contributing\nContributions are welcome! Please submit issues or pull requests to improve hulu_evaluate.\n\n### License\nThis project is licensed under the Apache License.\n",
"bugtrack_url": null,
"license": "Apache",
"summary": "Client library to train and evaluate models on the HuLu benchmark.",
"version": "0.0.2",
"project_urls": {
"Homepage": "https://hulu.nytud.hu/"
},
"split_keywords": [
"machine-learning",
"models",
"natural-language-processing",
"deep-learning",
"evaluation",
"benchmark"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "832cd475be4ea1e41ca2e44a4367e545df709da003215898d2eced41d5b2bbd3",
"md5": "728942850dc2e1e4b92d8b93b225a7ed",
"sha256": "51487245406840e1ae95e70d0fea1bd77e704ce924039efc9f1d240237f13088"
},
"downloads": -1,
"filename": "hulu_evaluate-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "728942850dc2e1e4b92d8b93b225a7ed",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 21065,
"upload_time": "2024-12-04T13:14:52",
"upload_time_iso_8601": "2024-12-04T13:14:52.089434Z",
"url": "https://files.pythonhosted.org/packages/83/2c/d475be4ea1e41ca2e44a4367e545df709da003215898d2eced41d5b2bbd3/hulu_evaluate-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7392046c6514c588cdbcb0bc24d62b1fbc80119c8d5fffb387826ccaea57324b",
"md5": "e264aef6ac218e5e9969baefc03bfef7",
"sha256": "3e9f867b0bd7b5740603f8f1bcc8d6d61ecaa40d77f6e47c0af583b9cef55088"
},
"downloads": -1,
"filename": "hulu_evaluate-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "e264aef6ac218e5e9969baefc03bfef7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 20483,
"upload_time": "2024-12-04T13:14:56",
"upload_time_iso_8601": "2024-12-04T13:14:56.276291Z",
"url": "https://files.pythonhosted.org/packages/73/92/046c6514c588cdbcb0bc24d62b1fbc80119c8d5fffb387826ccaea57324b/hulu_evaluate-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-04 13:14:56",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "hulu-evaluate"
}
JSON