<div align="center">
English | [简体中文](./README-zh_CN.md) | [日本語](./README-ja.md)
<h1>UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition</h1>
[[ Paper ]](https://arxiv.org/abs/2404.15254) [[ Website ]](https://github.com/opendatalab/UniMERNet/tree/main) [[ Dataset (OpenDataLab)]](https://opendatalab.com/OpenDataLab/UniMER-Dataset) [[ Dataset (Hugging Face) ]](https://huggingface.co/datasets/wanderkid/UniMER_Dataset)
[[Models 🤗(Hugging Face)]](https://huggingface.co/wanderkid/unimernet_base)
[[Models <img src="./asset/images/modelscope_logo.png" width="20px">(ModelScope)]](https://www.modelscope.cn/models/wanderkid/unimernet_base)
🔥🔥 [CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation](https://github.com/opendatalab/UniMERNet/tree/main/cdm)
</div>
Welcome to the official repository of UniMERNet, a solution that converts images of mathematical expressions into LaTeX, suitable for a wide range of real-world scenarios.
## News 🚀🚀🚀
**2024.09.06** 🎉🎉 <font color="red">UniMERNet Update: The new version features a smaller model and faster inference. Training code is now open-sourced. For details, see the latest paper [UniMERNet](https://arxiv.org/abs/2404.15254).</font>
**2024.09.06** 🎉🎉 <font color="red">Introducing a new metric for formula recognition: [CDM](https://github.com/opendatalab/UniMERNet/tree/main/cdm). Compared to BLEU/EditDistance, CDM provides a more intuitive and accurate evaluation score, allowing for fair comparison of different models without being affected by formula expression diversity. </font>
**2024.07.21** 🎉🎉 Add Math Formula Detection (MFD) Tutorial based on [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) MFD model.
**2024.06.06** 🎉🎉 Open-sourced evaluation code for UniMER dataset.
**2024.05.06** 🎉🎉 Open-sourced UniMER dataset, including UniMER-1M for model training and UniMER-Test for MER evaluation.
**2024.05.06** 🎉🎉 Add Streamlit formula recognition demo and provided local deployment App.
**2024.04.24** 🎉🎉 Paper now available on [ArXiv](https://arxiv.org/abs/2404.15254).
**2024.04.24** 🎉🎉 Inference code and checkpoints have been released.
## Demo Video
https://github.com/opendatalab/UniMERNet/assets/69186975/ac54c6b9-442c-48b0-95f9-a4a3fce8780b
https://github.com/opendatalab/UniMERNet/assets/69186975/09b71c55-c58a-4792-afc1-d5774880ccf8
## Quick Start
### Clone the repo and download the model
```bash
git clone https://github.com/opendatalab/UniMERNet.git
```
```bash
cd UniMERNet/models
# Download the model and tokenizer individually or use git-lfs
git lfs install
git clone https://huggingface.co/wanderkid/unimernet_base # 1.3GB
git clone https://huggingface.co/wanderkid/unimernet_small # 773MB
git clone https://huggingface.co/wanderkid/unimernet_tiny # 441MB
# you can also download the model from ModelScope
git clone https://www.modelscope.cn/wanderkid/unimernet_base.git
git clone https://www.modelscope.cn/wanderkid/unimernet_small.git
git clone https://www.modelscope.cn/wanderkid/unimernet_tiny.git
```
### Installation
> Create a clean Conda environment
```bash
conda create -n unimernet python=3.10
conda activate unimernet
```
> Method 1: Install via pip (recommended for general users)
```bash
pip install -U "unimernet[full]"
```
> Method 2: Local installation (recommended for developers)
```bash
pip install -e ."[full]"
```
### Running UniMERNet
1. **Streamlit Application**: For an interactive and user-friendly experience, use our Streamlit-based GUI. This application allows real-time formula recognition and rendering.
```bash
unimernet_gui
```
Ensure you have the latest version of UniMERNet installed (`pip install --upgrade unimernet & pip install "unimernet[full]"`) for the streamlit GUI application.
2. **Command-line Demo**: Predict LaTeX code from an image.
```bash
python demo.py
```
3. **Jupyter Notebook Demo**: Recognize and render formula from an image.
```bash
jupyter-lab ./demo.ipynb
```
## Performance Comparison (BLEU) with SOTA Methods.
> UniMERNet significantly outperforms mainstream models in recognizing real-world mathematical expressions, demonstrating superior performance across Simple Printed Expressions (SPE), Complex Printed Expressions (CPE), Screen-Captured Expressions (SCE), and Handwritten Expressions (HWE), as evidenced by the comparative BLEU Score evaluation.
## Performance Comparison (CDM) with SOTA Methods.
> Due to the diversity of expression of formulas, it is unfair to compare different models by BLEU metric. Therefore, we conduct evaluation by CDM, a specially designed metric for formula recognition. Our method is far superior to the open source model and has the same effect as that of commercial software Mathpix. CDM@ExpRate means that the proportion of correct formulas is completely predicted. Refer to [CDM](https://arxiv.org/pdf/2409.03643) paper for details.
## Visualization Result with Different Methods.
> UniMERNet excels in visual recognition of challenging samples, outperforming other methods.
## UniMER Dataset
### Introduction
The UniMER dataset is a specialized collection curated to advance the field of Mathematical Expression Recognition (MER). It encompasses the comprehensive UniMER-1M training set, featuring over one million instances that represent a diverse and intricate range of mathematical expressions, coupled with the UniMER Test Set, meticulously designed to benchmark MER models against real-world scenarios. The dataset details are as follows:
**UniMER-1M Training Set:**
- Total Samples: 1,061,791 Latex-Image pairs
- Composition: A balanced mix of concise and complex, extended formula expressions
- Aim: To train robust, high-accuracy MER models, enhancing recognition precision and generalization
**UniMER Test Set:**
- Total Samples: 23,757, categorized into four types of expressions:
- Simple Printed Expressions (SPE): 6,762 samples
- Complex Printed Expressions (CPE): 5,921 samples
- Screen Capture Expressions (SCE): 4,742 samples
- Handwritten Expressions (HWE): 6,332 samples
- Purpose: To provide a thorough evaluation of MER models across a spectrum of real-world conditions
### Dataset Download
You can download the dataset from [OpenDataLab](https://opendatalab.com/OpenDataLab/UniMER-Dataset) (recommended for users in China) or [HuggingFace](https://huggingface.co/datasets/wanderkid/UniMER_Dataset).
### Download UniMER-Test Dataset
Download the UniMER-1M dataset and extract it to the following directory:
```bash
./data/UniMER-1M
```
Download the UniMER-Test dataset and extract it to the following directory:
```bash
./data/UniMER-Test
```
## Training
To train the UniMERNet model, follow these steps:
1. **Specify the Training Dataset Path**: Open the `configs/train` fold and set the path to your training dataset.
2. **Run the Training Script**: Execute the following command to start the training process.
```bash
bash script/train.sh
```
### Notes:
- Ensure that the dataset path specified in the `configs/train` fold is correct and accessible.
- Monitor the training process for any errors or issues.
## Testing
To test the UniMERNet model, follow these steps:
1. **Specify the Test Dataset Path**: Open the `configs/val` fold and set the path to your test dataset.
2. **Run the Test Script**: Execute the following command to start the testing process.
```bash
bash script/test.sh
```
### Notes:
- Ensure that the dataset path specified in the `configs/val` fold is correct and accessible.
- The `test.py` script will use the specified test dataset for evaluation. Remember to change the test set path in test.py to your actual path.
- Review the test results for performance metrics and any potential issues.
## Math Formula Detection Tutorial
The prerequisite for formula recognition is to detect the areas within PDF or webpage screenshots where formulas are located. The [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) includes a powerful model for detecting formulas. If you wish to perform both formula detection and recognition by yourself, you can refer to the [Formula Detection Tutorial](./MFD/README.md) for guidance on deploying and using the formula detection model.
## TODO
[✅] Release inference code and checkpoints of UniMERNet.
[✅] Release UniMER-1M and UniMER-Test.
[✅] Open-source the Streamlit formula recognition GUI application.
[✅] Release the training code for UniMERNet.
## Citation
If you find our models / code / papers useful in your research, please consider giving us a star ⭐ and citing our work 📝, thank you :)
```bibtex
@misc{wang2024unimernetuniversalnetworkrealworld,
title={UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition},
author={Bin Wang and Zhuangcheng Gu and Guang Liang and Chao Xu and Bo Zhang and Botian Shi and Conghui He},
year={2024},
eprint={2404.15254},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2404.15254},
}
@misc{wang2024cdmreliablemetricfair,
title={CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation},
author={Bin Wang and Fan Wu and Linke Ouyang and Zhuangcheng Gu and Rui Zhang and Renqiu Xia and Bo Zhang and Conghui He},
year={2024},
eprint={2409.03643},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2409.03643},
}
```
## Acknowledgements
- [VIGC](https://github.com/opendatalab/VIGC). The model framework is dependent on VIGC.
- [Texify](https://github.com/VikParuchuri/texify). A mainstream MER algorithm, UniMERNet data processing refers to Texify.
- [Latex-OCR](https://github.com/lukas-blecher/LaTeX-OCR). Another mainstream MER algorithm.
- [Donut](https://huggingface.co/naver-clova-ix/donut-base). The UniMERNet's Transformer Encoder-Decoder are referenced from Donut.
- [Nougat](https://github.com/facebookresearch/nougat). The tokenizer uses Nougat.
## Contact Us
If you have any questions, comments, or suggestions, please do not hesitate to contact us at wangbin@pjlab.org.cn.
## License
[Apache License 2.0](LICENSE)
Raw data
{
"_id": null,
"home_page": "https://github.com/opendatalab/UniMERNet",
"name": "unimernet",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "MER, latex, markdown, pdf",
"author": "Bin Wang",
"author_email": "ictwangbin@gmail.com",
"download_url": null,
"platform": null,
"description": "<div align=\"center\">\n\nEnglish | [\u7b80\u4f53\u4e2d\u6587](./README-zh_CN.md) | [\u65e5\u672c\u8a9e](./README-ja.md)\n\n\n<h1>UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition</h1>\n\n\n[[ Paper ]](https://arxiv.org/abs/2404.15254) [[ Website ]](https://github.com/opendatalab/UniMERNet/tree/main) [[ Dataset (OpenDataLab)]](https://opendatalab.com/OpenDataLab/UniMER-Dataset) [[ Dataset (Hugging Face) ]](https://huggingface.co/datasets/wanderkid/UniMER_Dataset)\n\n\n[[Models \ud83e\udd17(Hugging Face)]](https://huggingface.co/wanderkid/unimernet_base)\n[[Models <img src=\"./asset/images/modelscope_logo.png\" width=\"20px\">(ModelScope)]](https://www.modelscope.cn/models/wanderkid/unimernet_base)\n\n\ud83d\udd25\ud83d\udd25 [CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation](https://github.com/opendatalab/UniMERNet/tree/main/cdm)\n\n</div>\n\nWelcome to the official repository of UniMERNet, a solution that converts images of mathematical expressions into LaTeX, suitable for a wide range of real-world scenarios.\n\n## News \ud83d\ude80\ud83d\ude80\ud83d\ude80\n**2024.09.06** \ud83c\udf89\ud83c\udf89 <font color=\"red\">UniMERNet Update: The new version features a smaller model and faster inference. Training code is now open-sourced. For details, see the latest paper [UniMERNet](https://arxiv.org/abs/2404.15254).</font> \n**2024.09.06** \ud83c\udf89\ud83c\udf89 <font color=\"red\">Introducing a new metric for formula recognition: [CDM](https://github.com/opendatalab/UniMERNet/tree/main/cdm). Compared to BLEU/EditDistance, CDM provides a more intuitive and accurate evaluation score, allowing for fair comparison of different models without being affected by formula expression diversity. </font> \n**2024.07.21** \ud83c\udf89\ud83c\udf89 Add Math Formula Detection (MFD) Tutorial based on [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) MFD model. \n**2024.06.06** \ud83c\udf89\ud83c\udf89 Open-sourced evaluation code for UniMER dataset. \n**2024.05.06** \ud83c\udf89\ud83c\udf89 Open-sourced UniMER dataset, including UniMER-1M for model training and UniMER-Test for MER evaluation. \n**2024.05.06** \ud83c\udf89\ud83c\udf89 Add Streamlit formula recognition demo and provided local deployment App. \n**2024.04.24** \ud83c\udf89\ud83c\udf89 Paper now available on [ArXiv](https://arxiv.org/abs/2404.15254). \n**2024.04.24** \ud83c\udf89\ud83c\udf89 Inference code and checkpoints have been released. \n\n\n## Demo Video\nhttps://github.com/opendatalab/UniMERNet/assets/69186975/ac54c6b9-442c-48b0-95f9-a4a3fce8780b\n\n\nhttps://github.com/opendatalab/UniMERNet/assets/69186975/09b71c55-c58a-4792-afc1-d5774880ccf8\n\n## Quick Start\n\n### Clone the repo and download the model\n```bash\ngit clone https://github.com/opendatalab/UniMERNet.git\n```\n\n```bash\ncd UniMERNet/models\n# Download the model and tokenizer individually or use git-lfs\ngit lfs install\ngit clone https://huggingface.co/wanderkid/unimernet_base # 1.3GB \ngit clone https://huggingface.co/wanderkid/unimernet_small # 773MB \ngit clone https://huggingface.co/wanderkid/unimernet_tiny # 441MB \n\n# you can also download the model from ModelScope\ngit clone https://www.modelscope.cn/wanderkid/unimernet_base.git\ngit clone https://www.modelscope.cn/wanderkid/unimernet_small.git\ngit clone https://www.modelscope.cn/wanderkid/unimernet_tiny.git\n\n```\n\n### Installation\n\n> Create a clean Conda environment\n\n```bash\nconda create -n unimernet python=3.10\nconda activate unimernet\n```\n\n> Method 1: Install via pip (recommended for general users)\n\n```bash\npip install -U \"unimernet[full]\"\n```\n\n> Method 2: Local installation (recommended for developers)\n\n```bash\npip install -e .\"[full]\"\n```\n\n\n\n\n\n### Running UniMERNet\n\n1. **Streamlit Application**: For an interactive and user-friendly experience, use our Streamlit-based GUI. This application allows real-time formula recognition and rendering.\n\n ```bash\n unimernet_gui\n ```\n Ensure you have the latest version of UniMERNet installed (`pip install --upgrade unimernet & pip install \"unimernet[full]\"`) for the streamlit GUI application.\n\n2. **Command-line Demo**: Predict LaTeX code from an image.\n\n ```bash\n python demo.py\n ```\n\n3. **Jupyter Notebook Demo**: Recognize and render formula from an image.\n\n ```bash\n jupyter-lab ./demo.ipynb\n ```\n\n\n## Performance Comparison (BLEU) with SOTA Methods.\n\n> UniMERNet significantly outperforms mainstream models in recognizing real-world mathematical expressions, demonstrating superior performance across Simple Printed Expressions (SPE), Complex Printed Expressions (CPE), Screen-Captured Expressions (SCE), and Handwritten Expressions (HWE), as evidenced by the comparative BLEU Score evaluation. \n\n\n\n## Performance Comparison (CDM) with SOTA Methods.\n\n> Due to the diversity of expression of formulas, it is unfair to compare different models by BLEU metric. Therefore, we conduct evaluation by CDM, a specially designed metric for formula recognition. Our method is far superior to the open source model and has the same effect as that of commercial software Mathpix. CDM@ExpRate means that the proportion of correct formulas is completely predicted. Refer to [CDM](https://arxiv.org/pdf/2409.03643) paper for details.\n\n\n\n## Visualization Result with Different Methods.\n\n> UniMERNet excels in visual recognition of challenging samples, outperforming other methods. \n\n\n\n## UniMER Dataset\n### Introduction\nThe UniMER dataset is a specialized collection curated to advance the field of Mathematical Expression Recognition (MER). It encompasses the comprehensive UniMER-1M training set, featuring over one million instances that represent a diverse and intricate range of mathematical expressions, coupled with the UniMER Test Set, meticulously designed to benchmark MER models against real-world scenarios. The dataset details are as follows:\n\n**UniMER-1M Training Set:**\n - Total Samples: 1,061,791 Latex-Image pairs\n - Composition: A balanced mix of concise and complex, extended formula expressions\n - Aim: To train robust, high-accuracy MER models, enhancing recognition precision and generalization\n\n**UniMER Test Set:**\n - Total Samples: 23,757, categorized into four types of expressions:\n - Simple Printed Expressions (SPE): 6,762 samples\n - Complex Printed Expressions (CPE): 5,921 samples\n - Screen Capture Expressions (SCE): 4,742 samples\n - Handwritten Expressions (HWE): 6,332 samples\n - Purpose: To provide a thorough evaluation of MER models across a spectrum of real-world conditions\n\n### Dataset Download\nYou can download the dataset from [OpenDataLab](https://opendatalab.com/OpenDataLab/UniMER-Dataset) (recommended for users in China) or [HuggingFace](https://huggingface.co/datasets/wanderkid/UniMER_Dataset).\n\n### Download UniMER-Test Dataset\n\n\nDownload the UniMER-1M dataset and extract it to the following directory:\n```bash\n./data/UniMER-1M\n```\n\nDownload the UniMER-Test dataset and extract it to the following directory:\n```bash\n./data/UniMER-Test\n```\n\n## Training\n\nTo train the UniMERNet model, follow these steps:\n\n1. **Specify the Training Dataset Path**: Open the `configs/train` fold and set the path to your training dataset.\n\n2. **Run the Training Script**: Execute the following command to start the training process.\n\n ```bash\n bash script/train.sh\n ```\n\n### Notes:\n- Ensure that the dataset path specified in the `configs/train` fold is correct and accessible.\n- Monitor the training process for any errors or issues.\n\n## Testing\n\nTo test the UniMERNet model, follow these steps:\n\n1. **Specify the Test Dataset Path**: Open the `configs/val` fold and set the path to your test dataset.\n\n2. **Run the Test Script**: Execute the following command to start the testing process.\n\n ```bash\n bash script/test.sh\n ```\n\n### Notes:\n- Ensure that the dataset path specified in the `configs/val` fold is correct and accessible.\n- The `test.py` script will use the specified test dataset for evaluation. Remember to change the test set path in test.py to your actual path.\n- Review the test results for performance metrics and any potential issues.\n\n## Math Formula Detection Tutorial\n\nThe prerequisite for formula recognition is to detect the areas within PDF or webpage screenshots where formulas are located. The [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit) includes a powerful model for detecting formulas. If you wish to perform both formula detection and recognition by yourself, you can refer to the [Formula Detection Tutorial](./MFD/README.md) for guidance on deploying and using the formula detection model.\n\n\n## TODO\n\n[\u2705] Release inference code and checkpoints of UniMERNet. \n[\u2705] Release UniMER-1M and UniMER-Test. \n[\u2705] Open-source the Streamlit formula recognition GUI application. \n[\u2705] Release the training code for UniMERNet. \n\n## Citation\nIf you find our models / code / papers useful in your research, please consider giving us a star \u2b50 and citing our work \ud83d\udcdd, thank you :)\n```bibtex\n@misc{wang2024unimernetuniversalnetworkrealworld,\n title={UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition}, \n author={Bin Wang and Zhuangcheng Gu and Guang Liang and Chao Xu and Bo Zhang and Botian Shi and Conghui He},\n year={2024},\n eprint={2404.15254},\n archivePrefix={arXiv},\n primaryClass={cs.CV},\n url={https://arxiv.org/abs/2404.15254}, \n}\n\n@misc{wang2024cdmreliablemetricfair,\n title={CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation}, \n author={Bin Wang and Fan Wu and Linke Ouyang and Zhuangcheng Gu and Rui Zhang and Renqiu Xia and Bo Zhang and Conghui He},\n year={2024},\n eprint={2409.03643},\n archivePrefix={arXiv},\n primaryClass={cs.CV},\n url={https://arxiv.org/abs/2409.03643}, \n}\n```\n\n## Acknowledgements\n- [VIGC](https://github.com/opendatalab/VIGC). The model framework is dependent on VIGC.\n- [Texify](https://github.com/VikParuchuri/texify). A mainstream MER algorithm, UniMERNet data processing refers to Texify.\n- [Latex-OCR](https://github.com/lukas-blecher/LaTeX-OCR). Another mainstream MER algorithm.\n- [Donut](https://huggingface.co/naver-clova-ix/donut-base). The UniMERNet's Transformer Encoder-Decoder are referenced from Donut.\n- [Nougat](https://github.com/facebookresearch/nougat). The tokenizer uses Nougat.\n\n## Contact Us\nIf you have any questions, comments, or suggestions, please do not hesitate to contact us at wangbin@pjlab.org.cn.\n\n## License\n[Apache License 2.0](LICENSE)\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/opendatalab/UniMERNet",
"Repository": "https://github.com/opendatalab/UniMERNet"
},
"split_keywords": [
"mer",
" latex",
" markdown",
" pdf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "31407b604dd088fe7b49081729e5136538a145e33a949cc35f5602b497961cd0",
"md5": "a5e5f0218984fddcbb3204b651f6162a",
"sha256": "538ebb3313bbb44f20778326696c8ba4dc6ba2dec28cc8baf225b44ae429f0ad"
},
"downloads": -1,
"filename": "unimernet-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a5e5f0218984fddcbb3204b651f6162a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 2260247,
"upload_time": "2024-12-26T06:09:04",
"upload_time_iso_8601": "2024-12-26T06:09:04.011697Z",
"url": "https://files.pythonhosted.org/packages/31/40/7b604dd088fe7b49081729e5136538a145e33a949cc35f5602b497961cd0/unimernet-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-26 06:09:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "opendatalab",
"github_project": "UniMERNet",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "unimernet"
}
JSON
