# gpt-resolve
Can GPT solve Brazilian university entrance exams?
This project is an implementation of how to use LLMs to solve challenging Brazilian university entrance exams.
We'll use `o1-preview`, which is the best OpenAI model so far with reasoning capabilities, and `gpt-4o` to describe the exam images so that `o1-preview` can solve them on question at a time (as it does not have image capabilities yet). Results are saved as txt files with LaTeX formatting, and you can optionally convert them to a nice PDF with this package or using some LaTeX editor.
The project begins with the ITA (Instituto Tecnológico de Aeronáutica) 2025 exam, focusing first on the Math essay section. This section, from the recent exam on November 5, 2024, demands deep subject understanding and step-by-step solutions. More details are in the [report](exams/ita_2025/report.md) documentation.
**Spoiler: o1-preview scored 90% in the ITA 2025 Math Essay Exam, 95% in Chemistry Essay and only 65% in Physics Essay.**
After the first ITA 2025 exam is fully solved, the project will try to expand to other sections and eventually other exams. Feel free to contribute with ideas and implementations of other exams!
Table of some exams to be solved:
| Exam | Year | Model | Status | Score | Report |
|------|------|-------|--------|-------|--------|
| ITA | 2025 | o1-preview | 🚧 In Progress | - | [Report](exams/ita_2025/report.md) |
| IME | 2025 | o1-preview | 📅 Todo | - | - |
| Escola Naval | 2025 | o1-preview | 📅 Todo | - | - |
| Fuvest/USP | 2025 | o1-preview | 📅 Todo | - | - |
| AFA | 2025 | o1-preview | 📅 Todo | - | - |
| UNICAMP | 2025 | o1-preview | 📅 Todo | - | - |
# Installation and How to use
gpt-resolve is distributed in pypi:
```bash
pip install gpt-resolve
```
`gpt-resolve` provides a simple CLI with two main commands: `resolve` for solving exam questions and `compile-solutions` for generating PDFs from the solutions:
### Solve exams with `resolve`
To generate solutions for an exam:
- save the exam images in the exam folder `exam_path`, one question per image file. File names should follow the pattern `q<question_number>.jpg`, e.g. `q1.jpg`, `q2.jpg`, etc.
- add `OPENAI_API_KEY` to your global environment variables or to a `.env` file in the current directory
then, run
```bash
gpt-resolve resolve -p exam_path
```
and grab a coffee while it runs.
If you want to test the process without making real API calls, you can use the `--dry-run` flag. See `gpt-resolve resolve --help` for more details about solving only a subset of questions or controlling token usage.
### Compile solutions with `compile-solutions`
Once you have the solutions in your exam folder `exam_path`, you can compile them into a single PDF running:
```bash
gpt-resolve compile-solutions -p exam_path --title "Your Exam Title"
```
For that command to work, you'll need a LaTeX distribution in your system. See some guidelines [here](https://www.tug.org/texlive/) (MacTeX for MacOS was used to start this project).
# Troubleshooting
Sometimes, it was observed that the output from `o1-preview` produced invalid LaTeX code when nesting display math environments (such as `\[...\]` and `\begin{align*} ... \end{align*}` together). The current prompt for `o1-preview` adds an instruction to avoid this, which works most of the time. If that happens, you can try to solve the question again by running `gpt-resolve resolve -p exam_path -q <question_number>`, or making more adjustments to the prompt, or fixing the output LaTeX code manually.
# Costs
The `o1-preview` model is so far [available only for Tiers 3, 4 and 5](https://help.openai.com/en/articles/9824962-openai-o1-preview-and-o1-mini-usage-limits-on-chatgpt-and-the-api). It is [6x more expensive](https://openai.com/api/pricing/) than `gpt-4o`, and also consumes much more tokens to "reason" (see more [here](https://platform.openai.com/docs/guides/reasoning/controlling-costs#controlling-costs)), so be mindful about the number of questions you are solving and how many max tokens you're allowing gpt-resolve to use (see `gpt-resolve resolve --help` to control `max-tokens-question-answer`, which drives the cost). You can roughly estimate an upper bound for costs of solving an exam by
```
(number of questions) * (max_tokens_question_answer / 1_000_000) * (price per 1M tokens)
```
For the current price for o1-preview of $15/$60 per 1M tokens for input/output tokens, an 10 question exam with 10000 max tokens per question would cost less than $6.
# Contributing
There are several ways you can contribute to this project:
- Add solutions for new exams or sections
- Improve existing solutions or model prompts
- Add automatic evaluation metrics for answers
- Create documentations for exams
- Report issues or suggest improvements
To contribute, simply fork the repository, create a new branch for your changes, and submit a pull request. Please ensure your PR includes:
- Clear description of the changes
- Updated table entry (if adding new exam solutions)
- Any relevant documentation
Feel free to open an issue first to discuss major changes or new ideas!
## Sponsors
<p align="center">
<a href="https://www.buser.com.br">
<img src="assets/sponsors/buser-logo.png" alt="Buser Logo" width="100"/>
</a>
</p>
This project is proudly sponsored by [Buser](https://www.buser.com.br), Brazil's leading bus travel platform.
Raw data
{
"_id": null,
"home_page": "https://github.com/lgabs/gpt-resolve",
"name": "gpt-resolve",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "gpt, education, exam-solving, brazil",
"author": "Luan Fernandes",
"author_email": "luangabriel70@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d4/81/dd9aad6b2adce7a0676fae708d62e1828c67d7bc366f63c32bcf020f83a5/gpt_resolve-0.2.2.tar.gz",
"platform": null,
"description": "# gpt-resolve\nCan GPT solve Brazilian university entrance exams?\n\nThis project is an implementation of how to use LLMs to solve challenging Brazilian university entrance exams.\n\nWe'll use `o1-preview`, which is the best OpenAI model so far with reasoning capabilities, and `gpt-4o` to describe the exam images so that `o1-preview` can solve them on question at a time (as it does not have image capabilities yet). Results are saved as txt files with LaTeX formatting, and you can optionally convert them to a nice PDF with this package or using some LaTeX editor.\n\nThe project begins with the ITA (Instituto Tecnol\u00f3gico de Aeron\u00e1utica) 2025 exam, focusing first on the Math essay section. This section, from the recent exam on November 5, 2024, demands deep subject understanding and step-by-step solutions. More details are in the [report](exams/ita_2025/report.md) documentation. \n\n**Spoiler: o1-preview scored 90% in the ITA 2025 Math Essay Exam, 95% in Chemistry Essay and only 65% in Physics Essay.**\n\nAfter the first ITA 2025 exam is fully solved, the project will try to expand to other sections and eventually other exams. Feel free to contribute with ideas and implementations of other exams! \n\nTable of some exams to be solved:\n\n| Exam | Year | Model | Status | Score | Report |\n|------|------|-------|--------|-------|--------|\n| ITA | 2025 | o1-preview | \ud83d\udea7 In Progress | - | [Report](exams/ita_2025/report.md) |\n| IME | 2025 | o1-preview | \ud83d\udcc5 Todo | - | - |\n| Escola Naval | 2025 | o1-preview | \ud83d\udcc5 Todo | - | - |\n| Fuvest/USP | 2025 | o1-preview | \ud83d\udcc5 Todo | - | - |\n| AFA | 2025 | o1-preview | \ud83d\udcc5 Todo | - | - |\n| UNICAMP | 2025 | o1-preview | \ud83d\udcc5 Todo | - | - |\n\n# Installation and How to use\ngpt-resolve is distributed in pypi:\n```bash\npip install gpt-resolve\n```\n\n`gpt-resolve` provides a simple CLI with two main commands: `resolve` for solving exam questions and `compile-solutions` for generating PDFs from the solutions:\n\n### Solve exams with `resolve`\n\nTo generate solutions for an exam:\n- save the exam images in the exam folder `exam_path`, one question per image file. File names should follow the pattern `q<question_number>.jpg`, e.g. `q1.jpg`, `q2.jpg`, etc.\n- add `OPENAI_API_KEY` to your global environment variables or to a `.env` file in the current directory\n\nthen, run\n```bash\ngpt-resolve resolve -p exam_path\n```\nand grab a coffee while it runs.\n\nIf you want to test the process without making real API calls, you can use the `--dry-run` flag. See `gpt-resolve resolve --help` for more details about solving only a subset of questions or controlling token usage.\n\n\n### Compile solutions with `compile-solutions`\n\nOnce you have the solutions in your exam folder `exam_path`, you can compile them into a single PDF running:\n```bash\ngpt-resolve compile-solutions -p exam_path --title \"Your Exam Title\"\n```\n\nFor that command to work, you'll need a LaTeX distribution in your system. See some guidelines [here](https://www.tug.org/texlive/) (MacTeX for MacOS was used to start this project).\n\n# Troubleshooting\n\nSometimes, it was observed that the output from `o1-preview` produced invalid LaTeX code when nesting display math environments (such as `\\[...\\]` and `\\begin{align*} ... \\end{align*}` together). The current prompt for `o1-preview` adds an instruction to avoid this, which works most of the time. If that happens, you can try to solve the question again by running `gpt-resolve resolve -p exam_path -q <question_number>`, or making more adjustments to the prompt, or fixing the output LaTeX code manually.\n\n# Costs\n\nThe `o1-preview` model is so far [available only for Tiers 3, 4 and 5](https://help.openai.com/en/articles/9824962-openai-o1-preview-and-o1-mini-usage-limits-on-chatgpt-and-the-api). It is [6x more expensive](https://openai.com/api/pricing/) than `gpt-4o`, and also consumes much more tokens to \"reason\" (see more [here](https://platform.openai.com/docs/guides/reasoning/controlling-costs#controlling-costs)), so be mindful about the number of questions you are solving and how many max tokens you're allowing gpt-resolve to use (see `gpt-resolve resolve --help` to control `max-tokens-question-answer`, which drives the cost). You can roughly estimate an upper bound for costs of solving an exam by \n```\n(number of questions) * (max_tokens_question_answer / 1_000_000) * (price per 1M tokens)\n```\nFor the current price for o1-preview of $15/$60 per 1M tokens for input/output tokens, an 10 question exam with 10000 max tokens per question would cost less than $6.\n\n# Contributing\n\nThere are several ways you can contribute to this project:\n\n- Add solutions for new exams or sections\n- Improve existing solutions or model prompts\n- Add automatic evaluation metrics for answers\n- Create documentations for exams\n- Report issues or suggest improvements\n\nTo contribute, simply fork the repository, create a new branch for your changes, and submit a pull request. Please ensure your PR includes:\n- Clear description of the changes\n- Updated table entry (if adding new exam solutions)\n- Any relevant documentation\n\nFeel free to open an issue first to discuss major changes or new ideas!\n\n## Sponsors\n\n<p align=\"center\">\n <a href=\"https://www.buser.com.br\">\n <img src=\"assets/sponsors/buser-logo.png\" alt=\"Buser Logo\" width=\"100\"/>\n </a>\n</p>\n\nThis project is proudly sponsored by [Buser](https://www.buser.com.br), Brazil's leading bus travel platform.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "GPT solutions for Brazilian university entrance exams.",
"version": "0.2.2",
"project_urls": {
"Homepage": "https://github.com/lgabs/gpt-resolve",
"Repository": "https://github.com/lgabs/gpt-resolve"
},
"split_keywords": [
"gpt",
" education",
" exam-solving",
" brazil"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8a59914119060677877f4fda8f502f4d83ad6d1af351cc7db0eebce6acb1ddd0",
"md5": "c8013e3cc767b22528af782719ffd27d",
"sha256": "80cf62ae3dbd65b8a2f0026311d98a2632b8155ae231541d2fba5f03e03ce4b7"
},
"downloads": -1,
"filename": "gpt_resolve-0.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c8013e3cc767b22528af782719ffd27d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 9925,
"upload_time": "2024-11-17T15:38:20",
"upload_time_iso_8601": "2024-11-17T15:38:20.581071Z",
"url": "https://files.pythonhosted.org/packages/8a/59/914119060677877f4fda8f502f4d83ad6d1af351cc7db0eebce6acb1ddd0/gpt_resolve-0.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d481dd9aad6b2adce7a0676fae708d62e1828c67d7bc366f63c32bcf020f83a5",
"md5": "64122ae76bcb50a38417b03c5916ab16",
"sha256": "d8f2b60f79ebbe52f63eabc0351c5d61eb3080ed5f620ed115f65c1309b90580"
},
"downloads": -1,
"filename": "gpt_resolve-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "64122ae76bcb50a38417b03c5916ab16",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 8584,
"upload_time": "2024-11-17T15:38:23",
"upload_time_iso_8601": "2024-11-17T15:38:23.284942Z",
"url": "https://files.pythonhosted.org/packages/d4/81/dd9aad6b2adce7a0676fae708d62e1828c67d7bc366f63c32bcf020f83a5/gpt_resolve-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-17 15:38:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lgabs",
"github_project": "gpt-resolve",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "gpt-resolve"
}
JSON
