| Name | swebench JSON |
| Version |
2.1.3
JSON |
| download |
| home_page | https://swebench.com |
| Summary | The official SWE-bench package - a benchmark for evaluating LMs on software engineering |
| upload_time | 2024-11-17 09:05:42 |
| maintainer | None |
| docs_url | None |
| author | John Yang |
| requires_python | >=3.8 |
| license | None |
| keywords |
nlp
benchmark
code
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<a href="https://github.com/princeton-nlp/Llamao">
<img src="assets/figures/swellama_banner.png" width="50%" alt="Kawi the SWE-Llama" />
</a>
</p>
<div align="center">
| [日本語](docs/README_JP.md) | [English](https://github.com/princeton-nlp/SWE-bench) | [中文简体](docs/README_CN.md) | [中文繁體](docs/README_TW.md) |
</div>
---
<p align="center">
Code and data for our ICLR 2024 paper <a href="http://swe-bench.github.io/paper.pdf">SWE-bench: Can Language Models Resolve Real-World GitHub Issues?</a>
</br>
</br>
<a href="https://www.python.org/">
<img alt="Build" src="https://img.shields.io/badge/Python-3.8+-1f425f.svg?color=purple">
</a>
<a href="https://copyright.princeton.edu/policy">
<img alt="License" src="https://img.shields.io/badge/License-MIT-blue">
</a>
<a href="https://badge.fury.io/py/swebench">
<img src="https://badge.fury.io/py/swebench.svg">
</a>
</p>
Please refer our [website](http://swe-bench.github.io) for the public leaderboard and the [change log](https://github.com/princeton-nlp/SWE-bench/blob/main/CHANGELOG.md) for information on the latest updates to the SWE-bench benchmark.
## 📰 News
* **[Aug. 13, 2024]**: Introducing *SWE-bench Verified*! Part 2 of our collaboration with [OpenAI Preparedness](https://openai.com/preparedness/). A subset of 500 problems that real software engineers have confirmed are solvable. Check out more in the [report](https://openai.com/index/introducing-swe-bench-verified/)!
* **[Jun. 27, 2024]**: We have an exciting update for SWE-bench - with support from [OpenAI's Preparedness](https://openai.com/preparedness/) team: We're moving to a fully containerized evaluation harness using Docker for more reproducible evaluations! Read more in our [report](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md).
* **[Apr. 15, 2024]**: SWE-bench has gone through major improvements to resolve issues with the evaluation harness. Read more in our [report](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240415_eval_bug/README.md).
* **[Apr. 2, 2024]**: We have released [SWE-agent](https://github.com/princeton-nlp/SWE-agent), which sets the state-of-the-art on the full SWE-bench test set! ([Tweet 🔗](https://twitter.com/jyangballin/status/1775114444370051582))
* **[Jan. 16, 2024]**: SWE-bench has been accepted to ICLR 2024 as an oral presentation! ([OpenReview 🔗](https://openreview.net/forum?id=VTF8yNQM66))
## 👋 Overview
SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub.
Given a *codebase* and an *issue*, a language model is tasked with generating a *patch* that resolves the described problem.
<img src="assets/figures/teaser.png">
To access SWE-bench, copy and run the following code:
```python
from datasets import load_dataset
swebench = load_dataset('princeton-nlp/SWE-bench', split='test')
```
## 🚀 Set Up
SWE-bench uses Docker for reproducible evaluations.
Follow the instructions in the [Docker setup guide](https://docs.docker.com/engine/install/) to install Docker on your machine.
If you're setting up on Linux, we recommend seeing the [post-installation steps](https://docs.docker.com/engine/install/linux-postinstall/) as well.
Finally, to build SWE-bench from source, follow these steps:
```bash
git clone git@github.com:princeton-nlp/SWE-bench.git
cd SWE-bench
pip install -e .
```
Test your installation by running:
```bash
python -m swebench.harness.run_evaluation \
--predictions_path gold \
--max_workers 1 \
--instance_ids sympy__sympy-20590 \
--run_id validate-gold
```
## 💽 Usage
> [!WARNING]
> Running fast evaluations on SWE-bench can be resource intensive
> We recommend running the evaluation harness on an `x86_64` machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores.
> You may need to experiment with the `--max_workers` argument to find the optimal number of workers for your machine, but we recommend using fewer than `min(0.75 * os.cpu_count(), 24)`.
>
> If running with docker desktop, make sure to increase your virtual disk space to have ~120 free GB available, and set max_workers to be consistent with the above for the CPUs available to docker.
>
> Support for `arm64` machines is experimental.
Evaluate model predictions on SWE-bench Lite using the evaluation harness with the following command:
```bash
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Lite \
--predictions_path <path_to_predictions> \
--max_workers <num_workers> \
--run_id <run_id>
# use --predictions_path 'gold' to verify the gold patches
# use --run_id to name the evaluation run
```
This command will generate docker build logs (`logs/build_images`) and evaluation logs (`logs/run_evaluation`) in the current directory.
The final evaluation results will be stored in the `evaluation_results` directory.
To see the full list of arguments for the evaluation harness, run:
```bash
python -m swebench.harness.run_evaluation --help
```
Additionally, the SWE-Bench repo can help you:
* Train your own models on our pre-processed datasets
* Run [inference](https://github.com/princeton-nlp/SWE-bench/blob/main/swebench/inference/README.md) on existing models (either models you have on-disk like LLaMA, or models you have access to through an API like GPT-4). The inference step is where you get a repo and an issue and have the model try to generate a fix for it.
* Run SWE-bench's [data collection procedure](https://github.com/princeton-nlp/SWE-bench/blob/main/swebench/collect/) on your own repositories, to make new SWE-Bench tasks.
## ⬇️ Downloads
| Datasets | Models |
| - | - |
| [🤗 SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) | [🦙 SWE-Llama 13b](https://huggingface.co/princeton-nlp/SWE-Llama-13b) |
| [🤗 "Oracle" Retrieval](https://huggingface.co/datasets/princeton-nlp/SWE-bench_oracle) | [🦙 SWE-Llama 13b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-13b-peft) |
| [🤗 BM25 Retrieval 13K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_13K) | [🦙 SWE-Llama 7b](https://huggingface.co/princeton-nlp/SWE-Llama-7b) |
| [🤗 BM25 Retrieval 27K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_27K) | [🦙 SWE-Llama 7b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-7b-peft) |
| [🤗 BM25 Retrieval 40K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_40K) | |
| [🤗 BM25 Retrieval 50K (Llama tokens)](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_50k_llama) | |
## 🍎 Tutorials
We've also written the following blog posts on how to use different parts of SWE-bench.
If you'd like to see a post about a particular topic, please let us know via an issue.
* [Nov 1. 2023] Collecting Evaluation Tasks for SWE-Bench ([🔗](https://github.com/princeton-nlp/SWE-bench/blob/main/assets/collection.md))
* [Nov 6. 2023] Evaluating on SWE-bench ([🔗](https://github.com/princeton-nlp/SWE-bench/blob/main/assets/evaluation.md))
## 💫 Contributions
We would love to hear from the broader NLP, Machine Learning, and Software Engineering research communities, and we welcome any contributions, pull requests, or issues!
To do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!
Contact person: [Carlos E. Jimenez](http://www.carlosejimenez.com/) and [John Yang](https://john-b-yang.github.io/) (Email: carlosej@princeton.edu, johnby@stanford.edu).
## ✍️ Citation
If you find our work helpful, please use the following citations.
```
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}
```
## 🪪 License
MIT. Check `LICENSE.md`.
Raw data
{
"_id": null,
"home_page": "https://swebench.com",
"name": "swebench",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "nlp, benchmark, code",
"author": "John Yang",
"author_email": "byjohnyang@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/04/91/82ffe8db72768fd6380f534e8accde63b504c739b6a28b616cd23ca0243e/swebench-2.1.3.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <a href=\"https://github.com/princeton-nlp/Llamao\">\n <img src=\"assets/figures/swellama_banner.png\" width=\"50%\" alt=\"Kawi the SWE-Llama\" />\n </a>\n</p>\n\n<div align=\"center\">\n\n | [\u65e5\u672c\u8a9e](docs/README_JP.md) | [English](https://github.com/princeton-nlp/SWE-bench) | [\u4e2d\u6587\u7b80\u4f53](docs/README_CN.md) | [\u4e2d\u6587\u7e41\u9ad4](docs/README_TW.md) |\n\n</div>\n\n\n---\n<p align=\"center\">\nCode and data for our ICLR 2024 paper <a href=\"http://swe-bench.github.io/paper.pdf\">SWE-bench: Can Language Models Resolve Real-World GitHub Issues?</a>\n </br>\n </br>\n <a href=\"https://www.python.org/\">\n <img alt=\"Build\" src=\"https://img.shields.io/badge/Python-3.8+-1f425f.svg?color=purple\">\n </a>\n <a href=\"https://copyright.princeton.edu/policy\">\n <img alt=\"License\" src=\"https://img.shields.io/badge/License-MIT-blue\">\n </a>\n <a href=\"https://badge.fury.io/py/swebench\">\n <img src=\"https://badge.fury.io/py/swebench.svg\">\n </a>\n</p>\n\nPlease refer our [website](http://swe-bench.github.io) for the public leaderboard and the [change log](https://github.com/princeton-nlp/SWE-bench/blob/main/CHANGELOG.md) for information on the latest updates to the SWE-bench benchmark.\n\n## \ud83d\udcf0 News\n* **[Aug. 13, 2024]**: Introducing *SWE-bench Verified*! Part 2 of our collaboration with [OpenAI Preparedness](https://openai.com/preparedness/). A subset of 500 problems that real software engineers have confirmed are solvable. Check out more in the [report](https://openai.com/index/introducing-swe-bench-verified/)!\n* **[Jun. 27, 2024]**: We have an exciting update for SWE-bench - with support from [OpenAI's Preparedness](https://openai.com/preparedness/) team: We're moving to a fully containerized evaluation harness using Docker for more reproducible evaluations! Read more in our [report](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md).\n* **[Apr. 15, 2024]**: SWE-bench has gone through major improvements to resolve issues with the evaluation harness. Read more in our [report](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240415_eval_bug/README.md).\n* **[Apr. 2, 2024]**: We have released [SWE-agent](https://github.com/princeton-nlp/SWE-agent), which sets the state-of-the-art on the full SWE-bench test set! ([Tweet \ud83d\udd17](https://twitter.com/jyangballin/status/1775114444370051582))\n* **[Jan. 16, 2024]**: SWE-bench has been accepted to ICLR 2024 as an oral presentation! ([OpenReview \ud83d\udd17](https://openreview.net/forum?id=VTF8yNQM66))\n\n## \ud83d\udc4b Overview\nSWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub.\nGiven a *codebase* and an *issue*, a language model is tasked with generating a *patch* that resolves the described problem.\n\n<img src=\"assets/figures/teaser.png\">\n\nTo access SWE-bench, copy and run the following code:\n```python\nfrom datasets import load_dataset\nswebench = load_dataset('princeton-nlp/SWE-bench', split='test')\n```\n\n## \ud83d\ude80 Set Up\nSWE-bench uses Docker for reproducible evaluations.\nFollow the instructions in the [Docker setup guide](https://docs.docker.com/engine/install/) to install Docker on your machine.\nIf you're setting up on Linux, we recommend seeing the [post-installation steps](https://docs.docker.com/engine/install/linux-postinstall/) as well.\n\nFinally, to build SWE-bench from source, follow these steps:\n```bash\ngit clone git@github.com:princeton-nlp/SWE-bench.git\ncd SWE-bench\npip install -e .\n```\n\nTest your installation by running:\n```bash\npython -m swebench.harness.run_evaluation \\\n --predictions_path gold \\\n --max_workers 1 \\\n --instance_ids sympy__sympy-20590 \\\n --run_id validate-gold\n```\n\n## \ud83d\udcbd Usage\n> [!WARNING]\n> Running fast evaluations on SWE-bench can be resource intensive\n> We recommend running the evaluation harness on an `x86_64` machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores.\n> You may need to experiment with the `--max_workers` argument to find the optimal number of workers for your machine, but we recommend using fewer than `min(0.75 * os.cpu_count(), 24)`.\n>\n> If running with docker desktop, make sure to increase your virtual disk space to have ~120 free GB available, and set max_workers to be consistent with the above for the CPUs available to docker.\n>\n> Support for `arm64` machines is experimental.\n\nEvaluate model predictions on SWE-bench Lite using the evaluation harness with the following command:\n```bash\npython -m swebench.harness.run_evaluation \\\n --dataset_name princeton-nlp/SWE-bench_Lite \\\n --predictions_path <path_to_predictions> \\\n --max_workers <num_workers> \\\n --run_id <run_id>\n # use --predictions_path 'gold' to verify the gold patches\n # use --run_id to name the evaluation run\n```\n\nThis command will generate docker build logs (`logs/build_images`) and evaluation logs (`logs/run_evaluation`) in the current directory.\n\nThe final evaluation results will be stored in the `evaluation_results` directory.\n\nTo see the full list of arguments for the evaluation harness, run:\n```bash\npython -m swebench.harness.run_evaluation --help\n```\n\nAdditionally, the SWE-Bench repo can help you:\n* Train your own models on our pre-processed datasets\n* Run [inference](https://github.com/princeton-nlp/SWE-bench/blob/main/swebench/inference/README.md) on existing models (either models you have on-disk like LLaMA, or models you have access to through an API like GPT-4). The inference step is where you get a repo and an issue and have the model try to generate a fix for it.\n* Run SWE-bench's [data collection procedure](https://github.com/princeton-nlp/SWE-bench/blob/main/swebench/collect/) on your own repositories, to make new SWE-Bench tasks.\n\n## \u2b07\ufe0f Downloads\n| Datasets | Models |\n| - | - |\n| [\ud83e\udd17 SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) | [\ud83e\udd99 SWE-Llama 13b](https://huggingface.co/princeton-nlp/SWE-Llama-13b) |\n| [\ud83e\udd17 \"Oracle\" Retrieval](https://huggingface.co/datasets/princeton-nlp/SWE-bench_oracle) | [\ud83e\udd99 SWE-Llama 13b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-13b-peft) |\n| [\ud83e\udd17 BM25 Retrieval 13K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_13K) | [\ud83e\udd99 SWE-Llama 7b](https://huggingface.co/princeton-nlp/SWE-Llama-7b) |\n| [\ud83e\udd17 BM25 Retrieval 27K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_27K) | [\ud83e\udd99 SWE-Llama 7b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-7b-peft) |\n| [\ud83e\udd17 BM25 Retrieval 40K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_40K) | |\n| [\ud83e\udd17 BM25 Retrieval 50K (Llama tokens)](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_50k_llama) | |\n\n## \ud83c\udf4e Tutorials\nWe've also written the following blog posts on how to use different parts of SWE-bench.\nIf you'd like to see a post about a particular topic, please let us know via an issue.\n* [Nov 1. 2023] Collecting Evaluation Tasks for SWE-Bench ([\ud83d\udd17](https://github.com/princeton-nlp/SWE-bench/blob/main/assets/collection.md))\n* [Nov 6. 2023] Evaluating on SWE-bench ([\ud83d\udd17](https://github.com/princeton-nlp/SWE-bench/blob/main/assets/evaluation.md))\n\n## \ud83d\udcab Contributions\nWe would love to hear from the broader NLP, Machine Learning, and Software Engineering research communities, and we welcome any contributions, pull requests, or issues!\nTo do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!\n\nContact person: [Carlos E. Jimenez](http://www.carlosejimenez.com/) and [John Yang](https://john-b-yang.github.io/) (Email: carlosej@princeton.edu, johnby@stanford.edu).\n\n## \u270d\ufe0f Citation\nIf you find our work helpful, please use the following citations.\n```\n@inproceedings{\n jimenez2024swebench,\n title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},\n author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},\n booktitle={The Twelfth International Conference on Learning Representations},\n year={2024},\n url={https://openreview.net/forum?id=VTF8yNQM66}\n}\n```\n\n## \ud83e\udeaa License\nMIT. Check `LICENSE.md`.\n",
"bugtrack_url": null,
"license": null,
"summary": "The official SWE-bench package - a benchmark for evaluating LMs on software engineering",
"version": "2.1.3",
"project_urls": {
"Bug Reports": "http://github.com/princeton-nlp/SWE-bench/issues",
"Documentation": "https://github.com/princeton-nlp/SWE-bench",
"Homepage": "https://swebench.com",
"Source Code": "http://github.com/princeton-nlp/SWE-bench",
"Website": "https://swebench.com"
},
"split_keywords": [
"nlp",
" benchmark",
" code"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b1a5e65a1101d26baa23884756c79aec6aec1dd61456be9d311612ff67794a46",
"md5": "7c363e84fc2ab97851bd01e619d2f986",
"sha256": "d0f262127d05a2db47dae94ac583ad344384d32fdc4b823a5ea2e560c26d19bc"
},
"downloads": -1,
"filename": "swebench-2.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7c363e84fc2ab97851bd01e619d2f986",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 102813,
"upload_time": "2024-11-17T09:05:41",
"upload_time_iso_8601": "2024-11-17T09:05:41.411669Z",
"url": "https://files.pythonhosted.org/packages/b1/a5/e65a1101d26baa23884756c79aec6aec1dd61456be9d311612ff67794a46/swebench-2.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "049182ffe8db72768fd6380f534e8accde63b504c739b6a28b616cd23ca0243e",
"md5": "69d39100f2661bde7e525c1361c54a8a",
"sha256": "e4e7bb9169714929ce2b78e2a4c4f07e938f3b98db59847b5b23b75e5a539fbc"
},
"downloads": -1,
"filename": "swebench-2.1.3.tar.gz",
"has_sig": false,
"md5_digest": "69d39100f2661bde7e525c1361c54a8a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 91724,
"upload_time": "2024-11-17T09:05:42",
"upload_time_iso_8601": "2024-11-17T09:05:42.747465Z",
"url": "https://files.pythonhosted.org/packages/04/91/82ffe8db72768fd6380f534e8accde63b504c739b6a28b616cd23ca0243e/swebench-2.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-17 09:05:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "princeton-nlp",
"github_project": "SWE-bench",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "swebench"
}
