| Name | synthegrator JSON |
| Version |
0.9.6.0
JSON |
| download |
| home_page | None |
| Summary | Framework for code synthesis and AI4SE research |
| upload_time | 2024-09-13 05:39:54 |
| maintainer | None |
| docs_url | None |
| author | David Gros, Claudio Spiess |
| requires_python | >=3.10 |
| license | Copyright 2024 David Gros, Claudio Spiess Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
code synthesis
llm
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Synthegrator
Synthegrator is a framework for code generation problems. It simplifies
the process of loading common datasets and solving them with language models.
# Installation
```bash
pip install synthegrator
```
Also, for execution you will need to [install docker](https://docs.docker.com/engine/install/).
# Example
Let's take a look at an example of how we can run a solver over
the HumanEval dataset, which collects 164 function synthesis problems.
```python
# Imports
from lmwrapper.openai_wrapper import get_open_ai_lm, OpenAiModelNames
from synthegrator.code_solver import LmCodeSolverAutoRegressive
from synthegrator.execution_threading import solve_and_evaluate_problems
from synthegrator.synthdatasets.human_eval import yield_human_eval
from synthegrator.df_converters import solution_evals_to_df
# Loading of a selection of AI4SE Datasets
problems = list(yield_human_eval())
# Create a solver that can solve a problem
lm = get_open_ai_lm(OpenAiModelNames.gpt_3_5_turbo_instruct)
# ^ Make sure to add your API key to OPENAI_API_KEY or a file.
# See https://github.com/DaiseyCode/lmwrapper for more.
solver = LmCodeSolverAutoRegressive(lm)
# Generate code and execute problems testcases
evals = list(solve_and_evaluate_problems(
solver=solver,
problems=problems,
max_threads_eval=4,
))
# Convert to a dataframe
df = solution_evals_to_df(
evals,
pickle_gzip_whole_solution_eval=True
)
print("Fraction Passing", df.main_metric__is_success.mean())
```
# Architecture
## Guiding Design Requirements
- DR-1 **Support Diverse Datasets and Tasks.** We want an architecture that can
support a diverse tasks (including potentially complex, repository-level tasks).
- DR-2 **Consistent & Efficient Execution.** Experiments often involve running LLM-generated code. We want this to be fast, efficient, and reasonably secure.
- DR-3 **Adaptable to State-of-the-Art Models.** This includes models like those from OpenAI or on HuggingFace. Additionally be adaptable to models
that might do complex retrieval or reasoning
- DR-4 **Maintainable.** Try to follow best practices around automated testing and continuous integration.
## Diagram
Raw data
{
"_id": null,
"home_page": null,
"name": "synthegrator",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "code synthesis, llm",
"author": "David Gros, Claudio Spiess",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/3c/45/dcc34b53fa8a96a64a2e424ee2521fa0fc0c0558087c0a865601dba74e81/synthegrator-0.9.6.0.tar.gz",
"platform": null,
"description": "# Synthegrator\n\nSynthegrator is a framework for code generation problems. It simplifies\nthe process of loading common datasets and solving them with language models.\n\n# Installation\n```bash\npip install synthegrator\n```\n\nAlso, for execution you will need to [install docker](https://docs.docker.com/engine/install/).\n\n\n# Example\nLet's take a look at an example of how we can run a solver over\nthe HumanEval dataset, which collects 164 function synthesis problems.\n\n```python\n# Imports\nfrom lmwrapper.openai_wrapper import get_open_ai_lm, OpenAiModelNames\nfrom synthegrator.code_solver import LmCodeSolverAutoRegressive\nfrom synthegrator.execution_threading import solve_and_evaluate_problems\nfrom synthegrator.synthdatasets.human_eval import yield_human_eval\nfrom synthegrator.df_converters import solution_evals_to_df\n\n# Loading of a selection of AI4SE Datasets\nproblems = list(yield_human_eval())\n\n# Create a solver that can solve a problem\nlm = get_open_ai_lm(OpenAiModelNames.gpt_3_5_turbo_instruct)\n# ^ Make sure to add your API key to OPENAI_API_KEY or a file. \n# See https://github.com/DaiseyCode/lmwrapper for more.\nsolver = LmCodeSolverAutoRegressive(lm)\n\n# Generate code and execute problems testcases\nevals = list(solve_and_evaluate_problems(\n solver=solver,\n problems=problems,\n max_threads_eval=4,\n))\n# Convert to a dataframe\ndf = solution_evals_to_df(\n evals, \n pickle_gzip_whole_solution_eval=True\n)\nprint(\"Fraction Passing\", df.main_metric__is_success.mean())\n```\n\n# Architecture\n## Guiding Design Requirements\n- DR-1 **Support Diverse Datasets and Tasks.** We want an architecture that can\nsupport a diverse tasks (including potentially complex, repository-level tasks).\n- DR-2 **Consistent & Efficient Execution.** Experiments often involve running LLM-generated code. We want this to be fast, efficient, and reasonably secure.\n- DR-3 **Adaptable to State-of-the-Art Models.** This includes models like those from OpenAI or on HuggingFace. Additionally be adaptable to models\nthat might do complex retrieval or reasoning\n- DR-4 **Maintainable.** Try to follow best practices around automated testing and continuous integration.\n\n## Diagram\n\n",
"bugtrack_url": null,
"license": "Copyright 2024 David Gros, Claudio Spiess Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u201cSoftware\u201d), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u201cAS IS\u201d, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Framework for code synthesis and AI4SE research",
"version": "0.9.6.0",
"project_urls": {
"Homepage": "https://github.com/DaiseyCode/synthegrator"
},
"split_keywords": [
"code synthesis",
" llm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c8df24a94f434e90aef9b58b8a950b2179c1d10f6d0ce1b474bd5e59c51c7c65",
"md5": "76af1dbf2dd99a9ca9393f0128371196",
"sha256": "7c1201911660a0449c84e7ca30c7768eb19459c6a3a02c09a3639105cd539cdb"
},
"downloads": -1,
"filename": "synthegrator-0.9.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "76af1dbf2dd99a9ca9393f0128371196",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 3197498,
"upload_time": "2024-09-13T05:39:48",
"upload_time_iso_8601": "2024-09-13T05:39:48.279420Z",
"url": "https://files.pythonhosted.org/packages/c8/df/24a94f434e90aef9b58b8a950b2179c1d10f6d0ce1b474bd5e59c51c7c65/synthegrator-0.9.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3c45dcc34b53fa8a96a64a2e424ee2521fa0fc0c0558087c0a865601dba74e81",
"md5": "2c6642a63c7937f48d080b15d5c47c74",
"sha256": "22075e9c92823023ba376294ed9b981ca318393840d1e8fc72835999cf933f80"
},
"downloads": -1,
"filename": "synthegrator-0.9.6.0.tar.gz",
"has_sig": false,
"md5_digest": "2c6642a63c7937f48d080b15d5c47c74",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 3166163,
"upload_time": "2024-09-13T05:39:54",
"upload_time_iso_8601": "2024-09-13T05:39:54.658218Z",
"url": "https://files.pythonhosted.org/packages/3c/45/dcc34b53fa8a96a64a2e424ee2521fa0fc0c0558087c0a865601dba74e81/synthegrator-0.9.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-13 05:39:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DaiseyCode",
"github_project": "synthegrator",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "synthegrator"
}
