| Name | openai-autopilot JSON |
| Version |
0.0.4
JSON |
| download |
| home_page | None |
| Summary | Autopilot enhances OpenAI interactions with concurrent task processing, faster data handling, reliable progress tracking, and transparency. |
| upload_time | 2024-01-16 11:42:57 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
autopilot
chatgpt
gpt3.5
gpt4
openai
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<img src="https://raw.githubusercontent.com/pop-srw/openai-autopilot/main/docs/logo.png" alt="Autopilot logo" width="150" />
<h1 align="center">OpenAI Autopilot</h1>
</p>
[](https://pypi.org/project/openai-autopilot)
[](https://pypi.org/project/openai-autopilot)
Autopilot simplifies your OpenAI interactions by concurrently processing multiple tasks, significantly speeding up data handling. It uses temporary files for checkpointing, ensuring no progress is lost during unexpected interruptions. Plus, with its real-time progress tracking, you're always informed about your task completion status, making your workflow not only faster but also more reliable and transparent.
---
**Table of Contents**
- [Features](#features)
- [How it works](#how-it-works)
- [Installation](#installation)
- [Usage](#usage)
- [Configuration Options](#configuration-options)
- [Example](#example)
- [License](#license)
## Features
- **Concurrent Processing**: Efficiently handles multiple OpenAI tasks in parallel, greatly speeding up data processing.
- **Checkpointing with Temporary Files**: Safely saves progress using temporary files, allowing continuation from the last checkpoint in case of interruptions.
- **Real-Time Progress Tracking**: Features a progress bar that provides immediate updates on the status of ongoing tasks, enhancing transparency and planning.
- **Asynchronous Operation**: Leverages Python's asyncio for non-blocking task execution, improving overall performance.
- **Error Handling**: Robust exception management to ensure smooth operation under various scenarios.
- **Type Checking with Pydantic**: Utilizes Pydantic for rigorous data validation and type checking, ensuring that the inputs and outputs of tasks adhere to defined schemas. This enhances the reliability and consistency of the data being processed.
## How it works
The Autopilot framework streamlines asynchronous data processing with OpenAI's API. It queues data items and processes them using multiple concurrent workers, each executing a user-defined function that interacts with the OpenAI API. Importantly, if a temporary file for a task already exists, Autopilot skips reprocessing it, efficiently avoiding redundant work. As tasks are completed, results are saved in temporary files, allowing the process to resume seamlessly from the last checkpoint in case of interruptions. This setup ensures efficient handling and error management. Progress is visually tracked, and upon completion, results are compiled into a structured format for final use, simplifying the complexity of asynchronous operations.
## Installation
```shell
pip install openai-autopilot
```
## Usage
In this section, we demonstrate how to utilize the Autopilot for efficient asynchronous data processing with OpenAI's API. The process involves defining a custom asynchronous function that interfaces with OpenAI, initializing the OpenAI client, setting up the Autopilot with the processing function, and executing the process with your data. The steps are as follows:
1. **Define the Processing Function**: Create an `async` function named `process`, which encapsulates the logic for interacting with OpenAI's API.
2. **Initialize OpenAI Client**: Set up the `AsyncOpenAI` client using your OpenAI API key.
3. **Configure Autopilot**: Instantiate the `Autopilot` class with the initialized OpenAI client and the custom processing function.
4. **Run the Process**: Use the `run` method of the Autopilot instance, passing in your data encapsulated in `AutopilotDataList`.
5. **Access Processed Data**: The `run` method returns the processed data, which can be used as needed.
This setup is designed to offer a straightforward approach to handling asynchronous tasks with OpenAI, enhancing both efficiency and ease of use.
```python
from typing import Coroutine, List
from openai import AsyncOpenAI
from openai_autopilot import Autopilot, AutopilotMessage, AutopilotDataList
# Define the processing function, wrapping around the normal OpenAI async API usage
async def process(
worker_id: int, client: AsyncOpenAI, data_id: int, messages: List[AutopilotMessage]
) -> Coroutine[None, None, str]:
chat_completion = await client.chat.completions.create(
# ...insert your configuration here...
)
completion_text = chat_completion.choices[0].message.content
# Return the string to be added to the response of your data
return completion_text
# Initialize the OpenAI asynchronous client
api_key = "...your api key..."
client = AsyncOpenAI(api_key=api_key)
autopilot = Autopilot(client=client, process_fn=process)
processed_data = autopilot.run(
AutopilotDataList(
# ...insert your data here...
)
)
```
## Configuration Options
When initializing the `Autopilot` class, you can configure it using the following parameters:
1. **client (`AsyncOpenAI`)**: The OpenAI client instance. It should be an instance of `AsyncOpenAI`. If not provided, it defaults to `None`.
2. **process_fn (Callable)**: The processing function that defines how each item in the data list is processed. It should be an asynchronous function that accepts parameters `worker_id`, `client`, `data_id`, and `messages`, and returns a coroutine with a string response. If not provided, it defaults to `None`.
3. **concurrency (`int`)**: Determines the number of concurrent workers that process the data. The default value is 5.
4. **tmp_dir (`str`)**: The directory path for storing temporary files used in checkpointing. Defaults to `"tmp"`.
5. **tmp_file_prefix (`str`)**: The prefix for naming temporary files. Defaults to `"data"`.
6. **verbose (`bool`)**: A boolean flag to enable verbose logging. Useful for debugging. Defaults to `False`.
These parameters allow you to customize the behavior of the Autopilot framework according to your specific requirements.
## Example
In our example, we demonstrate how to use the Autopilot for processing requests with OpenAI's GPT-4 model. This example generates recipes for various food items. The results are outputted in a JSON format, providing an easy-to-read display of the generated recipes.
```shell
export OPENAI_API_KEY='...your api key...'
python examples/simple_process.py
```
## License
`openai-autopilot` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.
## Acknowledgments
Thanks to ChatGPT for invaluable assistance in creating this document.
Raw data
{
"_id": null,
"home_page": null,
"name": "openai-autopilot",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "autopilot,chatgpt,gpt3.5,gpt4,openai",
"author": null,
"author_email": "pop-srw <khemmachotikun.s@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/26/f4/58a96cd829af37e92da5e31fa37826ff417eb59482fb41444c5a14b77d61/openai_autopilot-0.0.4.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://raw.githubusercontent.com/pop-srw/openai-autopilot/main/docs/logo.png\" alt=\"Autopilot logo\" width=\"150\" />\n <h1 align=\"center\">OpenAI Autopilot</h1>\n</p>\n\n[](https://pypi.org/project/openai-autopilot)\n[](https://pypi.org/project/openai-autopilot)\n\nAutopilot simplifies your OpenAI interactions by concurrently processing multiple tasks, significantly speeding up data handling. It uses temporary files for checkpointing, ensuring no progress is lost during unexpected interruptions. Plus, with its real-time progress tracking, you're always informed about your task completion status, making your workflow not only faster but also more reliable and transparent.\n\n---\n\n**Table of Contents**\n\n- [Features](#features)\n- [How it works](#how-it-works)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Configuration Options](#configuration-options)\n- [Example](#example)\n- [License](#license)\n\n## Features\n\n- **Concurrent Processing**: Efficiently handles multiple OpenAI tasks in parallel, greatly speeding up data processing.\n- **Checkpointing with Temporary Files**: Safely saves progress using temporary files, allowing continuation from the last checkpoint in case of interruptions.\n- **Real-Time Progress Tracking**: Features a progress bar that provides immediate updates on the status of ongoing tasks, enhancing transparency and planning.\n- **Asynchronous Operation**: Leverages Python's asyncio for non-blocking task execution, improving overall performance.\n- **Error Handling**: Robust exception management to ensure smooth operation under various scenarios.\n- **Type Checking with Pydantic**: Utilizes Pydantic for rigorous data validation and type checking, ensuring that the inputs and outputs of tasks adhere to defined schemas. This enhances the reliability and consistency of the data being processed.\n\n## How it works\n\nThe Autopilot framework streamlines asynchronous data processing with OpenAI's API. It queues data items and processes them using multiple concurrent workers, each executing a user-defined function that interacts with the OpenAI API. Importantly, if a temporary file for a task already exists, Autopilot skips reprocessing it, efficiently avoiding redundant work. As tasks are completed, results are saved in temporary files, allowing the process to resume seamlessly from the last checkpoint in case of interruptions. This setup ensures efficient handling and error management. Progress is visually tracked, and upon completion, results are compiled into a structured format for final use, simplifying the complexity of asynchronous operations.\n\n## Installation\n\n```shell\npip install openai-autopilot\n```\n\n## Usage\n\nIn this section, we demonstrate how to utilize the Autopilot for efficient asynchronous data processing with OpenAI's API. The process involves defining a custom asynchronous function that interfaces with OpenAI, initializing the OpenAI client, setting up the Autopilot with the processing function, and executing the process with your data. The steps are as follows:\n\n1. **Define the Processing Function**: Create an `async` function named `process`, which encapsulates the logic for interacting with OpenAI's API.\n\n2. **Initialize OpenAI Client**: Set up the `AsyncOpenAI` client using your OpenAI API key.\n\n3. **Configure Autopilot**: Instantiate the `Autopilot` class with the initialized OpenAI client and the custom processing function.\n\n4. **Run the Process**: Use the `run` method of the Autopilot instance, passing in your data encapsulated in `AutopilotDataList`.\n\n5. **Access Processed Data**: The `run` method returns the processed data, which can be used as needed.\n\nThis setup is designed to offer a straightforward approach to handling asynchronous tasks with OpenAI, enhancing both efficiency and ease of use.\n\n```python\nfrom typing import Coroutine, List\nfrom openai import AsyncOpenAI\nfrom openai_autopilot import Autopilot, AutopilotMessage, AutopilotDataList\n\n# Define the processing function, wrapping around the normal OpenAI async API usage\nasync def process(\n worker_id: int, client: AsyncOpenAI, data_id: int, messages: List[AutopilotMessage]\n) -> Coroutine[None, None, str]:\n chat_completion = await client.chat.completions.create(\n # ...insert your configuration here...\n )\n\n completion_text = chat_completion.choices[0].message.content\n\n # Return the string to be added to the response of your data\n return completion_text\n\n# Initialize the OpenAI asynchronous client\napi_key = \"...your api key...\"\nclient = AsyncOpenAI(api_key=api_key)\nautopilot = Autopilot(client=client, process_fn=process)\n\nprocessed_data = autopilot.run(\n AutopilotDataList(\n # ...insert your data here...\n )\n)\n```\n\n## Configuration Options\n\nWhen initializing the `Autopilot` class, you can configure it using the following parameters:\n\n1. **client (`AsyncOpenAI`)**: The OpenAI client instance. It should be an instance of `AsyncOpenAI`. If not provided, it defaults to `None`.\n\n2. **process_fn (Callable)**: The processing function that defines how each item in the data list is processed. It should be an asynchronous function that accepts parameters `worker_id`, `client`, `data_id`, and `messages`, and returns a coroutine with a string response. If not provided, it defaults to `None`.\n\n3. **concurrency (`int`)**: Determines the number of concurrent workers that process the data. The default value is 5.\n\n4. **tmp_dir (`str`)**: The directory path for storing temporary files used in checkpointing. Defaults to `\"tmp\"`.\n\n5. **tmp_file_prefix (`str`)**: The prefix for naming temporary files. Defaults to `\"data\"`.\n\n6. **verbose (`bool`)**: A boolean flag to enable verbose logging. Useful for debugging. Defaults to `False`.\n\nThese parameters allow you to customize the behavior of the Autopilot framework according to your specific requirements.\n\n## Example\n\nIn our example, we demonstrate how to use the Autopilot for processing requests with OpenAI's GPT-4 model. This example generates recipes for various food items. The results are outputted in a JSON format, providing an easy-to-read display of the generated recipes.\n\n```shell\nexport OPENAI_API_KEY='...your api key...'\npython examples/simple_process.py\n```\n\n## License\n\n`openai-autopilot` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.\n\n## Acknowledgments\n\nThanks to ChatGPT for invaluable assistance in creating this document.\n",
"bugtrack_url": null,
"license": null,
"summary": "Autopilot enhances OpenAI interactions with concurrent task processing, faster data handling, reliable progress tracking, and transparency.",
"version": "0.0.4",
"project_urls": {
"Documentation": "https://github.com/pop-srw/openai-autopilot#readme",
"Issues": "https://github.com/pop-srw/openai-autopilot/issues",
"Source": "https://github.com/pop-srw/openai-autopilot"
},
"split_keywords": [
"autopilot",
"chatgpt",
"gpt3.5",
"gpt4",
"openai"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "88e452c7268bbe2755938af7c4c71708cd5ba837b0c9b9c4d6da4d7b906aaf35",
"md5": "3f9c335d7cd764b4589cdd683560c3e7",
"sha256": "67fe0363cbdca3d1bc4d33dd35b96b2a525d7ff09a1a9472fa9c94bb4499ad0d"
},
"downloads": -1,
"filename": "openai_autopilot-0.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3f9c335d7cd764b4589cdd683560c3e7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 7387,
"upload_time": "2024-01-16T11:43:00",
"upload_time_iso_8601": "2024-01-16T11:43:00.181909Z",
"url": "https://files.pythonhosted.org/packages/88/e4/52c7268bbe2755938af7c4c71708cd5ba837b0c9b9c4d6da4d7b906aaf35/openai_autopilot-0.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "26f458a96cd829af37e92da5e31fa37826ff417eb59482fb41444c5a14b77d61",
"md5": "75277dd9cebd7fbfb675ac4c6ac5bc57",
"sha256": "3e04196d260628f2a203ca23749c2b6bd3d34b38d7ee5d5112356aeeda407f31"
},
"downloads": -1,
"filename": "openai_autopilot-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "75277dd9cebd7fbfb675ac4c6ac5bc57",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1520672,
"upload_time": "2024-01-16T11:42:57",
"upload_time_iso_8601": "2024-01-16T11:42:57.637118Z",
"url": "https://files.pythonhosted.org/packages/26/f4/58a96cd829af37e92da5e31fa37826ff417eb59482fb41444c5a14b77d61/openai_autopilot-0.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-16 11:42:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pop-srw",
"github_project": "openai-autopilot#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "openai-autopilot"
}
