# LLMCompiler
[](README.md)
[](README-zh.md)
 LLMCompiler is an Agent Architecture designed to speed up the execution of agent tasks by executing them quickly
in the DAG. It also saves the cost of redundant token use by reducing the number of calls to the LLM. The realization
inspiration comes from An LLM Compiler for Parallel Function Calling.
 Here is an example of using SQL to query data to illustrate the core role of the framework. The core process of generating an execution plan for SQL includes syntax parsing, semantic analysis, optimizer intervention, and generation of an execution plan. When LLMCompiler executes tool calls based on user instructions, it can actually be understood that LLM helps users do a process similar to SQL to generate execution plans, but the generated plan here is a DAG, and the DAG describes the call relationship between tools and the parameter dependency passing logic.
 This implementation is useful when the agent needs to call a large number of tools. If the tool you need exceeds
the context limit of the LLM, you can extend the agent node based on this tool.Divide the tool into different
agent and assemble them to create a more powerful LLMCompiler. Another case has been
proven in a production-level application, when about 60 Tools were configured, and the accuracy rate was more than 90%
when paired with few-shot.
## LLMCompiler Frame Diagram
 This image illustrates the system architecture of the LLMCompiler, detailing the entire process of how a user request is
handled, planned, executed, and fed back within the system. Below is a detailed description of each part of the diagram:
1. User Request:
 On the left, a user avatar represents the system's end-user. The user initiates a request to the system
through an interface or other input methods, marking the start of the entire process.
2. Planner:
 The user request first enters the "Planner" module, symbolized by a brain icon, representing intelligence and
decision-making capability (LLM). The planner's main responsibility is to parse the user request, understand its intent,
and generate a series of executable task plans based on this understanding. This plan is organized into a Directed
Acyclic Graph (DAG), representing the sequence and dependencies of the tasks.
3. Stream Task DAG:
 The task DAG stream generated by the planner is passed to the "Task Fetching Unit." The task DAG stream
is a structure that represents the relationships between tasks, ensuring that tasks are executed correctly according to
their dependencies. Each node in the DAG stream represents a specific task, and the edges indicate the dependencies
between tasks.
4. Task Fetching Unit:
 The Task Fetching Unit is the core execution module of the system. It is responsible for extracting
tasks from the DAG and scheduling them according to their dependencies. Tasks are executed in parallel as much as
possible to improve efficiency. In the diagram, tasks are represented by the letters A, B, C, and D, with arrows
indicating the dependencies and execution order. The tool icons (such as a hammer) signify that this module not only
schedules tasks but also executes them.
5. Update State with Task Results:
 After task execution is complete, the results are used to update the system's internal
state. State updates are a crucial step in ensuring that all tasks are correctly executed and progress is recorded.
6. Joiner (Replanner):
 The updated state is passed to the "Joiner" module, which is also represented by a brain icon,
indicating its complex decision-making capabilities. The Joiner's role is to evaluate the updated state, and if the task
results are insufficient to satisfy the user request, it will replan additional tasks and resubmit them to the Task
Fetching Unit for execution. If the task results are sufficient to fulfill the user request, the Joiner will prepare the
final results to be fed back to the user.
7. Respond to User:
 The final task results are generated by the Joiner and fed back to the user. This closes the loop, with
the user receiving the requested results or information through the system.
 Overall, this diagram presents a complex task scheduling and execution system, emphasizing the entire process from user
request to task planning, parallel execution, state updating, and feedback. The brain icon symbolizes the intelligent
decision-making modules (LLM) within the system, while the arrows and task nodes demonstrate the professional handling
of task flows and dependencies.
## Task Fetching Unit
 This image depicts the workflow diagram of the LLMCompiler framework, which is designed to efficiently execute tasks by
parallelly invoking LLMs (Large Language Models). The image is divided into several key sections, described as follows:
1. User Input:
 On the left side, the user inputs a question in natural language, such as "How much does Microsoft's market cap need to
increase to surpass Apple's market cap?"
2. LLM Planner:
 The user's input is passed to the LLM Planner, which parses the user's request into a series of tasks (DAG of Tasks).
For example:
- $1 = search(Microsoft Market Cap): Find Microsoft's market cap.
- $2 = search(Apple Market Cap): Find Apple's market cap.
- $3 = math($1 / $2): Perform a calculation to compare the two market caps.
- $4 = llm($3): Pass the result to the large language model for further processing.
3. Task Fetching Unit:
 The Task Fetching Unit is responsible for retrieving tasks from the LLM Planner and parsing the dependencies between the
tasks. This unit is represented by a diagram (circles and arrows) that shows how tasks are executed sequentially or in
parallel.
4. Executor:
 The Executor contains multiple "Function Calling Units," each equipped with a tool (Tool) and memory (Memory). All tool
calls are temporarily stored in memory for later use by the parser.
Each unit within the Executor is responsible for executing specific tasks, such as invoking a search engine, performing
mathematical calculations, or calling the LLM.
5. Toolbar:
 The bottom section displays icons of several tools, including search tools, mathematical tools, and large language
models (LLMs), among others. These tools are used to execute different parts of the user's request.
 The main functionality of the LLMCompiler framework is to achieve efficient and effective parallel function calls by
automatically identifying which tasks can be executed in parallel and which are interdependent.
 Overall, the image illustrates how the LLMCompiler framework begins with user input, plans tasks, parses task
dependencies, and finally invokes various tools to complete the tasks.
## How To Use
```shell
pip install llmcompiler
```
```py
from llmcompiler.result.chat import ChatRequest
from llmcompiler.tools.basic import Tools
from langchain_openai.chat_models.base import ChatOpenAI
from llmcompiler.chat.run import RunLLMCompiler
chat = ChatRequest(message="<YOUR_MESSAGE>")
# `tools` is a list based on Langchain BaseTool, `Tools.load_tools` can automatically load Tools from specified directories or `.py` files.
# The default configuration is only for demonstration purposes, it is recommended to inherit `BaseTool` or `CompilerBaseTool` to implement Tool, which can better control some details.
# No parameter dependencies are required, you can inherit `BaseTool` to implement Tool, with the implementation reference being `llmcompiler/tools/basetool/fund_basic_v1.py`.
# Parameter dependencies are required, you can inherit `CompilerBaseTool`, with the implementation references being `llmcompiler/tools/math/math_tools.py, llmcompiler/tools/basetool/fund_basic_v2.py`.
tools = Tools.load_tools("../llmcompiler/tools/math")
# The implementation class of BaseLanguageModel is supported.
llm = ChatOpenAI(model="gpt-4o", temperature=0, max_retries=3)
llm_compiler = RunLLMCompiler(chat, tools, llm)
# Run the full LLMCompiler process.
print(llm_compiler())
# Ignore the joiner process and return the task and execution result directly.
print(llm_compiler.runWithoutJoiner())
# More ways to use it can be discussed in the issue, and I will continue to improve the documentation in the future.
```
## Case
[Example of Performing Complex Mathematical Calculations](docs/dag-demo.md)
[Some implementation examples of tools](llmcompiler/tools/basetool)
## Reference Linking
- [Paper: An LLM Compiler for Parallel Function Calling](https://arxiv.org/abs/2312.04511)
- [Partial Code: LLMCompiler From Github](https://github.com/langchain-ai/langgraph/blob/main/examples/llm-compiler/LLMCompiler.ipynb)
- [ICML 2024 LLMCompiler: An LLM Compiler for Parallel Function Calling](https://github.com/SqueezeAILab/LLMCompiler)
Raw data
{
"_id": null,
"home_page": "https://github.com/crazyyanchao/llmcompiler",
"name": "llmcompiler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "LLMCompiler, Agent, Natural Language Processing, LLM, Machine Learning, AI, Compiler",
"author": "Yc-Ma",
"author_email": "yanchaoma@foxmail.com",
"download_url": "https://files.pythonhosted.org/packages/29/d6/075424b6c7dc51d1e3cc3adc450ffe182183aeb879402bc5eaf4c9606007/llmcompiler-1.3.0.tar.gz",
"platform": null,
"description": "# LLMCompiler\n\n[](README.md)\n[](README-zh.md)\n\n LLMCompiler is an Agent Architecture designed to speed up the execution of agent tasks by executing them quickly\nin the DAG. It also saves the cost of redundant token use by reducing the number of calls to the LLM. The realization\ninspiration comes from An LLM Compiler for Parallel Function Calling.\n\n Here is an example of using SQL to query data to illustrate the core role of the framework. The core process of generating an execution plan for SQL includes syntax parsing, semantic analysis, optimizer intervention, and generation of an execution plan. When LLMCompiler executes tool calls based on user instructions, it can actually be understood that LLM helps users do a process similar to SQL to generate execution plans, but the generated plan here is a DAG, and the DAG describes the call relationship between tools and the parameter dependency passing logic.\n\n This implementation is useful when the agent needs to call a large number of tools. If the tool you need exceeds\nthe context limit of the LLM, you can extend the agent node based on this tool.Divide the tool into different\nagent and assemble them to create a more powerful LLMCompiler. Another case has been\nproven in a production-level application, when about 60 Tools were configured, and the accuracy rate was more than 90%\nwhen paired with few-shot.\n\n## LLMCompiler Frame Diagram\n\n\n\n This image illustrates the system architecture of the LLMCompiler, detailing the entire process of how a user request is\nhandled, planned, executed, and fed back within the system. Below is a detailed description of each part of the diagram:\n\n1. User Request:\n\n On the left, a user avatar represents the system's end-user. The user initiates a request to the system\nthrough an interface or other input methods, marking the start of the entire process.\n\n2. Planner:\n\n The user request first enters the \"Planner\" module, symbolized by a brain icon, representing intelligence and\ndecision-making capability (LLM). The planner's main responsibility is to parse the user request, understand its intent,\nand generate a series of executable task plans based on this understanding. This plan is organized into a Directed\nAcyclic Graph (DAG), representing the sequence and dependencies of the tasks.\n\n3. Stream Task DAG:\n\n The task DAG stream generated by the planner is passed to the \"Task Fetching Unit.\" The task DAG stream\nis a structure that represents the relationships between tasks, ensuring that tasks are executed correctly according to\ntheir dependencies. Each node in the DAG stream represents a specific task, and the edges indicate the dependencies\nbetween tasks.\n\n4. Task Fetching Unit:\n\n The Task Fetching Unit is the core execution module of the system. It is responsible for extracting\ntasks from the DAG and scheduling them according to their dependencies. Tasks are executed in parallel as much as\npossible to improve efficiency. In the diagram, tasks are represented by the letters A, B, C, and D, with arrows\nindicating the dependencies and execution order. The tool icons (such as a hammer) signify that this module not only\nschedules tasks but also executes them.\n\n5. Update State with Task Results:\n\n After task execution is complete, the results are used to update the system's internal\nstate. State updates are a crucial step in ensuring that all tasks are correctly executed and progress is recorded.\n\n6. Joiner (Replanner):\n\n The updated state is passed to the \"Joiner\" module, which is also represented by a brain icon,\nindicating its complex decision-making capabilities. The Joiner's role is to evaluate the updated state, and if the task\nresults are insufficient to satisfy the user request, it will replan additional tasks and resubmit them to the Task\nFetching Unit for execution. If the task results are sufficient to fulfill the user request, the Joiner will prepare the\nfinal results to be fed back to the user.\n\n7. Respond to User:\n\n The final task results are generated by the Joiner and fed back to the user. This closes the loop, with\nthe user receiving the requested results or information through the system.\n\n Overall, this diagram presents a complex task scheduling and execution system, emphasizing the entire process from user\nrequest to task planning, parallel execution, state updating, and feedback. The brain icon symbolizes the intelligent\ndecision-making modules (LLM) within the system, while the arrows and task nodes demonstrate the professional handling\nof task flows and dependencies.\n\n## Task Fetching Unit\n\n\n\n This image depicts the workflow diagram of the LLMCompiler framework, which is designed to efficiently execute tasks by\nparallelly invoking LLMs (Large Language Models). The image is divided into several key sections, described as follows:\n\n1. User Input:\n\n On the left side, the user inputs a question in natural language, such as \"How much does Microsoft's market cap need to\nincrease to surpass Apple's market cap?\"\n\n2. LLM Planner:\n\n The user's input is passed to the LLM Planner, which parses the user's request into a series of tasks (DAG of Tasks).\nFor example:\n\n- $1 = search(Microsoft Market Cap): Find Microsoft's market cap.\n- $2 = search(Apple Market Cap): Find Apple's market cap.\n- $3 = math($1 / $2): Perform a calculation to compare the two market caps.\n- $4 = llm($3): Pass the result to the large language model for further processing.\n\n3. Task Fetching Unit:\n\n The Task Fetching Unit is responsible for retrieving tasks from the LLM Planner and parsing the dependencies between the\ntasks. This unit is represented by a diagram (circles and arrows) that shows how tasks are executed sequentially or in\nparallel.\n\n4. Executor:\n\n The Executor contains multiple \"Function Calling Units,\" each equipped with a tool (Tool) and memory (Memory). All tool\ncalls are temporarily stored in memory for later use by the parser.\nEach unit within the Executor is responsible for executing specific tasks, such as invoking a search engine, performing\nmathematical calculations, or calling the LLM.\n\n5. Toolbar:\n\n The bottom section displays icons of several tools, including search tools, mathematical tools, and large language\nmodels (LLMs), among others. These tools are used to execute different parts of the user's request.\n\n The main functionality of the LLMCompiler framework is to achieve efficient and effective parallel function calls by\nautomatically identifying which tasks can be executed in parallel and which are interdependent.\n\n Overall, the image illustrates how the LLMCompiler framework begins with user input, plans tasks, parses task\ndependencies, and finally invokes various tools to complete the tasks.\n\n## How To Use\n\n```shell\npip install llmcompiler\n```\n\n```py\nfrom llmcompiler.result.chat import ChatRequest\nfrom llmcompiler.tools.basic import Tools\nfrom langchain_openai.chat_models.base import ChatOpenAI\nfrom llmcompiler.chat.run import RunLLMCompiler\n\nchat = ChatRequest(message=\"<YOUR_MESSAGE>\")\n\n# `tools` is a list based on Langchain BaseTool, `Tools.load_tools` can automatically load Tools from specified directories or `.py` files.\n# The default configuration is only for demonstration purposes, it is recommended to inherit `BaseTool` or `CompilerBaseTool` to implement Tool, which can better control some details.\n# No parameter dependencies are required, you can inherit `BaseTool` to implement Tool, with the implementation reference being `llmcompiler/tools/basetool/fund_basic_v1.py`.\n# Parameter dependencies are required, you can inherit `CompilerBaseTool`, with the implementation references being `llmcompiler/tools/math/math_tools.py, llmcompiler/tools/basetool/fund_basic_v2.py`.\ntools = Tools.load_tools(\"../llmcompiler/tools/math\")\n\n# The implementation class of BaseLanguageModel is supported.\nllm = ChatOpenAI(model=\"gpt-4o\", temperature=0, max_retries=3)\n\nllm_compiler = RunLLMCompiler(chat, tools, llm)\n# Run the full LLMCompiler process.\nprint(llm_compiler())\n\n# Ignore the joiner process and return the task and execution result directly.\nprint(llm_compiler.runWithoutJoiner())\n\n# More ways to use it can be discussed in the issue, and I will continue to improve the documentation in the future.\n```\n\n## Case\n\n[Example of Performing Complex Mathematical Calculations](docs/dag-demo.md)\n\n[Some implementation examples of tools](llmcompiler/tools/basetool)\n\n## Reference Linking\n\n- [Paper: An LLM Compiler for Parallel Function Calling](https://arxiv.org/abs/2312.04511)\n- [Partial Code: LLMCompiler From Github](https://github.com/langchain-ai/langgraph/blob/main/examples/llm-compiler/LLMCompiler.ipynb)\n- [ICML 2024 LLMCompiler: An LLM Compiler for Parallel Function Calling](https://github.com/SqueezeAILab/LLMCompiler)\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "LLMCompiler",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/crazyyanchao/llmcompiler"
},
"split_keywords": [
"llmcompiler",
" agent",
" natural language processing",
" llm",
" machine learning",
" ai",
" compiler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "af31f1098f1831a4a793cc79260de11eff79136f66ad0ead3f4703953150233b",
"md5": "d19103ae1bc8c6f3c2c9b7920ec68948",
"sha256": "7e1945458d663423e550d6e4dcacea2655d2e86d78e0173b39eb1fef0d9fdecb"
},
"downloads": -1,
"filename": "llmcompiler-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d19103ae1bc8c6f3c2c9b7920ec68948",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 155951,
"upload_time": "2024-11-28T07:53:46",
"upload_time_iso_8601": "2024-11-28T07:53:46.888442Z",
"url": "https://files.pythonhosted.org/packages/af/31/f1098f1831a4a793cc79260de11eff79136f66ad0ead3f4703953150233b/llmcompiler-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "29d6075424b6c7dc51d1e3cc3adc450ffe182183aeb879402bc5eaf4c9606007",
"md5": "4978db1826b44a77cd6a355058f55bce",
"sha256": "1c2ebfe64e4e0acba2d836ac62223c18f0abe98c3bc3b208594f058a2c048298"
},
"downloads": -1,
"filename": "llmcompiler-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "4978db1826b44a77cd6a355058f55bce",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 128971,
"upload_time": "2024-11-28T07:53:48",
"upload_time_iso_8601": "2024-11-28T07:53:48.655153Z",
"url": "https://files.pythonhosted.org/packages/29/d6/075424b6c7dc51d1e3cc3adc450ffe182183aeb879402bc5eaf4c9606007/llmcompiler-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-28 07:53:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "crazyyanchao",
"github_project": "llmcompiler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "llmcompiler"
}
JSON
