# 💪 Hercules
[](https://github.com/test-zeus-ai/testzeus-hercules/actions/workflows/main-test.yml)
Testing modern web applications can be difficult, with frequent changes and complex features making it hard to keep up. That’s where **Hercules** comes in. Hercules is the world’s first open-source testing agent, built to handle the toughest testing tasks so you don’t have to. It turns simple, easy-to-write Gherkin steps into fully automated **end to end** tests—no coding skills needed. Whether you’re working with tricky platforms like Salesforce or running tests in your CI/CD pipeline, Hercules adapts to your needs and takes care of the details. With Hercules, testing becomes simple, reliable, and efficient, helping teams everywhere deliver better software.
Using Hercules is as simple as feeding in your Gherkin features, and getting the results:
At [TestZeus](www.testzeus.com), we believe that **trustworthy and open-source code** is the backbone of innovation. That's why we've built Hercules to be transparent, reliable, and community-driven.
Our mission? To **democratize and disrupt test automation**, making top-tier testing accessible to everyone, not just the elite few. No more gatekeeping—everyone deserves a hero on their testing team!
### Video Tutorials: [@TestZeus](https://www.youtube.com/@TestZeus)
---
#### 🚀 **Getting Started with TestZeus Hercules**
- **Introduction to TestZeus Hercules**
Learn about the core features of TestZeus Hercules and how it can streamline end-to-end testing for your projects.
_[Watch now](https://youtu.be/_m_NDjM6aZ0?si=ArtVKz8uSgGWTcAK)_
- **Installation and Setup Guide**
Step-by-step instructions for installing and configuring TestZeus Hercules in your environment.
_[Watch now](https://youtu.be/9D-SZGoDrfc?si=GL0IArWkB1ZgBdx5)_
---
#### 🧪 **Writing and Executing Test Cases**
- **Creating BDD Test Cases**
Learn how to write Behavior-Driven Development (BDD) test cases for Hercules and use dynamic testdata.
_[Watch now](https://www.youtube.com/watch?v=yJD0cZ7Bx6Q&t)_
---
#### 🌐 **Integrations and Advanced Features**
- **Testing Multilingual content**
Learn how Hercules interacts with web browsers to perform Testing on Multilingual content via Multilingual testcase.
_[Watch now](https://youtu.be/vI9iGSqKpGA?si=6NGAvKnwFboSyHT2)_
---
#### 🛠️ **Customization and Community Tools**
- **Enhancing Hercules with Community-Driven Tools**
Discover how to customize Hercules and incorporate additional tools provided by the community.
_[Watch now](https://youtu.be/C8hUy5leRF0?si=yVMpZ7WFcI01BoI3)_
---
#### 🛠️ **API Testing**
- **API testing all the way, new ways to do end to end**
_[Watch now](https://youtu.be/qMt89DQH6LQ?si=lcSJbKOCaqqUGeQ8)_
---
#### 🛠️ **Security Testing**
- **Security Testing done end to end**
_[Watch now](https://youtu.be/o6MmfTakIh4?si=JL8iu4e3i85SWrxU)_
---
## ⚙️ Installation and Usage
Hercules offers multiple ways to get started, catering to different user preferences and requirements.
For a quick taste of the solution, you can try the notebook here:
[](https://colab.research.google.com/drive/1YiZsXem1POTwkcr17QqflXnihhuSqwM2?usp=sharing)
- **Note**: Colab might ask you to restart the session as python3.11 and some libs are installed during the installation of testzeus-hercules. Please restart the session if required and continue the execution. Also , we recommend one of the approaches below for getting the full flavor of the solution.
### Approach 1: Using PyPI Package
#### Installation
Install Hercules from PyPI:
```bash
pip install testzeus-hercules
```
Hercules uses Playwright to interact with web pages, so you need to install Playwright and its dependencies:
```bash
playwright install --with-deps
```
#### Basic Parameters
Once installed, you will need to provide some basic parameters to run Hercules:
- `--input-file INPUT_FILE`: Path to the input Gherkin feature file to be tested.
- `--output-path OUTPUT_PATH`: Path to the output directory. The path of JUnit XML result and HTML report for the test run.
- `--test-data-path TEST_DATA_PATH`: Path to the test data directory. The path where Hercules expects test data to be present; all test data used in feature testing should be present here.
- `--project-base PROJECT_BASE`: Path to the project base directory. This is an optional parameter; if you populate this, `--input-file`, `--output-path`, and `--test-data-path` are not required, and Hercules will assume all the three folders exist in the following format inside the project base:
```
PROJECT_BASE/
├── gherkin_files/
├── input/
│ └── test.feature
├── log_files/
├── output/
│ ├── test.feature_result.html
│ └── test.feature_result.xml
├── proofs/
│ └── User_opens_Google_homepage/
│ ├── network_logs.json
│ ├── screenshots/
│ └── videos/
└── test_data/
└── test_data.txt
```
- `--llm-model LLM_MODEL`: Name of the LLM model to be used by the agent (recommended is `gpt-4o`, but it can take others).
- `--llm-model-api-key LLM_MODEL_API_KEY`: API key for the LLM model, something like `sk-proj-k.......`.
#### Running Hercules
After passing all the required parameters, the command to run Hercules should look like this:
```bash
testzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key sk-proj-k.......
```
## ⚙️ Running Hercules on a Windows Machine
To set up and run Hercules on a Windows machine:
1. **Open PowerShell in Administrator Mode:**
- Click on the **Start Menu**, search for **PowerShell**, and right-click on **Windows PowerShell**.
- Select **Run as Administrator** to open PowerShell in administrator mode.
2. **Navigate to the Helper Scripts Folder:**
- Use the `cd` command to navigate to the folder containing the `hercules_windows_setup.ps1` script. For example:
```powershell
cd path\to\helper_scripts
```
3. **Run the Setup Script:**
- Execute the script to install and configure Hercules:
```powershell
.\hercules_windows_setup.ps1
```
4. **Follow On-Screen Instructions:**
- The script will guide you through installing Python, Playwright, FFmpeg, and other required dependencies.
5. **Run Hercules:**
- Once the setup is complete, you can run Hercules from PowerShell or Command Prompt using the following command:
```bash
testzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key sk-proj-k.......
```
---
#### Supported AI Models for TestZeus-Hercules
- Anthropic Haiku: Compatible with Haiku 3.5 and above.
- Groq: Supports any version with function calling and coding capabilities.
- Mistral: Supports any version with function calling and coding capabilities.
- OpenAI: Fully compatible with GPT-4o and above. Note: OpenAI GPT-4o-mini is not supported.
- Ollama: Not supported based on current testing.
#### Execution Flow
Upon running the command:
- Hercules will start and attempt to open a web browser (default is Chromium).
- It will prepare a plan of execution based on the feature file steps provided.
- The plan internally expands the brief steps mentioned in the feature file into a more elaborated version.
- Hercules detects assertions in the feature file and plans the validation of expected results with the execution happening during the test run.
- All the steps, once elaborated, are passed to different tools based on the type of execution requirement of the step. For example, if a step wants to click on a button and capture the feedback, it will be passed to the `click_using_selector` tool.
#### Output and Logs
Once the execution is completed:
- Logs explaining the sequence of events are generated.
- The best place to start is the `output-path`, which will have the JUnit XML result file as well as an HTML report regarding the test case execution.
- You can also find proofs of execution such as video recordings, screenshots per event, and network logs in the `proofs` folder.
- To delve deeper and understand the chain of thoughts, refer to the `chat_messages.json` in the `log_files`. This will have exact steps that were planned by the agent.
#### Sample Feature File
Here's a sample feature file:
```gherkin
Feature: Account Creation in Salesforce
Scenario: Successfully create a new account
Given I am on the Salesforce login page
When I enter my username "user@example.com" and password "securePassword"
And I click on the "Log In" button
And I navigate to the "Accounts" tab
And I click on the "New" button
And I fill in the "Account Name" field with "Test Account"
And I select the "Account Type" as "Customer"
And I fill in the "Website" field with "www.testaccount.com"
And I fill in the "Phone" field with "123-456-7890"
And I click on the "Save" button
Then I should see a confirmation message "Account Test Account created successfully"
And I should see "Test Account" listed in the account records
```
#### Sample Result Screenshot
##### Sample HTML Result Screenshot
##### Sample XML Result Screenshot
---
### Approach 2: Using Docker
For all the scale lovers, Hercules is also available as a Docker image.
#### Pull the Docker Image
```bash
docker pull testzeus/hercules:latest
```
#### Running Hercules in Docker
Run the container using:
```bash
docker run --env-file=.env \
-v ./agents_llm_config.json:/testzeus-hercules/agents_llm_config.json \
-v ./opt:/testzeus-hercules/opt \
--rm -it testzeus/hercules:latest
```
- **Environment Variables**: All the required environment variables can be set by passing an `.env` file to the `docker run` command.
- **LLM Configuration**: If you plan to have complete control over Hercules and which LLM to use beyond the ones provided by OpenAI, you can pass `agents_llm_config.json` as a mount to the container. This is for advanced use cases and is not required for beginners. Refer to sample files `.env-example` and `agents_llm_config-example.json` for details and reference.
- **Mounting Directories**: Mount the `opt` folder to the Docker container so that all the inputs can be passed to Hercules running inside the container, and the output can be pulled out for further processing. The repository has a sample `opt` folder that can be mounted easily.
- **Simplified Parameters**: In the Docker case, there is no need for using `--input-file`, `--output-path`, `--test-data-path`, or `--project-base` as they are already handled by mounting the `opt` folder in the `docker run` command.
#### Browser Access in Docker
- While running in Docker mode, understand that Hercules has access only to a headless web browser.
- If you want Hercules to connect to a visible web browser, try the CDP URL option in the environment file. This option can help you connect Hercules running in your infrastructure to a remote browser like BrowserBase or your self-hosted grid.
- Use `CDP_ENDPOINT_URL` to set the CDP URL of the Chrome instance that has to be connected to the agent.
#### Output and Logs
After the command completion:
- The container terminates, and output is written in the mounted `opt` folder, in the same way as described in the directory structure.
- You will find the JUnit XML result file, HTML reports, proofs of execution, and logs in the respective folders.
---
### Approach 3: Building and running from Source Code
For the hardcore enthusiasts, you can use Hercules via the source code to get a complete experience of customization and extending Hercules with more tools.
#### Prerequisites
- Ensure you have **Python 3.11** installed on your system.
#### Steps to Run from Source
1. **Clone the Repository**
```bash
git clone git@github.com:test-zeus-ai/testzeus-hercules.git
```
2. **Navigate to the Directory**
```bash
cd testzeus-hercules
```
3. **Use Make Commands**
The repository provides handy `make` commands.
- Use `make help` to check out possible options.
4. **Install Poetry**
```bash
make setup-poetry
```
5. **Install Dependencies**
```bash
make install
```
6. **Run Hercules**
```bash
make run
```
- This command reads the relevant feature files from the `opt` folder and executes them, putting the output in the same folder.
- The `opt` folder has the following format:
```
opt/
├── input/
│ └── test.feature
├── output/
│ ├── test.feature_result.html
│ └── test.feature_result.xml
├── log_files/
├── proofs/
│ └── User_opens_Google_homepage/
│ ├── network_logs.json
│ ├── screenshots/
│ └── videos/
└── test_data/
└── test_data.txt
```
7. **Interactive Mode**
You can also run Hercules in interactive mode as an instruction execution agent, which is more useful for RPA and debugging test cases and Hercules's behavior on new environments while building new tooling and extending the agents.
```bash
make run-interactive
```
- This will trigger an input prompt where you can chat with Hercules, and it will perform actions based on your commands.
---
## 📝 Configuration Details
### Disabling Telemetry
To disable telemetry, set the `TELEMETRY_ENABLED` environment variable to `0`:
```bash
export TELEMETRY_ENABLED=0
```
### Auto Mode
If `AUTO_MODE` is set to `1`, Hercules will not request an email during the run:
```bash
export AUTO_MODE=1
```
### Understanding the Environment File (`.env`)
To configure Hercules in detail:
- Copy the base environment file:
```bash
cp .env-example .env
```
- Hercules is capable of running in two configuration forms:
1. **Using single LLM for all work**
- For all the activities within the agent, initialize `LLM_MODEL_NAME` and `LLM_MODEL_API_KEY`.
- If using a non-OpenAI hosted solution but still OpenAI LLMs (something like OpenAI via Groq), then pass the `LLM_MODEL_BASE_URL` URL as well.
2. **Custom LLMs for different work or using hosted LLMs**
- If you plan to configure local LLMs or non-OpenAI LLMs, use the other parameters like `AGENTS_LLM_CONFIG_FILE` and `AGENTS_LLM_CONFIG_FILE_REF_KEY`.
- These are powerful options and can affect the quality of Hercules outputs.
- Hercules considers a base folder that is by default `./opt` but can be changed by the environment variable `PROJECT_SOURCE_ROOT`.
- **Connecting to an Existing Chrome Instance**
- This is extremely useful when you are running Hercules in Docker for scale.
- You can connect Hercules running in your infrastructure to a remote browser like BrowserBase or your self-hosted grid.
- Use `CDP_ENDPOINT_URL` to set the CDP URL of the Chrome instance that has to be connected to the agent.
- **Controlling Other Behaviors**
You can control other behaviors of Hercules based on the following environment variables:
- `HEADLESS=true`
- `RECORD_VIDEO=false`
- `TAKE_SCREENSHOTS=false`
- `BROWSER_TYPE=chromium` (options: `firefox`, `chromium`)
- `CAPTURE_NETWORK=false`
For example: If you would like to run with a "Headful" browser, you can set the environment variable with ```export HEADLESS=false``` before triggering Hercules.
### Understanding `agents_llm_config-example.json`
- It's a list of configurations of LLMs that you want to provide to the agent.
- Example:
```json
{
"mistral-large": {
"planner_agent": {
"model_name": "mistral",
"model_api_key": "",
"model_base_url": "https://...",
"system_prompt": "You are a web automation task planner....",
"llm_config_params": {
"cache_seed": null,
"temperature": 0.1,
"top_p": 0.1
}
},
"browser_nav_agent": {
"model_name": "mistral",
"model_api_key": "",
"model_base_url": "https://...",
"system_prompt": "You will perform web navigation tasks with the functions that you have...\nOnce a task is completed, confirm completion with ##TERMINATE TASK##.",
"llm_config_params": {
"cache_seed": null,
"temperature": 0.1,
"top_p": 0.1
}
}
}
}
```
- The key is the name of the spec that is passed in `AGENTS_LLM_CONFIG_FILE_REF_KEY`, whereas the Hercules information is passed in sub-dicts `planner_agent` and `browser_nav_agent`.
- **Note**: This option should be ignored until you are sure what you are doing. Discuss with us while playing around with these options in our Slack communication. Join us at our [Slack](https://join.slack.com/t/testzeuscommunityhq/shared_invite/zt-2v2br8wog-FAmo_76xRHx~k~1oNaGQ0Q)
---
## ⚡️ Features
Hercules is production ready, and packs a punch with features:
### Gherkin In, Results Out
Hercules makes testing as simple as Gherkin in, results out. Just feed your end-to-end tests in Gherkin format, and watch Hercules spring into action. It takes care of the heavy lifting by running your tests automatically and presenting results in a neat JUnit format. No manual steps, no fuss—just efficient, seamless testing.
### Free and Open Source
With Hercules, you're harnessing the power of open source with zero licensing fees. Feel free to dive into the code, contribute, or customize it to your heart's content. Hercules is as free as it is mighty, giving you the flexibility and control you need.
### Salesforce Ready
Built to handle the most intricate UIs, Hercules conquers Salesforce and other complex platforms with ease. Whether it's complicated DOM or running your SOQL or Apex, Hercules is ready and configurable.
### No Code Required
Say goodbye to complex scripts and elusive locators. Hercules is here to make your life easier with its no-code approach, taking care of the automation of Gherkin features so you can focus on what matters most—building quality software.
### Multilingual
With multilingual support right out of the box, Hercules is ready to work with teams across the globe. Built to bridge language gaps, it empowers diverse teams to collaborate effortlessly on a unified testing platform.
### Precisely Accurate
Hercules records video of the execution, and captures network logs as well, so that you dont have to deal with "It works on my computer".
### No Maintenance
Autonomous and adaptive, Hercules takes care of itself with auto-healing capabilities. Forget about tedious maintenance—Hercules adjusts to changes and stays focused on achieving your testing goals.
### UI Assertions
Grounded in the powerful foundations of TestZeus, Hercules tackles UI assertions with unwavering focus, ensuring that no assertion goes unchecked and no bug goes unnoticed. It's thorough, it's sharp, and it's ready for action.
### CI/CD Ready
Run Hercules locally or integrate it seamlessly into your CI/CD pipeline. Docker-native and one-command ready, Hercules fits smoothly into your deployment workflows, keeping testing quick, consistent, and hassle-free.
With Hercules, testing is no longer just a step in the process—it's a powerful, streamlined experience that brings quality to the forefront.
---
## 🦾 Architecture
### Multi-Agentic Solution
Hercules leverages a multi-agent architecture based on the AutoGen framework. Building on the foundation provided by the AutoGen framework, Hercules's architecture leverages the interplay between tools and agents. Each tool embodies an atomic action, a fundamental building block that, when executed, returns a natural language description of its outcome. This granularity allows Hercules to flexibly assemble these tools to tackle complex web automation workflows.
#### System View
The diagram above shows the configuration chosen on top of AutoGen architecture. The tools can be partitioned differently, but this is the one that we chose for the time being. We chose to use tools that map to what humans learn about the web browser rather than allow the LLM to write code as it pleases. We see the use of configured tools to be safer and more predictable in its outcomes. Certainly, it can click on the wrong things, but at least it is not going to execute malicious unknown code.
#### Agents
At the moment, there are two agents:
1. **Planner Agent**: Executes the planning and decomposition of tasks.
2. **Browser Navigation Agent**: Embodies all the tools for interacting with the web browser.
#### Tools Library
At the core of Hercules's capabilities is the Tools Library, a repository of well-defined actions that Hercules can perform; for now, web actions. These tools are grouped into two main categories:
- **Sensing Tools**: Tools like `get_dom_with_content_type` and `geturl` that help Hercules understand the current state of the webpage or the browser.
- **Action Tools**: Tools that allow Hercules to interact with and manipulate the web environment, such as `click`, `enter_text`, and `openurl`.
Each tool is created with the intention to be as conversational as possible, making the interactions with LLMs more intuitive and error-tolerant. For instance, rather than simply returning a boolean value, a tool might explain in natural language what happened during its execution, enabling the LLM to better understand the context and correct course if necessary.
##### Implemented Tools
- **Sensing Tools**
- `geturl`: Fetches and returns the current URL.
- `get_dom_with_content_type`: Retrieves the HTML DOM of the active page based on the specified content type.
- `text_only`: Extracts the inner text of the HTML DOM. Responds with text output.
- `input_fields`: Extracts the interactive elements in the DOM (button, input, textarea, etc.) and responds with a compact JSON object.
- `all_fields`: Extracts all the fields in the DOM and responds with a compact JSON object.
- `get_user_input`: Provides the orchestrator with a mechanism to receive user feedback to disambiguate or seek clarity on fulfilling their request.
- **Action Tools**
- `click`: Given a DOM query selector, this will click on it.
- `enter_text`: Enters text in a field specified by the provided DOM query selector.
- `enter_text_and_click`: Optimized method that combines `enter_text` and `click` tools.
- `bulk_enter_text`: Optimized method that wraps `enter_text` method so that multiple text entries can be performed in one shot.
- `openurl`: Opens the given URL in the current or new tab.
#### DOM Distillation
Hercules's approach to managing the vast landscape of HTML DOM is methodical and essential for efficiency. We've introduced **DOM Distillation** to pare down the DOM to just the elements pertinent to the user's task.
In practice, this means taking the expansive DOM and delivering a more digestible JSON snapshot. This isn't about just reducing size; it's about honing in on relevance, serving the LLMs only what's necessary to fulfill a request. So far, we have three content types:
1. **Text Only**: For when the mission is information retrieval, and the text is the target. No distractions.
2. **Input Fields**: Zeroing in on elements that call for user interaction. It's about streamlining actions.
3. **All Content**: The full scope of distilled DOM, encompassing all elements when the task demands a comprehensive understanding.
It's a surgical procedure, carefully removing extraneous information while preserving the structure and content needed for the agent's operation. Of course, with any distillation, there could be casualties, but the idea is to refine this over time to limit/eliminate them.
Since we can't rely on all web page authors to use best practices, such as adding unique IDs to each HTML element, we had to inject our own attribute (`mmid`) in every DOM element. We can then guide the LLM to rely on using `mmid` in the generated DOM queries.
To cut down on some of the DOM noise, we use the **DOM Accessibility Tree** rather than the regular HTML DOM. The accessibility tree, by nature, is geared towards helping screen readers, which is closer to the mission of web automation than plain old HTML DOM.
The distillation process is a work in progress. We look to refine this process and condense the DOM further, aiming to make interactions faster, cost-effective, and more accurate.
---
## 🔬 Testing and Evaluation: QEvals
We wanted to ensure that Hercules stands up to the task of end-to-end testing with immense precision. So, we have run Hercules through a wide range of tests such as running APIs, interacting with complex UI scenarios, clicking through calendars, or iframes. A full list of evaluations can be found in the [tests folder](<Link to tests folder>).
### Running Tests
To run the full test suite, use the following command:
```bash
make test
```
To run a specific test:
```bash
make test-case
```
Hercules builds on the work done by WebArena and Agent-E, and beyond that, to iron out the issues in the previous, we have written our own test cases catering to complex QA scenarios and have created tests in the `./tests` folder.
---
## 💡 Opinions
We believe that great quality comes from opinions about a product. So we have incorporated a few of our opinions into the product design. We welcome the community to question them, use them, or build on top of them. Here are some examples:
1. **Gherkin is a Good Enough Format for Agents**: Gherkin provides a semi-structured format for the LLMs/AI Agents to follow test intent and user instructions. It provides the right amount of grammar (verbs like Given, When, Then) for humans to frame a scenario and agents to follow the instructions.
2. **Tests Should Be Atomic in Nature**: Software tests should be atomic because it ensures that each test is **focused, independent, and reliable**. Atomic tests target one specific behavior or functionality, making it easier to pinpoint the root cause of failures without ambiguity.
**Here's a good example (Atomic Test)**:
```gherkin
Feature: User Login
Scenario: Successful login with valid credentials
Given the user is on the login page
When the user enters valid credentials
And the user clicks the login button
Then the user should see the dashboard
```
A non-atomic test confuses both the tester and the AI agent.
3. **Open Core and Open Source**: Hercules is built on an **open-core model**, combining the spirit of open source with the support and expertise of a commercial company, **TestZeus**. By providing Hercules as open source (licensed under AGPL v3), TestZeus is committed to empowering the testing community with a robust, adaptable tool that's freely accessible and modifiable. Open source offers transparency, trust, and collaborative potential, allowing anyone to dive into the code, contribute, and shape the project's direction.
4. **Telemetry** : All great products are built on good feedback. We have setup telemetry so that we can take feedback, without disturbing the user. Telemetry is enabled by default, but we also believe strongly in the values of "Trust" and "Transparency" so it can be turned off by the users.
---
## 🪙 Token Usage
Hercules is an AI-native solution and relies on LLMs to perform reasoning and actions. Based on our experiments, we have found that a complex use case as below could cost up to **$0.20** using OpenAI's APIs gpt-4o, check the properties printed in testcase output to calculate for your testcase:
```gherkin
Feature: Account Creation in Salesforce
Scenario: Successfully create a new account
Given I am on the Salesforce login page
When I enter my username "user@example.com" and password "securePassword"
And I click on the "Log In" button
And I navigate to the "Accounts" tab
And I click on the "New" button
And I fill in the "Account Name" field with "Test Account"
And I select the "Account Type" as "Customer"
And I fill in the "Website" field with "www.testaccount.com"
And I fill in the "Phone" field with "123-456-7890"
And I click on the "Save" button
Then I should see a confirmation message "Account Test Account created successfully"
And I should see "Test Account" listed in the account records
```
---
## ⭐️ Difference from Other Tools
Hercules isn't just another testing tool—it's an **agent**. Powered by synthetic intelligence that can **think, reason, and react** based on requirements, Hercules goes beyond simple automation scripts. We bring an industry-first approach to open-source agents for **software testing**. This means faster, smarter, and more resilient testing cycles, especially for complex platforms.
With **industry-leading performance** and a fully open-source foundation, Hercules combines powerful capabilities with community-driven flexibility, making top-tier testing accessible and transformative for everyone.
---
## 🚀 High-Level Roadmap
- **Enhanced LLM Support**: Integration with more LLMs and support for local LLM deployments.
- **Advanced Tooling**: Addition of more tools to handle complex testing scenarios and environments.
- **Improved DOM Distillation**: Refinements to the DOM distillation process for better efficiency and accuracy.
- **Community Contributions**: Encourage and integrate community-driven features and tools.
- **Extensive Documentation**: Expand documentation for better onboarding and usage.
- **Bounty Program**: Launch a bounty program to incentivize contributions.
---
## 🤗 Contribution
We welcome contributions from the community!
- **Read the [CONTRIBUTING.md](CONTRIBUTING.md) file** to get started.
- **Bounty Program**: Stay tuned for upcoming opportunities! 😀
### How to Contribute Back
1. **Developing Tools**
- If you are developing tools for Hercules and want to contribute to the community, make sure you place the new tools in the `additional_tools` folder in your Pull Request.
2. **Fixes and Enhancements**
- If you have a fix on sensing tools that are fundamental to the system or something in prompts or something in the DOM distillation code, then put the changes in the relevant file and share the Pull Request.
### Extending and Attaching Tools
1. **Creating a New Tool**
- You can start extending by adding tools to Hercules.
- Refer to `testzeus_hercules/core/tools/sql_calls.py` as an example of how to create a new tool.
- The key is the decorator `@tool` over the method that you want Hercules to execute.
- The tool decorator should have a very clear description and name so that Hercules knows how to use the tool.
- Also, in the method, you should be clear with annotations on what parameter is used for what purpose so that function calling in the LLM works best.
2. **Adding the Tool**
- Once you have created the new tools files in some folder path, you can pass the folder path to Hercules in the environment variable so that Hercules can read the new tools during the boot time and make sure that they are available during the execution.
- Use `ADDITIONAL_TOOL_DIRS` to pass the path of the new tools folder where you have kept the new files.
3. **Direct Addition (Not Recommended)**
- In case you opt for adding the tools directly, then just put your new tools in the `testzeus_hercules/core/tools` path of the cloned repository.
- Then make sure you import your tool module in the `testzeus_hercules/core/agents/browser_nav_agent.py` file as:
```python
from testzeus_hercules.core.tools.sql_calls import *
```
- **Note**: This way is not recommended. We prefer you try to use the `ADDITIONAL_TOOL_DIRS` approach.
---
## 🤙 Contact Us
Join us at our [Slack](https://join.slack.com/t/testzeuscommunityhq/shared_invite/zt-2v2br8wog-FAmo_76xRHx~k~1oNaGQ0Q) to connect with the community, ask questions, and contribute.
---
## ✍️ Examples
- **Salesforce Examples**: [Link](tests/test_features/ebikes/ebikes.feature)
- **Wrangler Example**: [Link](tests/test_features/productSearch/productSearch.feature)
---
## 🙇 Credits
Hercules would not have been possible without the great work from the following sources:
1. [Agent-E](https://arxiv.org/abs/2407.13032)
2. [Q*](https://arxiv.org/abs/2312.10868)
3. [Agent Q](https://arxiv.org/abs/2408.07199)
4. [Autogen](https://arxiv.org/pdf/2308.08155)
The Hercules project is inferred and enhanced over the existing project of [Agent-E](https://github.com/EmergenceAI/Agent-E). We have improved lots of cases to make it capable of doing testing, especially in the area of complex DOM navigation and iframes. We have also added new tools and abilities (like Salesforce navigation) to Hercules so that it can perform better work over the base framework we had picked.
Hercules also picks some inspiration from the legacy TestZeus repo [here](https://www.testzeus.org).
---
With Hercules, testing is no longer just a step in the process—it's a powerful, streamlined experience that brings quality to the forefront.
Raw data
{
"_id": null,
"home_page": "https://testzeus.com/hercules",
"name": "testzeus-hercules",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.11",
"maintainer_email": null,
"keywords": "AI, end-to-end testing, automation, Hercules, TestZeus",
"author": "Shriyansh Agnihotri",
"author_email": "shriyansh@testzeus.com",
"download_url": "https://files.pythonhosted.org/packages/f0/5e/925ba4354ec8358d9852f2bf74a7a0ce0b22c6f904f65f2257f28f50b24f/testzeus_hercules-0.0.15.tar.gz",
"platform": null,
"description": "# \ud83d\udcaa Hercules\n\n[](https://github.com/test-zeus-ai/testzeus-hercules/actions/workflows/main-test.yml)\n\nTesting modern web applications can be difficult, with frequent changes and complex features making it hard to keep up. That\u2019s where **Hercules** comes in. Hercules is the world\u2019s first open-source testing agent, built to handle the toughest testing tasks so you don\u2019t have to. It turns simple, easy-to-write Gherkin steps into fully automated **end to end** tests\u2014no coding skills needed. Whether you\u2019re working with tricky platforms like Salesforce or running tests in your CI/CD pipeline, Hercules adapts to your needs and takes care of the details. With Hercules, testing becomes simple, reliable, and efficient, helping teams everywhere deliver better software.\n\nUsing Hercules is as simple as feeding in your Gherkin features, and getting the results:\n\n\n\nAt [TestZeus](www.testzeus.com), we believe that **trustworthy and open-source code** is the backbone of innovation. That's why we've built Hercules to be transparent, reliable, and community-driven.\n\nOur mission? To **democratize and disrupt test automation**, making top-tier testing accessible to everyone, not just the elite few. No more gatekeeping\u2014everyone deserves a hero on their testing team!\n\n### Video Tutorials: [@TestZeus](https://www.youtube.com/@TestZeus)\n---\n\n#### \ud83d\ude80 **Getting Started with TestZeus Hercules**\n- **Introduction to TestZeus Hercules** \n Learn about the core features of TestZeus Hercules and how it can streamline end-to-end testing for your projects. \n _[Watch now](https://youtu.be/_m_NDjM6aZ0?si=ArtVKz8uSgGWTcAK)_\n\n- **Installation and Setup Guide** \n Step-by-step instructions for installing and configuring TestZeus Hercules in your environment. \n _[Watch now](https://youtu.be/9D-SZGoDrfc?si=GL0IArWkB1ZgBdx5)_\n\n---\n\n#### \ud83e\uddea **Writing and Executing Test Cases**\n- **Creating BDD Test Cases** \n Learn how to write Behavior-Driven Development (BDD) test cases for Hercules and use dynamic testdata. \n _[Watch now](https://www.youtube.com/watch?v=yJD0cZ7Bx6Q&t)_\n\n---\n\n#### \ud83c\udf10 **Integrations and Advanced Features**\n- **Testing Multilingual content** \n Learn how Hercules interacts with web browsers to perform Testing on Multilingual content via Multilingual testcase. \n _[Watch now](https://youtu.be/vI9iGSqKpGA?si=6NGAvKnwFboSyHT2)_\n---\n\n#### \ud83d\udee0\ufe0f **Customization and Community Tools**\n- **Enhancing Hercules with Community-Driven Tools** \n Discover how to customize Hercules and incorporate additional tools provided by the community. \n _[Watch now](https://youtu.be/C8hUy5leRF0?si=yVMpZ7WFcI01BoI3)_\n---\n\n#### \ud83d\udee0\ufe0f **API Testing**\n- **API testing all the way, new ways to do end to end** \n _[Watch now](https://youtu.be/qMt89DQH6LQ?si=lcSJbKOCaqqUGeQ8)_\n---\n\n#### \ud83d\udee0\ufe0f **Security Testing**\n- **Security Testing done end to end** \n _[Watch now](https://youtu.be/o6MmfTakIh4?si=JL8iu4e3i85SWrxU)_\n\n---\n\n## \u2699\ufe0f Installation and Usage\n\nHercules offers multiple ways to get started, catering to different user preferences and requirements.\n\n\nFor a quick taste of the solution, you can try the notebook here: \n[](https://colab.research.google.com/drive/1YiZsXem1POTwkcr17QqflXnihhuSqwM2?usp=sharing)\n- **Note**: Colab might ask you to restart the session as python3.11 and some libs are installed during the installation of testzeus-hercules. Please restart the session if required and continue the execution. Also , we recommend one of the approaches below for getting the full flavor of the solution. \n\n### Approach 1: Using PyPI Package\n\n#### Installation\n\nInstall Hercules from PyPI:\n\n```bash\npip install testzeus-hercules\n```\n\nHercules uses Playwright to interact with web pages, so you need to install Playwright and its dependencies:\n\n```bash\nplaywright install --with-deps\n```\n\n#### Basic Parameters\n\nOnce installed, you will need to provide some basic parameters to run Hercules:\n\n- `--input-file INPUT_FILE`: Path to the input Gherkin feature file to be tested.\n- `--output-path OUTPUT_PATH`: Path to the output directory. The path of JUnit XML result and HTML report for the test run.\n- `--test-data-path TEST_DATA_PATH`: Path to the test data directory. The path where Hercules expects test data to be present; all test data used in feature testing should be present here.\n- `--project-base PROJECT_BASE`: Path to the project base directory. This is an optional parameter; if you populate this, `--input-file`, `--output-path`, and `--test-data-path` are not required, and Hercules will assume all the three folders exist in the following format inside the project base:\n\n```\nPROJECT_BASE/\n\u251c\u2500\u2500 gherkin_files/\n\u251c\u2500\u2500 input/\n\u2502 \u2514\u2500\u2500 test.feature\n\u251c\u2500\u2500 log_files/\n\u251c\u2500\u2500 output/\n\u2502 \u251c\u2500\u2500 test.feature_result.html\n\u2502 \u2514\u2500\u2500 test.feature_result.xml\n\u251c\u2500\u2500 proofs/\n\u2502 \u2514\u2500\u2500 User_opens_Google_homepage/\n\u2502 \u251c\u2500\u2500 network_logs.json\n\u2502 \u251c\u2500\u2500 screenshots/\n\u2502 \u2514\u2500\u2500 videos/\n\u2514\u2500\u2500 test_data/\n \u2514\u2500\u2500 test_data.txt\n```\n\n- `--llm-model LLM_MODEL`: Name of the LLM model to be used by the agent (recommended is `gpt-4o`, but it can take others).\n- `--llm-model-api-key LLM_MODEL_API_KEY`: API key for the LLM model, something like `sk-proj-k.......`.\n\n\n#### Running Hercules\n\nAfter passing all the required parameters, the command to run Hercules should look like this:\n\n```bash\ntestzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key sk-proj-k.......\n```\n\n\n## \u2699\ufe0f Running Hercules on a Windows Machine\n\nTo set up and run Hercules on a Windows machine:\n\n1. **Open PowerShell in Administrator Mode:**\n - Click on the **Start Menu**, search for **PowerShell**, and right-click on **Windows PowerShell**.\n - Select **Run as Administrator** to open PowerShell in administrator mode.\n\n2. **Navigate to the Helper Scripts Folder:**\n - Use the `cd` command to navigate to the folder containing the `hercules_windows_setup.ps1` script. For example:\n ```powershell\n cd path\\to\\helper_scripts\n ```\n\n3. **Run the Setup Script:**\n - Execute the script to install and configure Hercules:\n ```powershell\n .\\hercules_windows_setup.ps1\n ```\n\n4. **Follow On-Screen Instructions:**\n - The script will guide you through installing Python, Playwright, FFmpeg, and other required dependencies.\n\n5. **Run Hercules:**\n - Once the setup is complete, you can run Hercules from PowerShell or Command Prompt using the following command:\n ```bash\n testzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key sk-proj-k.......\n ```\n\n---\n\n\n#### Supported AI Models for TestZeus-Hercules\n- Anthropic Haiku: Compatible with Haiku 3.5 and above.\n- Groq: Supports any version with function calling and coding capabilities.\n- Mistral: Supports any version with function calling and coding capabilities.\n- OpenAI: Fully compatible with GPT-4o and above. Note: OpenAI GPT-4o-mini is not supported.\n- Ollama: Not supported based on current testing.\n\n#### Execution Flow\n\nUpon running the command:\n\n- Hercules will start and attempt to open a web browser (default is Chromium).\n- It will prepare a plan of execution based on the feature file steps provided.\n- The plan internally expands the brief steps mentioned in the feature file into a more elaborated version.\n- Hercules detects assertions in the feature file and plans the validation of expected results with the execution happening during the test run.\n- All the steps, once elaborated, are passed to different tools based on the type of execution requirement of the step. For example, if a step wants to click on a button and capture the feedback, it will be passed to the `click_using_selector` tool.\n\n#### Output and Logs\n\nOnce the execution is completed:\n\n- Logs explaining the sequence of events are generated.\n- The best place to start is the `output-path`, which will have the JUnit XML result file as well as an HTML report regarding the test case execution.\n- You can also find proofs of execution such as video recordings, screenshots per event, and network logs in the `proofs` folder.\n- To delve deeper and understand the chain of thoughts, refer to the `chat_messages.json` in the `log_files`. This will have exact steps that were planned by the agent.\n\n#### Sample Feature File\n\nHere's a sample feature file:\n\n```gherkin\nFeature: Account Creation in Salesforce\n\n Scenario: Successfully create a new account\n\n Given I am on the Salesforce login page\n When I enter my username \"user@example.com\" and password \"securePassword\"\n And I click on the \"Log In\" button\n And I navigate to the \"Accounts\" tab\n And I click on the \"New\" button\n And I fill in the \"Account Name\" field with \"Test Account\"\n And I select the \"Account Type\" as \"Customer\"\n And I fill in the \"Website\" field with \"www.testaccount.com\"\n And I fill in the \"Phone\" field with \"123-456-7890\"\n And I click on the \"Save\" button\n Then I should see a confirmation message \"Account Test Account created successfully\"\n And I should see \"Test Account\" listed in the account records\n```\n\n#### Sample Result Screenshot\n\n##### Sample HTML Result Screenshot\n\n##### Sample XML Result Screenshot\n\n\n---\n\n### Approach 2: Using Docker\n\nFor all the scale lovers, Hercules is also available as a Docker image.\n\n#### Pull the Docker Image\n\n```bash\ndocker pull testzeus/hercules:latest\n```\n\n#### Running Hercules in Docker\n\nRun the container using:\n\n```bash\ndocker run --env-file=.env \\\n -v ./agents_llm_config.json:/testzeus-hercules/agents_llm_config.json \\\n -v ./opt:/testzeus-hercules/opt \\\n --rm -it testzeus/hercules:latest\n```\n\n- **Environment Variables**: All the required environment variables can be set by passing an `.env` file to the `docker run` command.\n- **LLM Configuration**: If you plan to have complete control over Hercules and which LLM to use beyond the ones provided by OpenAI, you can pass `agents_llm_config.json` as a mount to the container. This is for advanced use cases and is not required for beginners. Refer to sample files `.env-example` and `agents_llm_config-example.json` for details and reference.\n- **Mounting Directories**: Mount the `opt` folder to the Docker container so that all the inputs can be passed to Hercules running inside the container, and the output can be pulled out for further processing. The repository has a sample `opt` folder that can be mounted easily.\n- **Simplified Parameters**: In the Docker case, there is no need for using `--input-file`, `--output-path`, `--test-data-path`, or `--project-base` as they are already handled by mounting the `opt` folder in the `docker run` command.\n\n#### Browser Access in Docker\n\n- While running in Docker mode, understand that Hercules has access only to a headless web browser.\n- If you want Hercules to connect to a visible web browser, try the CDP URL option in the environment file. This option can help you connect Hercules running in your infrastructure to a remote browser like BrowserBase or your self-hosted grid.\n- Use `CDP_ENDPOINT_URL` to set the CDP URL of the Chrome instance that has to be connected to the agent.\n\n#### Output and Logs\n\nAfter the command completion:\n\n- The container terminates, and output is written in the mounted `opt` folder, in the same way as described in the directory structure.\n- You will find the JUnit XML result file, HTML reports, proofs of execution, and logs in the respective folders.\n\n---\n\n### Approach 3: Building and running from Source Code\n\nFor the hardcore enthusiasts, you can use Hercules via the source code to get a complete experience of customization and extending Hercules with more tools.\n\n#### Prerequisites\n\n- Ensure you have **Python 3.11** installed on your system.\n\n#### Steps to Run from Source\n\n1. **Clone the Repository**\n\n ```bash\n git clone git@github.com:test-zeus-ai/testzeus-hercules.git\n ```\n\n2. **Navigate to the Directory**\n\n ```bash\n cd testzeus-hercules\n ```\n\n3. **Use Make Commands**\n\n The repository provides handy `make` commands.\n\n - Use `make help` to check out possible options.\n\n4. **Install Poetry**\n\n ```bash\n make setup-poetry\n ```\n\n5. **Install Dependencies**\n\n ```bash\n make install\n ```\n\n6. **Run Hercules**\n\n ```bash\n make run\n ```\n\n - This command reads the relevant feature files from the `opt` folder and executes them, putting the output in the same folder.\n - The `opt` folder has the following format:\n\n ```\n opt/\n \u251c\u2500\u2500 input/\n \u2502 \u2514\u2500\u2500 test.feature\n \u251c\u2500\u2500 output/\n \u2502 \u251c\u2500\u2500 test.feature_result.html\n \u2502 \u2514\u2500\u2500 test.feature_result.xml\n \u251c\u2500\u2500 log_files/\n \u251c\u2500\u2500 proofs/\n \u2502 \u2514\u2500\u2500 User_opens_Google_homepage/\n \u2502 \u251c\u2500\u2500 network_logs.json\n \u2502 \u251c\u2500\u2500 screenshots/\n \u2502 \u2514\u2500\u2500 videos/\n \u2514\u2500\u2500 test_data/\n \u2514\u2500\u2500 test_data.txt\n ```\n\n7. **Interactive Mode**\n\n You can also run Hercules in interactive mode as an instruction execution agent, which is more useful for RPA and debugging test cases and Hercules's behavior on new environments while building new tooling and extending the agents.\n\n ```bash\n make run-interactive\n ```\n\n - This will trigger an input prompt where you can chat with Hercules, and it will perform actions based on your commands.\n\n---\n\n## \ud83d\udcdd Configuration Details\n\n### Disabling Telemetry\n\nTo disable telemetry, set the `TELEMETRY_ENABLED` environment variable to `0`:\n\n```bash\nexport TELEMETRY_ENABLED=0\n```\n\n### Auto Mode\n\nIf `AUTO_MODE` is set to `1`, Hercules will not request an email during the run:\n\n```bash\nexport AUTO_MODE=1\n```\n\n### Understanding the Environment File (`.env`)\n\nTo configure Hercules in detail:\n\n- Copy the base environment file:\n\n ```bash\n cp .env-example .env\n ```\n\n- Hercules is capable of running in two configuration forms:\n\n 1. **Using single LLM for all work**\n\n - For all the activities within the agent, initialize `LLM_MODEL_NAME` and `LLM_MODEL_API_KEY`.\n - If using a non-OpenAI hosted solution but still OpenAI LLMs (something like OpenAI via Groq), then pass the `LLM_MODEL_BASE_URL` URL as well.\n\n 2. **Custom LLMs for different work or using hosted LLMs**\n\n - If you plan to configure local LLMs or non-OpenAI LLMs, use the other parameters like `AGENTS_LLM_CONFIG_FILE` and `AGENTS_LLM_CONFIG_FILE_REF_KEY`.\n - These are powerful options and can affect the quality of Hercules outputs.\n\n- Hercules considers a base folder that is by default `./opt` but can be changed by the environment variable `PROJECT_SOURCE_ROOT`.\n\n- **Connecting to an Existing Chrome Instance**\n\n - This is extremely useful when you are running Hercules in Docker for scale.\n - You can connect Hercules running in your infrastructure to a remote browser like BrowserBase or your self-hosted grid.\n - Use `CDP_ENDPOINT_URL` to set the CDP URL of the Chrome instance that has to be connected to the agent.\n\n- **Controlling Other Behaviors**\n\n You can control other behaviors of Hercules based on the following environment variables:\n\n - `HEADLESS=true`\n - `RECORD_VIDEO=false`\n - `TAKE_SCREENSHOTS=false`\n - `BROWSER_TYPE=chromium` (options: `firefox`, `chromium`)\n - `CAPTURE_NETWORK=false`\n \nFor example: If you would like to run with a \"Headful\" browser, you can set the environment variable with ```export HEADLESS=false``` before triggering Hercules.\n\n\n### Understanding `agents_llm_config-example.json`\n\n- It's a list of configurations of LLMs that you want to provide to the agent.\n\n- Example:\n\n ```json\n {\n \"mistral-large\": {\n \"planner_agent\": {\n \"model_name\": \"mistral\",\n \"model_api_key\": \"\",\n \"model_base_url\": \"https://...\",\n \"system_prompt\": \"You are a web automation task planner....\",\n \"llm_config_params\": {\n \"cache_seed\": null,\n \"temperature\": 0.1,\n \"top_p\": 0.1\n }\n },\n \"browser_nav_agent\": {\n \"model_name\": \"mistral\",\n \"model_api_key\": \"\",\n \"model_base_url\": \"https://...\",\n \"system_prompt\": \"You will perform web navigation tasks with the functions that you have...\\nOnce a task is completed, confirm completion with ##TERMINATE TASK##.\",\n \"llm_config_params\": {\n \"cache_seed\": null,\n \"temperature\": 0.1,\n \"top_p\": 0.1\n }\n }\n }\n }\n ```\n\n- The key is the name of the spec that is passed in `AGENTS_LLM_CONFIG_FILE_REF_KEY`, whereas the Hercules information is passed in sub-dicts `planner_agent` and `browser_nav_agent`.\n\n- **Note**: This option should be ignored until you are sure what you are doing. Discuss with us while playing around with these options in our Slack communication. Join us at our [Slack](https://join.slack.com/t/testzeuscommunityhq/shared_invite/zt-2v2br8wog-FAmo_76xRHx~k~1oNaGQ0Q)\n\n---\n\n\n## \u26a1\ufe0f Features\n\nHercules is production ready, and packs a punch with features:\n\n### Gherkin In, Results Out\n\nHercules makes testing as simple as Gherkin in, results out. Just feed your end-to-end tests in Gherkin format, and watch Hercules spring into action. It takes care of the heavy lifting by running your tests automatically and presenting results in a neat JUnit format. No manual steps, no fuss\u2014just efficient, seamless testing.\n\n### Free and Open Source\n\nWith Hercules, you're harnessing the power of open source with zero licensing fees. Feel free to dive into the code, contribute, or customize it to your heart's content. Hercules is as free as it is mighty, giving you the flexibility and control you need.\n\n### Salesforce Ready\n\nBuilt to handle the most intricate UIs, Hercules conquers Salesforce and other complex platforms with ease. Whether it's complicated DOM or running your SOQL or Apex, Hercules is ready and configurable.\n\n### No Code Required\n\nSay goodbye to complex scripts and elusive locators. Hercules is here to make your life easier with its no-code approach, taking care of the automation of Gherkin features so you can focus on what matters most\u2014building quality software.\n\n### Multilingual\n\nWith multilingual support right out of the box, Hercules is ready to work with teams across the globe. Built to bridge language gaps, it empowers diverse teams to collaborate effortlessly on a unified testing platform.\n\n### Precisely Accurate\n\nHercules records video of the execution, and captures network logs as well, so that you dont have to deal with \"It works on my computer\".\n\n\n### No Maintenance\n\nAutonomous and adaptive, Hercules takes care of itself with auto-healing capabilities. Forget about tedious maintenance\u2014Hercules adjusts to changes and stays focused on achieving your testing goals.\n\n### UI Assertions\n\nGrounded in the powerful foundations of TestZeus, Hercules tackles UI assertions with unwavering focus, ensuring that no assertion goes unchecked and no bug goes unnoticed. It's thorough, it's sharp, and it's ready for action.\n\n### CI/CD Ready\n\nRun Hercules locally or integrate it seamlessly into your CI/CD pipeline. Docker-native and one-command ready, Hercules fits smoothly into your deployment workflows, keeping testing quick, consistent, and hassle-free.\n\nWith Hercules, testing is no longer just a step in the process\u2014it's a powerful, streamlined experience that brings quality to the forefront.\n\n---\n\n\n## \ud83e\uddbe Architecture\n\n### Multi-Agentic Solution\n\nHercules leverages a multi-agent architecture based on the AutoGen framework. Building on the foundation provided by the AutoGen framework, Hercules's architecture leverages the interplay between tools and agents. Each tool embodies an atomic action, a fundamental building block that, when executed, returns a natural language description of its outcome. This granularity allows Hercules to flexibly assemble these tools to tackle complex web automation workflows.\n\n#### System View\n\n\n\nThe diagram above shows the configuration chosen on top of AutoGen architecture. The tools can be partitioned differently, but this is the one that we chose for the time being. We chose to use tools that map to what humans learn about the web browser rather than allow the LLM to write code as it pleases. We see the use of configured tools to be safer and more predictable in its outcomes. Certainly, it can click on the wrong things, but at least it is not going to execute malicious unknown code.\n\n\n#### Agents\n\nAt the moment, there are two agents:\n\n1. **Planner Agent**: Executes the planning and decomposition of tasks.\n2. **Browser Navigation Agent**: Embodies all the tools for interacting with the web browser.\n\n#### Tools Library\n\nAt the core of Hercules's capabilities is the Tools Library, a repository of well-defined actions that Hercules can perform; for now, web actions. These tools are grouped into two main categories:\n\n- **Sensing Tools**: Tools like `get_dom_with_content_type` and `geturl` that help Hercules understand the current state of the webpage or the browser.\n- **Action Tools**: Tools that allow Hercules to interact with and manipulate the web environment, such as `click`, `enter_text`, and `openurl`.\n\nEach tool is created with the intention to be as conversational as possible, making the interactions with LLMs more intuitive and error-tolerant. For instance, rather than simply returning a boolean value, a tool might explain in natural language what happened during its execution, enabling the LLM to better understand the context and correct course if necessary.\n\n##### Implemented Tools\n\n- **Sensing Tools**\n\n - `geturl`: Fetches and returns the current URL.\n - `get_dom_with_content_type`: Retrieves the HTML DOM of the active page based on the specified content type.\n - `text_only`: Extracts the inner text of the HTML DOM. Responds with text output.\n - `input_fields`: Extracts the interactive elements in the DOM (button, input, textarea, etc.) and responds with a compact JSON object.\n - `all_fields`: Extracts all the fields in the DOM and responds with a compact JSON object.\n - `get_user_input`: Provides the orchestrator with a mechanism to receive user feedback to disambiguate or seek clarity on fulfilling their request.\n\n- **Action Tools**\n\n - `click`: Given a DOM query selector, this will click on it.\n - `enter_text`: Enters text in a field specified by the provided DOM query selector.\n - `enter_text_and_click`: Optimized method that combines `enter_text` and `click` tools.\n - `bulk_enter_text`: Optimized method that wraps `enter_text` method so that multiple text entries can be performed in one shot.\n - `openurl`: Opens the given URL in the current or new tab.\n\n#### DOM Distillation\n\nHercules's approach to managing the vast landscape of HTML DOM is methodical and essential for efficiency. We've introduced **DOM Distillation** to pare down the DOM to just the elements pertinent to the user's task.\n\nIn practice, this means taking the expansive DOM and delivering a more digestible JSON snapshot. This isn't about just reducing size; it's about honing in on relevance, serving the LLMs only what's necessary to fulfill a request. So far, we have three content types:\n\n1. **Text Only**: For when the mission is information retrieval, and the text is the target. No distractions.\n2. **Input Fields**: Zeroing in on elements that call for user interaction. It's about streamlining actions.\n3. **All Content**: The full scope of distilled DOM, encompassing all elements when the task demands a comprehensive understanding.\n\nIt's a surgical procedure, carefully removing extraneous information while preserving the structure and content needed for the agent's operation. Of course, with any distillation, there could be casualties, but the idea is to refine this over time to limit/eliminate them.\n\nSince we can't rely on all web page authors to use best practices, such as adding unique IDs to each HTML element, we had to inject our own attribute (`mmid`) in every DOM element. We can then guide the LLM to rely on using `mmid` in the generated DOM queries.\n\nTo cut down on some of the DOM noise, we use the **DOM Accessibility Tree** rather than the regular HTML DOM. The accessibility tree, by nature, is geared towards helping screen readers, which is closer to the mission of web automation than plain old HTML DOM.\n\nThe distillation process is a work in progress. We look to refine this process and condense the DOM further, aiming to make interactions faster, cost-effective, and more accurate.\n\n---\n\n## \ud83d\udd2c Testing and Evaluation: QEvals\n\nWe wanted to ensure that Hercules stands up to the task of end-to-end testing with immense precision. So, we have run Hercules through a wide range of tests such as running APIs, interacting with complex UI scenarios, clicking through calendars, or iframes. A full list of evaluations can be found in the [tests folder](<Link to tests folder>).\n\n### Running Tests\n\nTo run the full test suite, use the following command:\n\n```bash\nmake test\n```\n\nTo run a specific test:\n\n```bash\nmake test-case\n```\n\nHercules builds on the work done by WebArena and Agent-E, and beyond that, to iron out the issues in the previous, we have written our own test cases catering to complex QA scenarios and have created tests in the `./tests` folder.\n\n---\n\n## \ud83d\udca1 Opinions\n\nWe believe that great quality comes from opinions about a product. So we have incorporated a few of our opinions into the product design. We welcome the community to question them, use them, or build on top of them. Here are some examples:\n\n1. **Gherkin is a Good Enough Format for Agents**: Gherkin provides a semi-structured format for the LLMs/AI Agents to follow test intent and user instructions. It provides the right amount of grammar (verbs like Given, When, Then) for humans to frame a scenario and agents to follow the instructions.\n\n2. **Tests Should Be Atomic in Nature**: Software tests should be atomic because it ensures that each test is **focused, independent, and reliable**. Atomic tests target one specific behavior or functionality, making it easier to pinpoint the root cause of failures without ambiguity.\n\n **Here's a good example (Atomic Test)**:\n\n ```gherkin\n Feature: User Login\n\n Scenario: Successful login with valid credentials\n\n Given the user is on the login page\n When the user enters valid credentials\n And the user clicks the login button\n Then the user should see the dashboard\n ```\n\n A non-atomic test confuses both the tester and the AI agent.\n\n3. **Open Core and Open Source**: Hercules is built on an **open-core model**, combining the spirit of open source with the support and expertise of a commercial company, **TestZeus**. By providing Hercules as open source (licensed under AGPL v3), TestZeus is committed to empowering the testing community with a robust, adaptable tool that's freely accessible and modifiable. Open source offers transparency, trust, and collaborative potential, allowing anyone to dive into the code, contribute, and shape the project's direction.\n\n4. **Telemetry** : All great products are built on good feedback. We have setup telemetry so that we can take feedback, without disturbing the user. Telemetry is enabled by default, but we also believe strongly in the values of \"Trust\" and \"Transparency\" so it can be turned off by the users.\n\n---\n\n## \ud83e\ude99 Token Usage\n\nHercules is an AI-native solution and relies on LLMs to perform reasoning and actions. Based on our experiments, we have found that a complex use case as below could cost up to **$0.20** using OpenAI's APIs gpt-4o, check the properties printed in testcase output to calculate for your testcase:\n\n```gherkin\nFeature: Account Creation in Salesforce\n\n Scenario: Successfully create a new account\n\n Given I am on the Salesforce login page\n When I enter my username \"user@example.com\" and password \"securePassword\"\n And I click on the \"Log In\" button\n And I navigate to the \"Accounts\" tab\n And I click on the \"New\" button\n And I fill in the \"Account Name\" field with \"Test Account\"\n And I select the \"Account Type\" as \"Customer\"\n And I fill in the \"Website\" field with \"www.testaccount.com\"\n And I fill in the \"Phone\" field with \"123-456-7890\"\n And I click on the \"Save\" button\n Then I should see a confirmation message \"Account Test Account created successfully\"\n And I should see \"Test Account\" listed in the account records\n```\n\n---\n\n## \u2b50\ufe0f Difference from Other Tools\n\nHercules isn't just another testing tool\u2014it's an **agent**. Powered by synthetic intelligence that can **think, reason, and react** based on requirements, Hercules goes beyond simple automation scripts. We bring an industry-first approach to open-source agents for **software testing**. This means faster, smarter, and more resilient testing cycles, especially for complex platforms.\n\nWith **industry-leading performance** and a fully open-source foundation, Hercules combines powerful capabilities with community-driven flexibility, making top-tier testing accessible and transformative for everyone.\n\n---\n\n## \ud83d\ude80 High-Level Roadmap\n\n- **Enhanced LLM Support**: Integration with more LLMs and support for local LLM deployments.\n- **Advanced Tooling**: Addition of more tools to handle complex testing scenarios and environments.\n- **Improved DOM Distillation**: Refinements to the DOM distillation process for better efficiency and accuracy.\n- **Community Contributions**: Encourage and integrate community-driven features and tools.\n- **Extensive Documentation**: Expand documentation for better onboarding and usage.\n- **Bounty Program**: Launch a bounty program to incentivize contributions.\n\n---\n\n## \ud83e\udd17 Contribution\n\nWe welcome contributions from the community!\n\n- **Read the [CONTRIBUTING.md](CONTRIBUTING.md) file** to get started.\n- **Bounty Program**: Stay tuned for upcoming opportunities! \ud83d\ude00\n\n### How to Contribute Back\n\n1. **Developing Tools**\n\n - If you are developing tools for Hercules and want to contribute to the community, make sure you place the new tools in the `additional_tools` folder in your Pull Request.\n\n2. **Fixes and Enhancements**\n\n - If you have a fix on sensing tools that are fundamental to the system or something in prompts or something in the DOM distillation code, then put the changes in the relevant file and share the Pull Request.\n\n### Extending and Attaching Tools\n\n1. **Creating a New Tool**\n\n - You can start extending by adding tools to Hercules.\n - Refer to `testzeus_hercules/core/tools/sql_calls.py` as an example of how to create a new tool.\n - The key is the decorator `@tool` over the method that you want Hercules to execute.\n - The tool decorator should have a very clear description and name so that Hercules knows how to use the tool.\n - Also, in the method, you should be clear with annotations on what parameter is used for what purpose so that function calling in the LLM works best.\n\n2. **Adding the Tool**\n\n - Once you have created the new tools files in some folder path, you can pass the folder path to Hercules in the environment variable so that Hercules can read the new tools during the boot time and make sure that they are available during the execution.\n - Use `ADDITIONAL_TOOL_DIRS` to pass the path of the new tools folder where you have kept the new files.\n\n3. **Direct Addition (Not Recommended)**\n\n - In case you opt for adding the tools directly, then just put your new tools in the `testzeus_hercules/core/tools` path of the cloned repository.\n - Then make sure you import your tool module in the `testzeus_hercules/core/agents/browser_nav_agent.py` file as:\n\n ```python\n from testzeus_hercules.core.tools.sql_calls import *\n ```\n\n - **Note**: This way is not recommended. We prefer you try to use the `ADDITIONAL_TOOL_DIRS` approach.\n\n---\n\n## \ud83e\udd19 Contact Us\n\nJoin us at our [Slack](https://join.slack.com/t/testzeuscommunityhq/shared_invite/zt-2v2br8wog-FAmo_76xRHx~k~1oNaGQ0Q) to connect with the community, ask questions, and contribute.\n\n---\n\n## \u270d\ufe0f Examples\n\n- **Salesforce Examples**: [Link](tests/test_features/ebikes/ebikes.feature)\n- **Wrangler Example**: [Link](tests/test_features/productSearch/productSearch.feature)\n\n---\n\n## \ud83d\ude47 Credits\n\nHercules would not have been possible without the great work from the following sources:\n\n1. [Agent-E](https://arxiv.org/abs/2407.13032)\n2. [Q*](https://arxiv.org/abs/2312.10868)\n3. [Agent Q](https://arxiv.org/abs/2408.07199)\n4. [Autogen](https://arxiv.org/pdf/2308.08155)\n\nThe Hercules project is inferred and enhanced over the existing project of [Agent-E](https://github.com/EmergenceAI/Agent-E). We have improved lots of cases to make it capable of doing testing, especially in the area of complex DOM navigation and iframes. We have also added new tools and abilities (like Salesforce navigation) to Hercules so that it can perform better work over the base framework we had picked.\n\nHercules also picks some inspiration from the legacy TestZeus repo [here](https://www.testzeus.org).\n\n\n---\n\nWith Hercules, testing is no longer just a step in the process\u2014it's a powerful, streamlined experience that brings quality to the forefront.\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Hercules: The World's First Open-Source AI Agent for End-to-End Testing",
"version": "0.0.15",
"project_urls": {
"Documentation": "https://testzeus.com/hercules/docs",
"Homepage": "https://testzeus.com/hercules",
"Repository": "https://github.com/testzeus/testzeus-hercules"
},
"split_keywords": [
"ai",
" end-to-end testing",
" automation",
" hercules",
" testzeus"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2784e85f7950781aadc959fbf5b033ef58a265e4660edd8a9ff9010256778e4d",
"md5": "85f04d3b24d6c1641466ab42a0f4cb5c",
"sha256": "edc04bd44278a00a589a08c7bc24070b24e34c4ecc9b03010bd4609bb3f4109b"
},
"downloads": -1,
"filename": "testzeus_hercules-0.0.15-py3-none-any.whl",
"has_sig": false,
"md5_digest": "85f04d3b24d6c1641466ab42a0f4cb5c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.11",
"size": 166444,
"upload_time": "2024-12-12T14:13:01",
"upload_time_iso_8601": "2024-12-12T14:13:01.153919Z",
"url": "https://files.pythonhosted.org/packages/27/84/e85f7950781aadc959fbf5b033ef58a265e4660edd8a9ff9010256778e4d/testzeus_hercules-0.0.15-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f05e925ba4354ec8358d9852f2bf74a7a0ce0b22c6f904f65f2257f28f50b24f",
"md5": "a14b78c5a9675c0c96206d5b163c575d",
"sha256": "3b0a9b57e746b9cb129f7e51d798b1586fd4fa93c8cac8f10529e63112ab3fe5"
},
"downloads": -1,
"filename": "testzeus_hercules-0.0.15.tar.gz",
"has_sig": false,
"md5_digest": "a14b78c5a9675c0c96206d5b163c575d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.11",
"size": 146856,
"upload_time": "2024-12-12T14:13:02",
"upload_time_iso_8601": "2024-12-12T14:13:02.710834Z",
"url": "https://files.pythonhosted.org/packages/f0/5e/925ba4354ec8358d9852f2bf74a7a0ce0b22c6f904f65f2257f28f50b24f/testzeus_hercules-0.0.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-12 14:13:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "testzeus",
"github_project": "testzeus-hercules",
"github_not_found": true,
"lcname": "testzeus-hercules"
}
JSON
