# Web Research Agent
A research implementation of the ReAct (Reasoning + Acting) paradigm for web research. The system analyzes a task, plans concrete tool actions, executes them (search, browse, optional code), and synthesizes an answer with sources and basic verification. The design is task-agnostic: no topic-specific heuristics are required to operate across different questions.
> This repository is intended for studying structured approaches to web research and evaluation. It is not a production system.
## Research Contributions
- Task-agnostic analysis of question structure to select a synthesis strategy without topic-specific rules
- Adaptive planning that produces actionable steps over registered tools (search, browser, present, optional code)
- Multi-strategy synthesis (extract-and-verify, aggregate-and-filter, collect-and-organize, comprehensive-synthesis)
- Robust parameter and URL handling with snippet fallback when pages cannot be fetched
- Source attribution and lightweight cross-source verification
## Features
- Task-Adaptive Reasoning: infers expected answer type (factual lookup, list, comparison, narrative) from the task text
- Planning and Tool Use: generates steps that call SearchTool, BrowserTool, and PresentationTool with validated parameters
- Content Extraction: prefers main content from web pages; falls back to search snippets when needed
- Verification: groups paraphrases and scores support by distinct domains (extract-and-verify strategy)
- Structured Output: formats results deterministically for the requested answer type
- Configuration and Logging: configurable limits and detailed logs for analysis and evaluation
## Task-Agnostic Design
- No topic-specific filters or keywords are required for operation
- Requested list size is inferred from the task text (e.g., “10 statements”), not hardcoded
- De-duplication is based on content similarity and source/date keys, not hand-written topic rules
- Statement/quote extraction uses generic patterns (quotes and sentence heuristics)
- Verification relies on cross-source/domain support rather than task-specific logic
## Tool Interfaces
- SearchTool: returns a list of results (title, link, snippet)
- BrowserTool: fetches a URL and extracts main content or full page; can aggregate search snippets if a URL is not available
- PresentationTool: assembles the final answer for the chosen strategy
- CodeGeneratorTool: optional, for tasks that require computation (e.g., filtering or plotting)
See implementation: agent/agent.py, agent/planner.py, tools/search.py, tools/browser.py, tools/presentation_tool.py.
## Execution Flow
1. Analyze the task to determine answer type and information targets
2. Create a plan: search → browse (one or more pages) → present (optionally code when needed)
3. Execute tools with parameter and URL resolution; use snippet fallback as needed
4. Synthesize the answer using one of four strategies; attribute sources and verify where applicable
## Architecture & ReAct Implementation
This project implements the ReAct paradigm with dynamic task analysis and adaptive synthesis:
```mermaid
graph TD
A[Main] --> B[WebResearchAgent]
B --> C1[Memory]
B --> C2[Planner]
B --> C3[Comprehension]
B --> C4[ToolRegistry]
%% Enhanced ReAct: Dynamic Reasoning
C3 -->|"Dynamic Analysis"| G1[Task Analysis]
G1 --> G2[Answer Type Detection]
G1 --> G3[Information Target ID]
G1 --> G4[Output Structure Inference]
C2 -->|"Adaptive Planning"| D[Plan]
D -->|Contains| E[PlanSteps]
%% ReAct: Acting component
C4 -->|Registers| F1[SearchTool]
C4 -->|Registers| F2[BrowserTool]
C4 -->|Registers| F3[CodeGeneratorTool]
C4 -->|Registers| F4[PresentationTool]
%% Enhanced ReAct: Multi-Strategy Synthesis
C3 -->|"Strategy Selection"| S1[Extract & Verify]
C3 -->|"Strategy Selection"| S2[Aggregate & Filter]
C3 -->|"Strategy Selection"| S3[Collect & Organize]
C3 -->|"Strategy Selection"| S4[Comprehensive Synthesis]
%% ReAct: Observation component
C1 -->|"Stores"| M1[Results & Entities]
%% ReAct: Iteration cycle
B -->|"1. Analyze Task"| G1
G1 -->|"2. Plan Strategy"| C2
C2 -->|"3. Execute Actions"| C4
C4 -->|"4. Synthesize Answer"| S1
S1 -->|"5. Verify & Refine"| B
style B fill:#f9f,stroke:#333,stroke-width:2px
style G1 fill:#fbb,stroke:#333,stroke-width:2px
style S1 fill:#bfb,stroke:#333,stroke-width:2px
style C1 fill:#bbf,stroke:#333
style C2 fill:#bbf,stroke:#333
style C3 fill:#bbf,stroke:#333
style C4 fill:#bbf,stroke:#333
style F1 fill:#bfb,stroke:#333
style F2 fill:#bfb,stroke:#333
style F3 fill:#bfb,stroke:#333
style F4 fill:#bfb,stroke:#333
```
### Workflow Explanation
The diagram above illustrates how the Web Research Agent processes research tasks:
1. **Task Analysis Phase**:
- When a user submits a research question, the system first analyzes the task structure
- The Comprehension component uses pattern recognition to detect answer types (factual, comparative, list-based, etc.)
- It identifies specific information targets needed to answer the question
- It determines the appropriate output structure for the anticipated answer
2. **Planning Phase**:
- Based on the task analysis, the Planner creates a series of search strategies
- It generates concrete plan steps targeting the identified information needs
- Each plan step specifies what information to retrieve and how to process it
3. **Action Phase**:
- The ToolRegistry orchestrates the execution of research tools:
- SearchTool finds relevant information sources
- BrowserTool extracts content from web pages
- CodeGeneratorTool creates analysis scripts when needed
- PresentationTool formats findings appropriately
4. **Synthesis Phase**:
- Based on the question type, one of four synthesis strategies is selected:
- Extract-and-Verify for factual questions
- Aggregate-and-Filter for comparative analyses
- Collect-and-Organize for list-building tasks
- Comprehensive-Synthesis for complex, multi-faceted questions
- The Memory component provides context by storing intermediate findings and entities
5. **Refinement Loop**:
- If the synthesized answer is incomplete, the system may return to planning
- This iterative process continues until a satisfactory answer is produced
- The final output is tailored to directly address the specific question asked
This research implementation demonstrates how a structured approach to web research can adapt to different question types without relying on hardcoded rules.
## Installation
### Prerequisites
- Python 3.9 or higher
- pip (Python package installer)
### Setup
1. Clone the repository:
```bash
git clone https://github.com/ashioyajotham/web_research_agent.git
cd web_research_agent
```
2. Create a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. Install dependencies:
```bash
pip install -r requirements.txt
```
## Research Environment Setup
The system requires the following external services for operation:
1. **Gemini API**: Language model for reasoning and synthesis
2. **Serper API**: Web search results for information gathering
### Setting up API keys
#### Option 1: .env file (Recommended for research)
Create a `.env` file in the project root:
```bash
GEMINI_API_KEY=your_gemini_api_key
SERPER_API_KEY=your_serper_api_key
```
#### Option 2: Environment Variables
```bash
export GEMINI_API_KEY=your_gemini_api_key
export SERPER_API_KEY=your_serper_api_key
```
#### Option 3: Programmatic Configuration
```python
from config.config_manager import init_config
config = init_config()
config.update('gemini_api_key', 'your_gemini_api_key')
config.update('serper_api_key', 'your_serper_api_key')
```
### Research Configuration Parameters
These parameters control the system's behavior and can be modified for experimental purposes:
| Parameter | Environment Variable | Description | Default |
|-----------|---------------------|-------------|---------|
| gemini_api_key | GEMINI_API_KEY | API key for Gemini LLM | - |
| serper_api_key | SERPER_API_KEY | API key for Serper.dev search | - |
| log_level | LOG_LEVEL | Logging detail level | INFO |
| max_search_results | MAX_SEARCH_RESULTS | Search results to process | 5 |
| memory_limit | MEMORY_LIMIT | Working memory capacity | 100 |
| output_format | OUTPUT_FORMAT | Results format (markdown, text, html) | markdown |
| timeout | REQUEST_TIMEOUT | Web request timeout (seconds) | 30 |
## Usage
### Basic Research Tasks
1. Create a text file with research questions:
```txt
# tasks.txt
Find the name of the COO of the organization that mediated talks between US and Chinese AI companies in Geneva in 2023.
By what percentage did Volkswagen reduce their Scope 1 and Scope 2 greenhouse gas emissions in 2023 compared to 2021?
```
Note: Empty lines between tasks help the system distinguish between separate questions.
2. Run the research process:
```bash
python main.py tasks.txt
```
3. Results will be saved to the `results/` directory as Markdown files.
### Multi-Criteria Research Tasks
For complex queries with multiple requirements:
```txt
# multi_criteria_tasks.txt
Compile a list of companies satisfying the following criteria:
They are based in the EU
They operate within the motor vehicle sector
Their greenhouse gas emissions are available for 2021-2023
They earned more than €1B in revenue in 2023
```
The system recognizes this as a single multi-criteria task and adapts its synthesis strategy accordingly.
### Command Line Options
```bash
python main.py tasks.txt --output custom_output_dir
```
| Option | Description | Default |
|--------|-------------|---------|
| task_file | Path to text file containing tasks | (required) |
| --output | Directory to store results | results/ |
## Project Structure
The project structure reflects the enhanced ReAct implementation with dynamic analysis:
- **agent/**: Core reasoning and coordination
- **agent.py**: Main controller with dynamic task analysis and multi-strategy synthesis
- **comprehension.py**: Enhanced reasoning with pattern recognition for answer types
- **memory.py**: Short-term memory for tracking observations and synthesis context
- **planner.py**: Adaptive planning based on identified information targets
- **tools/**: Action components
- **search.py**: Information retrieval with robust URL resolution
- **browser.py**: Content extraction with multiple fallback strategies
- **code_generator.py**: Data analysis when computational tasks are detected
- **presentation_tool.py**: Task-adaptive result formatting
- **tool_registry.py**: Tool management system
- **utils/**: Supporting functions
- **console_ui.py**: Interface components
- **formatters.py**: Dynamic output structuring
- **task_parser.py**: Multi-criteria task parsing
- **criteria_filter.py**: Multi-criteria verification
- **logger.py**: Detailed reasoning and synthesis tracking
- **config/**: Research environment configuration
- **main.py**: Entry point and experiment runner
## Research Implementation Details
### Dynamic Task Analysis System
The system implements pattern recognition to analyze any research question and determine:
1. **Answer Type Detection**: Identifies whether the question expects a factual answer, comparison, list, or comprehensive analysis
2. **Information Target Identification**: Determines what specific information needs to be gathered
3. **Output Structure Inference**: Predicts the appropriate format for presenting the answer
4. **Synthesis Strategy Selection**: Chooses from four synthesis approaches based on task characteristics
### Multi-Strategy Synthesis Approaches
#### Extract-and-Verify Strategy
Used for factual lookup questions requiring specific information:
- Searches for target information across multiple sources
- Cross-validates findings for accuracy
- Provides direct answers with source verification
#### Aggregate-and-Filter Strategy
Applied to comparison and analytical questions:
- Collects relevant data points from multiple sources
- Applies filtering criteria to focus on relevant information
- Synthesizes comparative or analytical insights
#### Collect-and-Organize Strategy
Employed for list-building and compilation tasks:
- Systematically gathers items meeting specified criteria
- Organizes findings in structured formats
- Validates completeness of collected information
#### Comprehensive-Synthesis Strategy
Used for complex, multi-faceted research questions:
- Integrates information from diverse sources
- Builds coherent narratives or explanations
- Balances breadth and depth of coverage
### Enhanced Parameter Resolution
The system includes robust handling of web search challenges:
- Multiple URL extraction strategies from search results
- Fallback mechanisms for content retrieval failures
- Validation of information sources and URLs
- Graceful degradation when full content is unavailable
### Entity Extraction & Relationship Mapping
The system extracts relevant entities while maintaining focus on answering the specific question:
- **People**: Names of individuals relevant to the research question
- **Organizations**: Companies, agencies, groups mentioned in sources
- **Roles**: Job titles and positions when relevant to the query
- **Locations**: Geographic information pertinent to the task
- **Dates**: Temporal references important for the research context
Entity extraction supports the synthesis process but does not drive the output format.
### Error Recovery and Robustness
The system implements multiple fallback strategies:
1. **Content Access Failures**: When websites block access, falls back to search snippet analysis
2. **URL Resolution Issues**: Multiple strategies for extracting valid URLs from search results
3. **Information Gaps**: Acknowledges limitations and reports partial findings when complete answers aren't available
4. **Synthesis Failures**: Provides available information even when preferred synthesis strategy fails
### Customization Options
You can modify system behavior through configuration:
```python
from config.config_manager import init_config
config = init_config()
config.update('output_format', 'html') # Options: markdown, json, html
config.update('max_search_results', 10) # Increase search breadth
```
## Research Limitations & Observations
As a research implementation, this project provides insights into both capabilities and current limitations:
### Current Research Limitations
1. **Web Access Constraints**: Sites with anti-scraping measures may limit data collection, providing opportunities to study fallback strategies
2. **Complex Query Formulation**: Highly specialized domains sometimes require domain-specific search strategies
3. **Synthesis Boundary Cases**: Edge cases in task analysis provide insights into pattern recognition limitations
4. **Computational Requirements**: Multi-criteria tasks with extensive search requirements demonstrate resource scaling behavior
### Research Insights from Implementation
Detailed logs in the `logs/` directory provide research data on:
- Dynamic task analysis decision patterns
- Synthesis strategy selection effectiveness
- URL resolution fallback frequency and success rates
- Entity extraction accuracy across different content types
- Error recovery mechanism performance
These logs are valuable for understanding the system's behavior and identifying areas for algorithmic improvement.
## Contributing
This research implementation welcomes contributions, particularly in areas of:
- Enhanced pattern recognition for task analysis
- Additional synthesis strategies for specialized question types
- Improved robustness in web content extraction
- Performance optimization for large-scale research tasks
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
## Research Background & Extensions
This project implements and extends the ReAct (Reasoning + Acting) paradigm from ["ReAct: Synergizing Reasoning and Acting in Language Models"](https://arxiv.org/abs/2210.03629) (Yao et al., 2022).
### Core ReAct Implementation
The foundational ReAct components:
1. **Reasoning**: Task decomposition and solution planning
2. **Acting**: Tool execution based on reasoning
3. **Observation**: Processing action results
4. **Iteration**: Feedback loops for refinement
### Research Extensions
This implementation extends ReAct with:
- **Dynamic Task Analysis**: Pattern recognition for answer type detection without hardcoded rules
- **Multi-Strategy Synthesis**: Adaptive synthesis based on task characteristics rather than fixed approaches
- **Robust Parameter Resolution**: Multiple fallback mechanisms for real-world web research challenges
- **Task-Focused Output**: Direct answer generation aligned with question intent
### Research Findings
Key observations from this implementation:
1. **Pattern Recognition Effectiveness**: Dynamic task analysis successfully identifies answer types across diverse question structures
2. **Synthesis Strategy Impact**: Different synthesis strategies show measurable differences in answer quality for different question types
3. **Fallback Strategy Value**: Robust parameter resolution significantly improves success rates for web content access
4. **Entity vs. Answer Focus**: Maintaining task focus while extracting entities produces more relevant outputs than entity-driven approaches
### Acknowledgements
This research implementation draws from established agent concepts and development approaches, including:
- [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling) - Best practices for tool-using agents
- [Anthropic's Claude Agent Guide](https://www.anthropic.com/research/claude-agent) - Methods for reliable agent construction
- [LangChain ReAct Implementation](https://python.langchain.com/docs/modules/agents/agent_types/react) - Technical approaches for implementing ReAct
### Related Research
- [Chain-of-Thought Prompting](https://arxiv.org/abs/2201.11903) - Wei et al. (2022)
- [Language Models as Zero-Shot Planners](https://arxiv.org/abs/2201.07207) - Huang et al. (2022)
- [Faithful Reasoning Using Large Language Models](https://arxiv.org/abs/2208.14271) - Creswell et al. (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761) - Schick et al. (2023)
Raw data
{
"_id": null,
"home_page": "https://github.com/ashioyajotham/web_research_agent",
"name": "web-research-agent",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ai, research, web, agent, search",
"author": "Victor Jotham Ashioya",
"author_email": "victorashioya960@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/95/86/212269c2430291e5dadf09e63d61668841c9403e8c862a8b4e9b9a482396/web_research_agent-1.1.11.tar.gz",
"platform": null,
"description": "# Web Research Agent\n\nA research implementation of the ReAct (Reasoning + Acting) paradigm for web research. The system analyzes a task, plans concrete tool actions, executes them (search, browse, optional code), and synthesizes an answer with sources and basic verification. The design is task-agnostic: no topic-specific heuristics are required to operate across different questions.\n\n> This repository is intended for studying structured approaches to web research and evaluation. It is not a production system.\n\n## Research Contributions\n\n- Task-agnostic analysis of question structure to select a synthesis strategy without topic-specific rules\n- Adaptive planning that produces actionable steps over registered tools (search, browser, present, optional code)\n- Multi-strategy synthesis (extract-and-verify, aggregate-and-filter, collect-and-organize, comprehensive-synthesis)\n- Robust parameter and URL handling with snippet fallback when pages cannot be fetched\n- Source attribution and lightweight cross-source verification\n\n## Features\n\n- Task-Adaptive Reasoning: infers expected answer type (factual lookup, list, comparison, narrative) from the task text\n- Planning and Tool Use: generates steps that call SearchTool, BrowserTool, and PresentationTool with validated parameters\n- Content Extraction: prefers main content from web pages; falls back to search snippets when needed\n- Verification: groups paraphrases and scores support by distinct domains (extract-and-verify strategy)\n- Structured Output: formats results deterministically for the requested answer type\n- Configuration and Logging: configurable limits and detailed logs for analysis and evaluation\n\n## Task-Agnostic Design\n\n- No topic-specific filters or keywords are required for operation\n- Requested list size is inferred from the task text (e.g., \u201c10 statements\u201d), not hardcoded\n- De-duplication is based on content similarity and source/date keys, not hand-written topic rules\n- Statement/quote extraction uses generic patterns (quotes and sentence heuristics)\n- Verification relies on cross-source/domain support rather than task-specific logic\n\n## Tool Interfaces\n\n- SearchTool: returns a list of results (title, link, snippet)\n- BrowserTool: fetches a URL and extracts main content or full page; can aggregate search snippets if a URL is not available\n- PresentationTool: assembles the final answer for the chosen strategy\n- CodeGeneratorTool: optional, for tasks that require computation (e.g., filtering or plotting)\n\nSee implementation: agent/agent.py, agent/planner.py, tools/search.py, tools/browser.py, tools/presentation_tool.py.\n\n## Execution Flow\n\n1. Analyze the task to determine answer type and information targets\n2. Create a plan: search \u2192 browse (one or more pages) \u2192 present (optionally code when needed)\n3. Execute tools with parameter and URL resolution; use snippet fallback as needed\n4. Synthesize the answer using one of four strategies; attribute sources and verify where applicable\n\n## Architecture & ReAct Implementation\n\nThis project implements the ReAct paradigm with dynamic task analysis and adaptive synthesis:\n\n```mermaid\ngraph TD\n A[Main] --> B[WebResearchAgent]\n B --> C1[Memory]\n B --> C2[Planner]\n B --> C3[Comprehension]\n B --> C4[ToolRegistry]\n \n %% Enhanced ReAct: Dynamic Reasoning\n C3 -->|\"Dynamic Analysis\"| G1[Task Analysis]\n G1 --> G2[Answer Type Detection]\n G1 --> G3[Information Target ID]\n G1 --> G4[Output Structure Inference]\n \n C2 -->|\"Adaptive Planning\"| D[Plan]\n D -->|Contains| E[PlanSteps]\n \n %% ReAct: Acting component\n C4 -->|Registers| F1[SearchTool]\n C4 -->|Registers| F2[BrowserTool]\n C4 -->|Registers| F3[CodeGeneratorTool]\n C4 -->|Registers| F4[PresentationTool]\n \n %% Enhanced ReAct: Multi-Strategy Synthesis\n C3 -->|\"Strategy Selection\"| S1[Extract & Verify]\n C3 -->|\"Strategy Selection\"| S2[Aggregate & Filter]\n C3 -->|\"Strategy Selection\"| S3[Collect & Organize]\n C3 -->|\"Strategy Selection\"| S4[Comprehensive Synthesis]\n \n %% ReAct: Observation component\n C1 -->|\"Stores\"| M1[Results & Entities]\n \n %% ReAct: Iteration cycle\n B -->|\"1. Analyze Task\"| G1\n G1 -->|\"2. Plan Strategy\"| C2\n C2 -->|\"3. Execute Actions\"| C4\n C4 -->|\"4. Synthesize Answer\"| S1\n S1 -->|\"5. Verify & Refine\"| B\n \n style B fill:#f9f,stroke:#333,stroke-width:2px\n style G1 fill:#fbb,stroke:#333,stroke-width:2px\n style S1 fill:#bfb,stroke:#333,stroke-width:2px\n style C1 fill:#bbf,stroke:#333\n style C2 fill:#bbf,stroke:#333\n style C3 fill:#bbf,stroke:#333\n style C4 fill:#bbf,stroke:#333\n style F1 fill:#bfb,stroke:#333\n style F2 fill:#bfb,stroke:#333\n style F3 fill:#bfb,stroke:#333\n style F4 fill:#bfb,stroke:#333\n```\n\n### Workflow Explanation\n\nThe diagram above illustrates how the Web Research Agent processes research tasks:\n\n1. **Task Analysis Phase**:\n - When a user submits a research question, the system first analyzes the task structure\n - The Comprehension component uses pattern recognition to detect answer types (factual, comparative, list-based, etc.)\n - It identifies specific information targets needed to answer the question\n - It determines the appropriate output structure for the anticipated answer\n\n2. **Planning Phase**:\n - Based on the task analysis, the Planner creates a series of search strategies\n - It generates concrete plan steps targeting the identified information needs\n - Each plan step specifies what information to retrieve and how to process it\n\n3. **Action Phase**:\n - The ToolRegistry orchestrates the execution of research tools:\n - SearchTool finds relevant information sources\n - BrowserTool extracts content from web pages\n - CodeGeneratorTool creates analysis scripts when needed\n - PresentationTool formats findings appropriately\n\n4. **Synthesis Phase**:\n - Based on the question type, one of four synthesis strategies is selected:\n - Extract-and-Verify for factual questions\n - Aggregate-and-Filter for comparative analyses\n - Collect-and-Organize for list-building tasks\n - Comprehensive-Synthesis for complex, multi-faceted questions\n - The Memory component provides context by storing intermediate findings and entities\n\n5. **Refinement Loop**:\n - If the synthesized answer is incomplete, the system may return to planning\n - This iterative process continues until a satisfactory answer is produced\n - The final output is tailored to directly address the specific question asked\n\nThis research implementation demonstrates how a structured approach to web research can adapt to different question types without relying on hardcoded rules.\n\n## Installation\n\n### Prerequisites\n\n- Python 3.9 or higher\n- pip (Python package installer)\n\n### Setup\n\n1. Clone the repository:\n\n ```bash\n git clone https://github.com/ashioyajotham/web_research_agent.git\n cd web_research_agent\n ```\n\n2. Create a virtual environment:\n\n ```bash\n python -m venv venv\n source venv/bin/activate # On Windows: venv\\Scripts\\activate\n ```\n\n3. Install dependencies:\n\n ```bash\n pip install -r requirements.txt\n ```\n\n## Research Environment Setup\n\nThe system requires the following external services for operation:\n\n1. **Gemini API**: Language model for reasoning and synthesis\n2. **Serper API**: Web search results for information gathering\n\n### Setting up API keys\n\n#### Option 1: .env file (Recommended for research)\n\nCreate a `.env` file in the project root:\n\n```bash\nGEMINI_API_KEY=your_gemini_api_key\nSERPER_API_KEY=your_serper_api_key\n```\n\n#### Option 2: Environment Variables\n\n```bash\nexport GEMINI_API_KEY=your_gemini_api_key\nexport SERPER_API_KEY=your_serper_api_key\n```\n\n#### Option 3: Programmatic Configuration\n\n```python\nfrom config.config_manager import init_config\n\nconfig = init_config()\nconfig.update('gemini_api_key', 'your_gemini_api_key')\nconfig.update('serper_api_key', 'your_serper_api_key')\n```\n\n### Research Configuration Parameters\n\nThese parameters control the system's behavior and can be modified for experimental purposes:\n\n| Parameter | Environment Variable | Description | Default |\n|-----------|---------------------|-------------|---------|\n| gemini_api_key | GEMINI_API_KEY | API key for Gemini LLM | - |\n| serper_api_key | SERPER_API_KEY | API key for Serper.dev search | - |\n| log_level | LOG_LEVEL | Logging detail level | INFO |\n| max_search_results | MAX_SEARCH_RESULTS | Search results to process | 5 |\n| memory_limit | MEMORY_LIMIT | Working memory capacity | 100 |\n| output_format | OUTPUT_FORMAT | Results format (markdown, text, html) | markdown |\n| timeout | REQUEST_TIMEOUT | Web request timeout (seconds) | 30 |\n\n## Usage\n\n### Basic Research Tasks\n\n1. Create a text file with research questions:\n\n ```txt\n # tasks.txt\n Find the name of the COO of the organization that mediated talks between US and Chinese AI companies in Geneva in 2023.\n By what percentage did Volkswagen reduce their Scope 1 and Scope 2 greenhouse gas emissions in 2023 compared to 2021?\n ```\n\n Note: Empty lines between tasks help the system distinguish between separate questions.\n\n2. Run the research process:\n\n ```bash\n python main.py tasks.txt\n ```\n\n3. Results will be saved to the `results/` directory as Markdown files.\n\n### Multi-Criteria Research Tasks\n\nFor complex queries with multiple requirements:\n\n```txt\n# multi_criteria_tasks.txt\nCompile a list of companies satisfying the following criteria:\n They are based in the EU\n They operate within the motor vehicle sector\n Their greenhouse gas emissions are available for 2021-2023\n They earned more than \u20ac1B in revenue in 2023\n```\n\nThe system recognizes this as a single multi-criteria task and adapts its synthesis strategy accordingly.\n\n### Command Line Options\n\n```bash\npython main.py tasks.txt --output custom_output_dir\n```\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| task_file | Path to text file containing tasks | (required) |\n| --output | Directory to store results | results/ |\n\n## Project Structure\n\nThe project structure reflects the enhanced ReAct implementation with dynamic analysis:\n\n- **agent/**: Core reasoning and coordination\n - **agent.py**: Main controller with dynamic task analysis and multi-strategy synthesis\n - **comprehension.py**: Enhanced reasoning with pattern recognition for answer types\n - **memory.py**: Short-term memory for tracking observations and synthesis context\n - **planner.py**: Adaptive planning based on identified information targets\n \n- **tools/**: Action components\n - **search.py**: Information retrieval with robust URL resolution\n - **browser.py**: Content extraction with multiple fallback strategies\n - **code_generator.py**: Data analysis when computational tasks are detected\n - **presentation_tool.py**: Task-adaptive result formatting\n - **tool_registry.py**: Tool management system\n\n- **utils/**: Supporting functions\n - **console_ui.py**: Interface components\n - **formatters.py**: Dynamic output structuring\n - **task_parser.py**: Multi-criteria task parsing\n - **criteria_filter.py**: Multi-criteria verification\n - **logger.py**: Detailed reasoning and synthesis tracking\n\n- **config/**: Research environment configuration\n \n- **main.py**: Entry point and experiment runner\n\n## Research Implementation Details\n\n### Dynamic Task Analysis System\n\nThe system implements pattern recognition to analyze any research question and determine:\n\n1. **Answer Type Detection**: Identifies whether the question expects a factual answer, comparison, list, or comprehensive analysis\n2. **Information Target Identification**: Determines what specific information needs to be gathered\n3. **Output Structure Inference**: Predicts the appropriate format for presenting the answer\n4. **Synthesis Strategy Selection**: Chooses from four synthesis approaches based on task characteristics\n\n### Multi-Strategy Synthesis Approaches\n\n#### Extract-and-Verify Strategy\n\nUsed for factual lookup questions requiring specific information:\n\n- Searches for target information across multiple sources\n- Cross-validates findings for accuracy\n- Provides direct answers with source verification\n\n#### Aggregate-and-Filter Strategy\n\nApplied to comparison and analytical questions:\n\n- Collects relevant data points from multiple sources\n- Applies filtering criteria to focus on relevant information\n- Synthesizes comparative or analytical insights\n\n#### Collect-and-Organize Strategy\n\nEmployed for list-building and compilation tasks:\n\n- Systematically gathers items meeting specified criteria\n- Organizes findings in structured formats\n- Validates completeness of collected information\n\n#### Comprehensive-Synthesis Strategy\n\nUsed for complex, multi-faceted research questions:\n\n- Integrates information from diverse sources\n- Builds coherent narratives or explanations\n- Balances breadth and depth of coverage\n\n### Enhanced Parameter Resolution\n\nThe system includes robust handling of web search challenges:\n\n- Multiple URL extraction strategies from search results\n- Fallback mechanisms for content retrieval failures\n- Validation of information sources and URLs\n- Graceful degradation when full content is unavailable\n\n### Entity Extraction & Relationship Mapping\n\nThe system extracts relevant entities while maintaining focus on answering the specific question:\n\n- **People**: Names of individuals relevant to the research question\n- **Organizations**: Companies, agencies, groups mentioned in sources\n- **Roles**: Job titles and positions when relevant to the query\n- **Locations**: Geographic information pertinent to the task\n- **Dates**: Temporal references important for the research context\n\nEntity extraction supports the synthesis process but does not drive the output format.\n\n### Error Recovery and Robustness\n\nThe system implements multiple fallback strategies:\n\n1. **Content Access Failures**: When websites block access, falls back to search snippet analysis\n2. **URL Resolution Issues**: Multiple strategies for extracting valid URLs from search results\n3. **Information Gaps**: Acknowledges limitations and reports partial findings when complete answers aren't available\n4. **Synthesis Failures**: Provides available information even when preferred synthesis strategy fails\n\n### Customization Options\n\nYou can modify system behavior through configuration:\n\n```python\nfrom config.config_manager import init_config\n\nconfig = init_config()\nconfig.update('output_format', 'html') # Options: markdown, json, html\nconfig.update('max_search_results', 10) # Increase search breadth\n```\n\n## Research Limitations & Observations\n\nAs a research implementation, this project provides insights into both capabilities and current limitations:\n\n### Current Research Limitations\n\n1. **Web Access Constraints**: Sites with anti-scraping measures may limit data collection, providing opportunities to study fallback strategies\n2. **Complex Query Formulation**: Highly specialized domains sometimes require domain-specific search strategies\n3. **Synthesis Boundary Cases**: Edge cases in task analysis provide insights into pattern recognition limitations\n4. **Computational Requirements**: Multi-criteria tasks with extensive search requirements demonstrate resource scaling behavior\n\n### Research Insights from Implementation\n\nDetailed logs in the `logs/` directory provide research data on:\n\n- Dynamic task analysis decision patterns\n- Synthesis strategy selection effectiveness\n- URL resolution fallback frequency and success rates\n- Entity extraction accuracy across different content types\n- Error recovery mechanism performance\n\nThese logs are valuable for understanding the system's behavior and identifying areas for algorithmic improvement.\n\n## Contributing\n\nThis research implementation welcomes contributions, particularly in areas of:\n\n- Enhanced pattern recognition for task analysis\n- Additional synthesis strategies for specialized question types\n- Improved robustness in web content extraction\n- Performance optimization for large-scale research tasks\n\nPlease see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n## Research Background & Extensions\n\nThis project implements and extends the ReAct (Reasoning + Acting) paradigm from [\"ReAct: Synergizing Reasoning and Acting in Language Models\"](https://arxiv.org/abs/2210.03629) (Yao et al., 2022).\n\n### Core ReAct Implementation\n\nThe foundational ReAct components:\n\n1. **Reasoning**: Task decomposition and solution planning\n2. **Acting**: Tool execution based on reasoning\n3. **Observation**: Processing action results\n4. **Iteration**: Feedback loops for refinement\n\n### Research Extensions\n\nThis implementation extends ReAct with:\n\n- **Dynamic Task Analysis**: Pattern recognition for answer type detection without hardcoded rules\n- **Multi-Strategy Synthesis**: Adaptive synthesis based on task characteristics rather than fixed approaches \n- **Robust Parameter Resolution**: Multiple fallback mechanisms for real-world web research challenges\n- **Task-Focused Output**: Direct answer generation aligned with question intent\n\n### Research Findings\n\nKey observations from this implementation:\n\n1. **Pattern Recognition Effectiveness**: Dynamic task analysis successfully identifies answer types across diverse question structures\n2. **Synthesis Strategy Impact**: Different synthesis strategies show measurable differences in answer quality for different question types\n3. **Fallback Strategy Value**: Robust parameter resolution significantly improves success rates for web content access\n4. **Entity vs. Answer Focus**: Maintaining task focus while extracting entities produces more relevant outputs than entity-driven approaches\n\n### Acknowledgements\n\nThis research implementation draws from established agent concepts and development approaches, including:\n\n- [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling) - Best practices for tool-using agents\n- [Anthropic's Claude Agent Guide](https://www.anthropic.com/research/claude-agent) - Methods for reliable agent construction\n- [LangChain ReAct Implementation](https://python.langchain.com/docs/modules/agents/agent_types/react) - Technical approaches for implementing ReAct\n\n### Related Research\n\n- [Chain-of-Thought Prompting](https://arxiv.org/abs/2201.11903) - Wei et al. (2022)\n- [Language Models as Zero-Shot Planners](https://arxiv.org/abs/2201.07207) - Huang et al. (2022)\n- [Faithful Reasoning Using Large Language Models](https://arxiv.org/abs/2208.14271) - Creswell et al. (2022)\n- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761) - Schick et al. (2023)\n",
"bugtrack_url": null,
"license": null,
"summary": "An agent for web research, capable of understanding complex tasks and executing them using various tools.",
"version": "1.1.11",
"project_urls": {
"Bug Tracker": "https://github.com/ashioyajotham/web_research_agent/issues",
"Documentation": "https://github.com/ashioyajotham/web_research_agent#readme",
"Homepage": "https://github.com/ashioyajotham/web_research_agent",
"Source Code": "https://github.com/ashioyajotham/web_research_agent"
},
"split_keywords": [
"ai",
" research",
" web",
" agent",
" search"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "6ffb71440825dbaf59d610cbcffe3f75a3f204f6b3afaef60efbc51f993c81fa",
"md5": "1de71017acfdc4ebd900b68858ade463",
"sha256": "26740a0983cb128a60982bc881eef4ab27d51a00aafe5e6d3609b8cee714fdc7"
},
"downloads": -1,
"filename": "web_research_agent-1.1.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1de71017acfdc4ebd900b68858ade463",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 52684,
"upload_time": "2025-09-01T08:32:28",
"upload_time_iso_8601": "2025-09-01T08:32:28.080814Z",
"url": "https://files.pythonhosted.org/packages/6f/fb/71440825dbaf59d610cbcffe3f75a3f204f6b3afaef60efbc51f993c81fa/web_research_agent-1.1.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "9586212269c2430291e5dadf09e63d61668841c9403e8c862a8b4e9b9a482396",
"md5": "3b8d13ea1b02908f5d4d0b1485fd76ba",
"sha256": "8a67e5785f04a7f0f9779347786f10c6770d353ee6e38f0dbb34113e875db75b"
},
"downloads": -1,
"filename": "web_research_agent-1.1.11.tar.gz",
"has_sig": false,
"md5_digest": "3b8d13ea1b02908f5d4d0b1485fd76ba",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 53683,
"upload_time": "2025-09-01T08:32:28",
"upload_time_iso_8601": "2025-09-01T08:32:28.998647Z",
"url": "https://files.pythonhosted.org/packages/95/86/212269c2430291e5dadf09e63d61668841c9403e8c862a8b4e9b9a482396/web_research_agent-1.1.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-01 08:32:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ashioyajotham",
"github_project": "web_research_agent",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "web-research-agent"
}
JSON
