# qufe
A comprehensive Python utility library for data processing, file handling, database management, and automation tasks.
Born from the need to streamline repetitive tasks in Jupyter Lab environments, qufe addresses common pain points encountered during interactive development and data exploration work.
## Installation
### Quick Start (Core Features Only)
```bash
# Install core functionality with no external dependencies
pip install qufe
```
### Feature-Specific Installation
Install only the features you need:
```bash
# Database operations (PostgreSQL)
pip install qufe[database]
# Data processing with pandas/numpy
pip install qufe[data]
# Web browser automation
pip install qufe[web]
# Screen capture and image processing
pip install qufe[vision]
```
## Features Overview
### Always Available (Core Features)
#### Base Utilities (`qufe.base`)
- **Timestamp handling**: Convert timestamps with timezone support
- **Code comparison**: Compare code snippets with multiple diff formats
- **Dynamic imports**: Import Python modules from file paths
- **List flattening**: Flatten nested structures with configurable depth
- **Dictionary utilities**: Advanced nested dictionary operations
#### Text Processing (`qufe.texthandler`, `qufe.excludebracket`)
- **Bracket content removal**: Validate and remove bracketed content
- **DokuWiki formatting**: Convert data to DokuWiki table format
- **String utilities**: Advanced search with context extraction
- **Pretty printing**: Format nested dictionaries and lists
- **Column display**: Multi-column text formatting with alignment
#### File Operations (`qufe.filehandler`)
- **Directory traversal**: Recursive file listing with Unicode normalization
- **Pattern matching**: Find latest files by datetime patterns
- **Pickle operations**: Simplified Python object persistence
- **Path utilities**: Safe filename generation and path management
- **Content extraction**: Text extraction from directory structures
### Optional Features (Require Additional Packages)
#### Database Management (`qufe.dbhandler`) - *[database]*
- **PostgreSQL integration**: Easy connections using SQLAlchemy
- **Database exploration**: List databases and tables with metadata
- **Connection management**: Automatic pooling and cleanup
- **Environment integration**: .env file and environment variable support
#### Data Analysis (`qufe.pdhandler`) - *[data]*
- **DataFrame utilities**: Type conversion and structure analysis
- **Column comparison**: Compare schemas across multiple DataFrames
- **Missing data detection**: Find NA values and empty strings
- **Data validation**: Comprehensive quality checks
#### Web Automation (`qufe.wbhandler`) - *[web]*
- **SeleniumBase integration**: Enhanced browser automation
- **Network monitoring**: Capture fetch/XHR requests
- **Element discovery**: Interactive element finding utilities
- **URL parsing**: Extract and manipulate URL parameters
- **Multi-browser support**: Firefox implementation with profile detection
#### Screen Interaction (`qufe.interactionhandler`) - *[vision]*
- **Screen capture**: Full screen or region-specific screenshots
- **Image processing**: Color detection and comparison algorithms
- **Mouse automation**: Randomized clicking for natural automation
- **Progress tracking**: Real-time updates in Jupyter notebooks
- **Color analysis**: Extract and analyze color information
## Quick Start Guide
### 1. Check Installation Status
```python
import qufe
# See what's available
qufe.help()
# Check dependencies programmatically
status = qufe.check_dependencies()
print(status)
# Get installation commands for missing features
missing = qufe.get_missing_dependencies()
for module, command in missing.items():
print(f"{module}: {command}")
```
### 2. Core Features (Always Available)
```python
from qufe import base, texthandler, filehandler
# Timestamp handling with timezone
ts = base.TS('Asia/Seoul')
formatted = ts.get_ts_formatted(1640995200)
# File operations
fh = filehandler.FileHandler()
files = fh.get_tree('/path/to/directory')
# Text processing
data = [['Name', 'Age'], ['Alice', '25'], ['Bob', '30']]
texthandler.list_to_doku_wiki_table(data)
# Directory exploration
pf = filehandler.PathFinder('/starting/path')
root, dirs, files = pf.get_one_depth()
```
### 3. Database Operations (Optional)
#### Configuration Options
**Option 1: .env file (Recommended)**
```bash
# Create .env file in your project root
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=your_database
```
**Option 2: Environment variables**
```bash
export POSTGRES_USER=your_username
export POSTGRES_PASSWORD=your_password
# ... etc
```
**Option 3: Direct parameters**
```python
from qufe.dbhandler import PostgreSQLHandler
# Use .env or environment variables
db = PostgreSQLHandler()
# Or specify directly
db = PostgreSQLHandler(
user='username',
password='password',
host='localhost',
port=5432,
db_name='database'
)
# Usage
databases = db.get_database_list()
tables = db.get_table_list()
results = db.execute_query("SELECT * FROM users LIMIT 5")
```
### 4. Data Processing (Optional)
```python
from qufe.pdhandler import PandasHandler
import pandas as pd
# Initialize handler with default settings
handler = PandasHandler()
# Or with default exclude columns for NA/empty checks
handler = PandasHandler(default_exclude_cols=['id', 'created_at'])
# Compare DataFrames
df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'B': [5, 6], 'C': [7, 8]})
col_dict, comparison = handler.show_col_names([df1, df2])
# Find missing data
na_subset = handler.show_all_na(df1)
# Find problematic rows (uses default_exclude_cols if set)
problem_rows = handler.show_all_na_or_empty_rows(df1)
# Convert lists to tuples in DataFrame
df_with_lists = pd.DataFrame({'col1': [[1, 2], [3, 4]], 'col2': ['a', 'b']})
df_converted = handler.convert_list_to_tuple_in_df(df_with_lists)
```
### 5. Web Automation (Optional)
```python
from qufe.wbhandler import Firefox
# Start browser
browser = Firefox(private_mode=True)
browser.sb.open('https://example.com')
# Network monitoring
browser.inject_network_capture()
# ... perform actions ...
logs = browser.get_network_logs()
# Clean up
browser.quit_driver()
```
### 6. Screen Automation (Optional)
```python
from qufe.interactionhandler import get_screenshot, display_image, get_color_boxes
# Capture screen
screenshot = get_screenshot(100, 100, 800, 600)
display_image(screenshot, is_bgra=True)
# Find colored regions
red_boxes = get_color_boxes(screenshot, (255, 0, 0), tolerance=0.1)
```
## Module-Specific Help
Each module provides detailed help information:
```python
# General help
import qufe
qufe.help()
# Module-specific help
from qufe import dbhandler, pdhandler, wbhandler, interactionhandler
dbhandler.help() # Database operations guide
pdhandler.help() # pandas utilities guide
wbhandler.help() # Browser automation guide
interactionhandler.help() # Screen interaction guide
```
## Dependency Details
| Feature Group | Dependencies | Purpose |
|---------------|--------------|---------|
| `database` | sqlalchemy≥1.3.0, python-dotenv≥0.15.0 | PostgreSQL operations |
| `data` | pandas≥1.1.0, numpy≥1.17.0 | Data processing |
| `web` | seleniumbase≥3.0.0, selenium≥3.141.0 | Browser automation |
| `vision` | opencv-python≥4.1.0, matplotlib≥3.1.0, pyautogui≥0.9.48, mss≥4.0.0 | Screen interaction |
| `jupyter` | ipython≥6.0.0 | Notebook integration |
All versions are set to compatible minimums to avoid conflicts in existing environments.
## Configuration
### Database Setup
qufe supports multiple database configuration methods:
1. **`.env` file (Recommended)**: Works consistently across all environments
2. **Environment variables**: System-level configuration
3. **Direct parameters**: Programmatic configuration
The `.env` approach is recommended because it:
- Works in Jupyter Lab, PyCharm, terminal, and other environments
- Keeps credentials separate from code
- Doesn't require system-level configuration
- Can be easily excluded from version control
### Web Automation Setup
Browser automation requires WebDriver installation:
- **Firefox**: Usually works out of the box (GeckoDriver auto-download)
## Documentation
- **Online docs**: https://qufe.readthedocs.io
- **Local help**: Use `help()` functions in each module
## License
MIT License
## Security & Ethics Guidelines
### Database Security
- Store credentials in `.env` files, never in source code
- Add `.env` to `.gitignore` to prevent credential leaks
- Use environment variables in production environments
- Consider using database connection pooling for production
### Responsible Usage
When using automation and web interaction features, we encourage:
- Respecting website terms of service and limits
- Being mindful of server resources and privacy considerations
- Following ethical practices in data collection and automation
These are personal choices, but we believe technology works best when used responsibly.
## Support and Troubleshooting
### Common Issues
**ImportError on module load:**
```python
# Check what's available
import qufe
qufe.check_dependencies()
# Install missing features
pip install qufe[database] # or other feature groups
```
**Database connection issues:**
```python
# Check configuration
from qufe.dbhandler import help
help() # Shows configuration options
```
**Browser automation problems:**
```python
# Check WebDriver status
from qufe.wbhandler import help
help() # Shows WebDriver requirements
```
### Getting Help
1. **Check module help**: Call `help()` on any module
2. **GitHub Issues**: https://github.com/qufe/qufe/issues
3. **Documentation**: https://qufe.readthedocs.io
Raw data
{
"_id": null,
"home_page": null,
"name": "qufe",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Bongtae Jeon <bongtae.jeon@gmail.com>",
"keywords": "utilities, automation, data-processing, database, file-handling, web-scraping, image-processing",
"author": null,
"author_email": "Bongtae Jeon <bongtae.jeon@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/49/14/43095f31585f4f84fb92d3c0c1fdc298039c271715e8242b2b2233e25fcd/qufe-0.4.1.tar.gz",
"platform": null,
"description": "# qufe\n\nA comprehensive Python utility library for data processing, file handling, database management, and automation tasks.\n\nBorn from the need to streamline repetitive tasks in Jupyter Lab environments, qufe addresses common pain points encountered during interactive development and data exploration work.\n\n## Installation\n\n### Quick Start (Core Features Only)\n```bash\n# Install core functionality with no external dependencies\npip install qufe\n```\n\n### Feature-Specific Installation\nInstall only the features you need:\n\n```bash\n# Database operations (PostgreSQL)\npip install qufe[database]\n\n# Data processing with pandas/numpy \npip install qufe[data]\n\n# Web browser automation\npip install qufe[web]\n\n# Screen capture and image processing\npip install qufe[vision]\n```\n\n## Features Overview\n\n### Always Available (Core Features)\n\n#### Base Utilities (`qufe.base`)\n- **Timestamp handling**: Convert timestamps with timezone support\n- **Code comparison**: Compare code snippets with multiple diff formats \n- **Dynamic imports**: Import Python modules from file paths\n- **List flattening**: Flatten nested structures with configurable depth\n- **Dictionary utilities**: Advanced nested dictionary operations\n\n#### Text Processing (`qufe.texthandler`, `qufe.excludebracket`) \n- **Bracket content removal**: Validate and remove bracketed content\n- **DokuWiki formatting**: Convert data to DokuWiki table format\n- **String utilities**: Advanced search with context extraction\n- **Pretty printing**: Format nested dictionaries and lists\n- **Column display**: Multi-column text formatting with alignment\n\n#### File Operations (`qufe.filehandler`)\n- **Directory traversal**: Recursive file listing with Unicode normalization\n- **Pattern matching**: Find latest files by datetime patterns\n- **Pickle operations**: Simplified Python object persistence\n- **Path utilities**: Safe filename generation and path management\n- **Content extraction**: Text extraction from directory structures\n\n### Optional Features (Require Additional Packages)\n\n#### Database Management (`qufe.dbhandler`) - *[database]*\n- **PostgreSQL integration**: Easy connections using SQLAlchemy\n- **Database exploration**: List databases and tables with metadata\n- **Connection management**: Automatic pooling and cleanup\n- **Environment integration**: .env file and environment variable support\n\n#### Data Analysis (`qufe.pdhandler`) - *[data]*\n- **DataFrame utilities**: Type conversion and structure analysis\n- **Column comparison**: Compare schemas across multiple DataFrames\n- **Missing data detection**: Find NA values and empty strings\n- **Data validation**: Comprehensive quality checks\n\n#### Web Automation (`qufe.wbhandler`) - *[web]*\n- **SeleniumBase integration**: Enhanced browser automation\n- **Network monitoring**: Capture fetch/XHR requests\n- **Element discovery**: Interactive element finding utilities \n- **URL parsing**: Extract and manipulate URL parameters\n- **Multi-browser support**: Firefox implementation with profile detection\n\n#### Screen Interaction (`qufe.interactionhandler`) - *[vision]*\n- **Screen capture**: Full screen or region-specific screenshots\n- **Image processing**: Color detection and comparison algorithms\n- **Mouse automation**: Randomized clicking for natural automation\n- **Progress tracking**: Real-time updates in Jupyter notebooks\n- **Color analysis**: Extract and analyze color information\n\n## Quick Start Guide\n\n### 1. Check Installation Status\n```python\nimport qufe\n\n# See what's available\nqufe.help()\n\n# Check dependencies programmatically\nstatus = qufe.check_dependencies()\nprint(status)\n\n# Get installation commands for missing features\nmissing = qufe.get_missing_dependencies()\nfor module, command in missing.items():\n print(f\"{module}: {command}\")\n```\n\n### 2. Core Features (Always Available)\n```python\nfrom qufe import base, texthandler, filehandler\n\n# Timestamp handling with timezone\nts = base.TS('Asia/Seoul')\nformatted = ts.get_ts_formatted(1640995200)\n\n# File operations\nfh = filehandler.FileHandler()\nfiles = fh.get_tree('/path/to/directory')\n\n# Text processing\ndata = [['Name', 'Age'], ['Alice', '25'], ['Bob', '30']]\ntexthandler.list_to_doku_wiki_table(data)\n\n# Directory exploration\npf = filehandler.PathFinder('/starting/path')\nroot, dirs, files = pf.get_one_depth()\n```\n\n### 3. Database Operations (Optional)\n\n#### Configuration Options\n\n**Option 1: .env file (Recommended)**\n```bash\n# Create .env file in your project root\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_DB=your_database\n```\n\n**Option 2: Environment variables**\n```bash\nexport POSTGRES_USER=your_username\nexport POSTGRES_PASSWORD=your_password\n# ... etc\n```\n\n**Option 3: Direct parameters**\n```python\nfrom qufe.dbhandler import PostgreSQLHandler\n\n# Use .env or environment variables\ndb = PostgreSQLHandler()\n\n# Or specify directly\ndb = PostgreSQLHandler(\n user='username',\n password='password', \n host='localhost',\n port=5432,\n db_name='database'\n)\n\n# Usage\ndatabases = db.get_database_list()\ntables = db.get_table_list()\nresults = db.execute_query(\"SELECT * FROM users LIMIT 5\")\n```\n\n### 4. Data Processing (Optional)\n```python\nfrom qufe.pdhandler import PandasHandler\nimport pandas as pd\n\n# Initialize handler with default settings\nhandler = PandasHandler()\n\n# Or with default exclude columns for NA/empty checks\nhandler = PandasHandler(default_exclude_cols=['id', 'created_at'])\n\n# Compare DataFrames\ndf1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})\ndf2 = pd.DataFrame({'B': [5, 6], 'C': [7, 8]})\ncol_dict, comparison = handler.show_col_names([df1, df2])\n\n# Find missing data\nna_subset = handler.show_all_na(df1)\n\n# Find problematic rows (uses default_exclude_cols if set)\nproblem_rows = handler.show_all_na_or_empty_rows(df1)\n\n# Convert lists to tuples in DataFrame\ndf_with_lists = pd.DataFrame({'col1': [[1, 2], [3, 4]], 'col2': ['a', 'b']})\ndf_converted = handler.convert_list_to_tuple_in_df(df_with_lists)\n```\n\n### 5. Web Automation (Optional)\n```python\nfrom qufe.wbhandler import Firefox\n\n# Start browser\nbrowser = Firefox(private_mode=True)\nbrowser.sb.open('https://example.com')\n\n# Network monitoring\nbrowser.inject_network_capture()\n# ... perform actions ...\nlogs = browser.get_network_logs()\n\n# Clean up\nbrowser.quit_driver()\n```\n\n### 6. Screen Automation (Optional)\n```python\nfrom qufe.interactionhandler import get_screenshot, display_image, get_color_boxes\n\n# Capture screen\nscreenshot = get_screenshot(100, 100, 800, 600)\ndisplay_image(screenshot, is_bgra=True)\n\n# Find colored regions\nred_boxes = get_color_boxes(screenshot, (255, 0, 0), tolerance=0.1)\n```\n\n## Module-Specific Help\n\nEach module provides detailed help information:\n\n```python\n# General help\nimport qufe\nqufe.help()\n\n# Module-specific help\nfrom qufe import dbhandler, pdhandler, wbhandler, interactionhandler\ndbhandler.help() # Database operations guide\npdhandler.help() # pandas utilities guide \nwbhandler.help() # Browser automation guide\ninteractionhandler.help() # Screen interaction guide\n```\n\n## Dependency Details\n\n| Feature Group | Dependencies | Purpose |\n|---------------|--------------|---------|\n| `database` | sqlalchemy\u22651.3.0, python-dotenv\u22650.15.0 | PostgreSQL operations |\n| `data` | pandas\u22651.1.0, numpy\u22651.17.0 | Data processing |\n| `web` | seleniumbase\u22653.0.0, selenium\u22653.141.0 | Browser automation |\n| `vision` | opencv-python\u22654.1.0, matplotlib\u22653.1.0, pyautogui\u22650.9.48, mss\u22654.0.0 | Screen interaction |\n| `jupyter` | ipython\u22656.0.0 | Notebook integration |\n\nAll versions are set to compatible minimums to avoid conflicts in existing environments.\n\n## Configuration\n\n### Database Setup\n\nqufe supports multiple database configuration methods:\n\n1. **`.env` file (Recommended)**: Works consistently across all environments\n2. **Environment variables**: System-level configuration \n3. **Direct parameters**: Programmatic configuration\n\nThe `.env` approach is recommended because it:\n- Works in Jupyter Lab, PyCharm, terminal, and other environments\n- Keeps credentials separate from code\n- Doesn't require system-level configuration\n- Can be easily excluded from version control\n\n### Web Automation Setup\n\nBrowser automation requires WebDriver installation:\n- **Firefox**: Usually works out of the box (GeckoDriver auto-download)\n\n## Documentation\n- **Online docs**: https://qufe.readthedocs.io\n- **Local help**: Use `help()` functions in each module\n\n## License\n\nMIT License\n\n## Security & Ethics Guidelines\n\n### Database Security\n- Store credentials in `.env` files, never in source code\n- Add `.env` to `.gitignore` to prevent credential leaks\n- Use environment variables in production environments\n- Consider using database connection pooling for production\n\n### Responsible Usage\nWhen using automation and web interaction features, we encourage:\n- Respecting website terms of service and limits\n- Being mindful of server resources and privacy considerations\n- Following ethical practices in data collection and automation\n\nThese are personal choices, but we believe technology works best when used responsibly.\n\n## Support and Troubleshooting\n\n### Common Issues\n\n**ImportError on module load:**\n```python\n# Check what's available\nimport qufe\nqufe.check_dependencies()\n\n# Install missing features\npip install qufe[database] # or other feature groups\n```\n\n**Database connection issues:**\n```python\n# Check configuration\nfrom qufe.dbhandler import help\nhelp() # Shows configuration options\n```\n\n**Browser automation problems:**\n```python\n# Check WebDriver status\nfrom qufe.wbhandler import help \nhelp() # Shows WebDriver requirements\n```\n\n### Getting Help\n\n1. **Check module help**: Call `help()` on any module\n2. **GitHub Issues**: https://github.com/qufe/qufe/issues\n3. **Documentation**: https://qufe.readthedocs.io\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A comprehensive Python utility library for data processing, file handling, database management, and automation tasks",
"version": "0.4.1",
"project_urls": {
"Bug Tracker": "https://github.com/qufe/qufe/issues",
"Documentation": "https://qufe.readthedocs.io",
"Homepage": "https://dev.qufe.net/",
"Repository": "https://github.com/qufe/qufe"
},
"split_keywords": [
"utilities",
" automation",
" data-processing",
" database",
" file-handling",
" web-scraping",
" image-processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "d9f25f4b0649b00ce6b6f8322e129adb19ab28ef4ed2bff0b695a0cd25305c87",
"md5": "b415e71beae23f6fe5b110eee2939313",
"sha256": "ac33fea8e009701f7800e72956370bd2cc7d99e923c576d4703b5bd5b2ef8a89"
},
"downloads": -1,
"filename": "qufe-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b415e71beae23f6fe5b110eee2939313",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 36126,
"upload_time": "2025-09-09T01:30:15",
"upload_time_iso_8601": "2025-09-09T01:30:15.736618Z",
"url": "https://files.pythonhosted.org/packages/d9/f2/5f4b0649b00ce6b6f8322e129adb19ab28ef4ed2bff0b695a0cd25305c87/qufe-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "491443095f31585f4f84fb92d3c0c1fdc298039c271715e8242b2b2233e25fcd",
"md5": "12769da76cdfc8787b582c1912ce54d6",
"sha256": "29bf97d772a22c9d6787fa1bf92d1985f5e4911a899fa4a60d9f5932d6290bca"
},
"downloads": -1,
"filename": "qufe-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "12769da76cdfc8787b582c1912ce54d6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 39351,
"upload_time": "2025-09-09T01:30:17",
"upload_time_iso_8601": "2025-09-09T01:30:17.097725Z",
"url": "https://files.pythonhosted.org/packages/49/14/43095f31585f4f84fb92d3c0c1fdc298039c271715e8242b2b2233e25fcd/qufe-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-09 01:30:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "qufe",
"github_project": "qufe",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "qufe"
}
JSON
