# qufe
A comprehensive Python utility library for data processing, file handling, database management, and automation tasks.
## Features
### Core Utilities (`base`)
- **Timestamp handling**: Convert timestamps to datetime objects with timezone support
- **Code comparison**: Compare code snippets with multiple diff formats (simple, unified, ndiff)
- **Dynamic module import**: Import Python modules dynamically from file paths
- **List flattening**: Flatten nested lists with configurable depth
- **Dictionary utilities**: Flatten three-level nested dictionaries with suffix support
### Database Management (`dbhandler`)
- **PostgreSQL integration**: Easy PostgreSQL database connections and queries using SQLAlchemy
- **Database exploration**: List databases and tables with metadata
- **Connection management**: Automatic connection pooling and cleanup
### Text Processing (`texthandler`, `excludebracket`)
- **Bracket content removal**: Remove content within brackets with validation
- **DokuWiki formatting**: Convert lists to DokuWiki table format
- **String search utilities**: Find all occurrences of substrings with context
- **Dictionary printing**: Pretty-print nested dictionaries with indentation
- **Column formatting**: Display lists in multiple columns with alignment
### File Operations (`filehandler`)
- **Directory traversal**: Get file lists and directory trees with Unicode normalization
- **Pattern matching**: Find latest files based on datetime patterns
- **Pickle operations**: Save and load Python objects to/from pickle files
- **Path utilities**: Create unique filenames and sanitize file names
- **Content extraction**: Extract text content from directory structures
### Data Analysis (`pdhandler`)
- **DataFrame utilities**: Convert lists to tuples in pandas DataFrames
- **Column analysis**: Compare column names across multiple DataFrames
- **Missing data detection**: Find rows and columns with NA or empty values
- **Data validation**: Comprehensive data quality checks
### Automation & Screen Interaction (`interactionhandler`)
- **Screen capture**: Take screenshots of full screen or specific regions
- **Image processing**: Color detection, image comparison, and difference highlighting
- **Mouse automation**: Random clicking within regions for automation
- **Progress tracking**: Real-time progress updates in Jupyter notebooks
- **Color analysis**: Extract and analyze color codes from screen regions
### Web Browser Automation (`wbhandler`)
- **SeleniumBase integration**: Enhanced browser automation with custom timeouts
- **Network monitoring**: Capture fetch/XHR requests with JavaScript injection
- **Element discovery**: Interactive element finding with common attribute detection
- **URL parsing**: Extract parameters and values from URLs
- **Multi-browser support**: Chrome and Firefox browser implementations
## Installation
```bash
pip install qufe
```
## Quick Start
### Basic Usage
```python
from qufe import base, texthandler, filehandler
# Timestamp handling
ts = base.TS('Asia/Seoul')
formatted_time = ts.get_ts_formatted(1640995200)
# File operations
fh = filehandler.FileHandler()
files = fh.get_tree('/path/to/directory')
# Text processing
texthandler.print_dict({'key': ['value1', 'value2']})
```
### Database Operations
Before using database operations, you need to configure your PostgreSQL credentials.
#### Option 1: Using .env file (Recommended)
1. Copy the `.env.example` file to `.env` in your project root:
```bash
cp .env.example .env
```
2. Edit the `.env` file with your database credentials:
```bash
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=your_database
```
3. Use the database handler:
```python
from qufe.dbhandler import PostgreSQLHandler
# Credentials will be loaded automatically from .env file
db = PostgreSQLHandler()
databases = db.get_database_list()
tables = db.get_table_list()
```
#### Option 2: Using environment variables
```bash
export POSTGRES_USER=your_username
export POSTGRES_PASSWORD=your_password
export POSTGRES_HOST=localhost
export POSTGRES_PORT=5432
export POSTGRES_DB=your_database
```
#### Option 3: Passing credentials directly
```python
from qufe.dbhandler import PostgreSQLHandler
db = PostgreSQLHandler(
user='your_username',
password='your_password',
host='localhost',
port=5432,
db_name='your_database'
)
```
### Screen Automation
```python
from qufe.interactionhandler import get_sc, display_img
# Capture screen
screenshot = get_sc(100, 100, 800, 600)
display_img(screenshot, is_bgra=True)
```
### Web Browser Automation
```python
from qufe.wbhandler import FireFox
# Start browser session
browser = FireFox()
browser.sb.open('https://example.com')
browser.inject_capture_with_js()
logs = browser.get_capture()
browser.quit_driver()
```
## Configuration
### Database Configuration
For database operations, qufe supports multiple ways to configure PostgreSQL connections:
1. **`.env` file (Recommended)**: Create a `.env` file in your project root with your database credentials
2. **Environment variables**: Set environment variables in your system or shell
3. **Direct parameters**: Pass credentials directly to the constructor
The `.env` file approach is recommended because it:
- Works consistently across different development environments (Jupyter Lab, PyCharm, terminal, etc.)
- Keeps credentials separate from code
- Is easy to manage and doesn't require system-level configuration
## Requirements
- Python 3.8+
- pandas >= 1.3.0
- numpy >= 1.20.0
- sqlalchemy >= 1.4.0
- seleniumbase >= 4.0.0
- opencv-python >= 4.5.0
- matplotlib >= 3.3.0
- pyautogui >= 0.9.50
- mss >= 6.0.0
- python-dotenv >= 1.0.0
## License
MIT License
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## Security & Ethics
### Database Configuration
For security, database credentials should be stored in a `.env` file or environment variables, never hardcoded in your source code. The `.env` file should be added to your `.gitignore` to prevent accidental commits of sensitive information.
### Automation Guidelines
When using screen capture and browser automation features:
- Respect website terms of service and robots.txt
- Be mindful of rate limiting and server load
- Only automate interactions you're authorized to perform
- Consider privacy implications of screen capture functionality
### Web Scraping Ethics
- Always check and comply with robots.txt
- Respect rate limits and implement delays
- Review website terms of service before scraping
- Be considerate of server resources
## Support
If you encounter any problems, please file an issue along with a detailed description.
Raw data
{
"_id": null,
"home_page": null,
"name": "qufe",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Bongtae Jeon <bongtae.jeon@gmail.com>",
"keywords": "utilities, automation, data-processing, database, file-handling, web-scraping, image-processing",
"author": null,
"author_email": "Bongtae Jeon <bongtae.jeon@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/ce/30/00b0f2c0ca9c81a78c4fabba27764880ab63c5484dce3c0dce5e9738edf3/qufe-0.3.1.tar.gz",
"platform": null,
"description": "# qufe\n\nA comprehensive Python utility library for data processing, file handling, database management, and automation tasks.\n\n## Features\n\n### Core Utilities (`base`)\n- **Timestamp handling**: Convert timestamps to datetime objects with timezone support\n- **Code comparison**: Compare code snippets with multiple diff formats (simple, unified, ndiff)\n- **Dynamic module import**: Import Python modules dynamically from file paths\n- **List flattening**: Flatten nested lists with configurable depth\n- **Dictionary utilities**: Flatten three-level nested dictionaries with suffix support\n\n### Database Management (`dbhandler`)\n- **PostgreSQL integration**: Easy PostgreSQL database connections and queries using SQLAlchemy\n- **Database exploration**: List databases and tables with metadata\n- **Connection management**: Automatic connection pooling and cleanup\n\n### Text Processing (`texthandler`, `excludebracket`)\n- **Bracket content removal**: Remove content within brackets with validation\n- **DokuWiki formatting**: Convert lists to DokuWiki table format\n- **String search utilities**: Find all occurrences of substrings with context\n- **Dictionary printing**: Pretty-print nested dictionaries with indentation\n- **Column formatting**: Display lists in multiple columns with alignment\n\n### File Operations (`filehandler`)\n- **Directory traversal**: Get file lists and directory trees with Unicode normalization\n- **Pattern matching**: Find latest files based on datetime patterns\n- **Pickle operations**: Save and load Python objects to/from pickle files\n- **Path utilities**: Create unique filenames and sanitize file names\n- **Content extraction**: Extract text content from directory structures\n\n### Data Analysis (`pdhandler`)\n- **DataFrame utilities**: Convert lists to tuples in pandas DataFrames\n- **Column analysis**: Compare column names across multiple DataFrames\n- **Missing data detection**: Find rows and columns with NA or empty values\n- **Data validation**: Comprehensive data quality checks\n\n### Automation & Screen Interaction (`interactionhandler`)\n- **Screen capture**: Take screenshots of full screen or specific regions\n- **Image processing**: Color detection, image comparison, and difference highlighting\n- **Mouse automation**: Random clicking within regions for automation\n- **Progress tracking**: Real-time progress updates in Jupyter notebooks\n- **Color analysis**: Extract and analyze color codes from screen regions\n\n### Web Browser Automation (`wbhandler`)\n- **SeleniumBase integration**: Enhanced browser automation with custom timeouts\n- **Network monitoring**: Capture fetch/XHR requests with JavaScript injection\n- **Element discovery**: Interactive element finding with common attribute detection\n- **URL parsing**: Extract parameters and values from URLs\n- **Multi-browser support**: Chrome and Firefox browser implementations\n\n## Installation\n\n```bash\npip install qufe\n```\n\n## Quick Start\n\n### Basic Usage\n\n```python\nfrom qufe import base, texthandler, filehandler\n\n# Timestamp handling\nts = base.TS('Asia/Seoul')\nformatted_time = ts.get_ts_formatted(1640995200)\n\n# File operations\nfh = filehandler.FileHandler()\nfiles = fh.get_tree('/path/to/directory')\n\n# Text processing\ntexthandler.print_dict({'key': ['value1', 'value2']})\n```\n\n### Database Operations\n\nBefore using database operations, you need to configure your PostgreSQL credentials.\n\n#### Option 1: Using .env file (Recommended)\n\n1. Copy the `.env.example` file to `.env` in your project root:\n```bash\ncp .env.example .env\n```\n\n2. Edit the `.env` file with your database credentials:\n```bash\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_DB=your_database\n```\n\n3. Use the database handler:\n```python\nfrom qufe.dbhandler import PostgreSQLHandler\n\n# Credentials will be loaded automatically from .env file\ndb = PostgreSQLHandler()\ndatabases = db.get_database_list()\ntables = db.get_table_list()\n```\n\n#### Option 2: Using environment variables\n```bash\nexport POSTGRES_USER=your_username\nexport POSTGRES_PASSWORD=your_password\nexport POSTGRES_HOST=localhost\nexport POSTGRES_PORT=5432\nexport POSTGRES_DB=your_database\n```\n\n#### Option 3: Passing credentials directly\n```python\nfrom qufe.dbhandler import PostgreSQLHandler\n\ndb = PostgreSQLHandler(\n user='your_username',\n password='your_password',\n host='localhost',\n port=5432,\n db_name='your_database'\n)\n```\n\n### Screen Automation\n\n```python\nfrom qufe.interactionhandler import get_sc, display_img\n\n# Capture screen\nscreenshot = get_sc(100, 100, 800, 600)\ndisplay_img(screenshot, is_bgra=True)\n```\n\n### Web Browser Automation\n\n```python\nfrom qufe.wbhandler import FireFox\n\n# Start browser session\nbrowser = FireFox()\nbrowser.sb.open('https://example.com')\nbrowser.inject_capture_with_js()\nlogs = browser.get_capture()\nbrowser.quit_driver()\n```\n\n## Configuration\n\n### Database Configuration\n\nFor database operations, qufe supports multiple ways to configure PostgreSQL connections:\n\n1. **`.env` file (Recommended)**: Create a `.env` file in your project root with your database credentials\n2. **Environment variables**: Set environment variables in your system or shell\n3. **Direct parameters**: Pass credentials directly to the constructor\n\nThe `.env` file approach is recommended because it:\n- Works consistently across different development environments (Jupyter Lab, PyCharm, terminal, etc.)\n- Keeps credentials separate from code\n- Is easy to manage and doesn't require system-level configuration\n\n## Requirements\n\n- Python 3.8+\n- pandas >= 1.3.0\n- numpy >= 1.20.0\n- sqlalchemy >= 1.4.0\n- seleniumbase >= 4.0.0\n- opencv-python >= 4.5.0\n- matplotlib >= 3.3.0\n- pyautogui >= 0.9.50\n- mss >= 6.0.0\n- python-dotenv >= 1.0.0\n\n## License\n\nMIT License\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## Security & Ethics\n\n### Database Configuration\nFor security, database credentials should be stored in a `.env` file or environment variables, never hardcoded in your source code. The `.env` file should be added to your `.gitignore` to prevent accidental commits of sensitive information.\n\n### Automation Guidelines\nWhen using screen capture and browser automation features:\n- Respect website terms of service and robots.txt\n- Be mindful of rate limiting and server load\n- Only automate interactions you're authorized to perform\n- Consider privacy implications of screen capture functionality\n\n### Web Scraping Ethics\n- Always check and comply with robots.txt\n- Respect rate limits and implement delays\n- Review website terms of service before scraping\n- Be considerate of server resources\n\n## Support\n\nIf you encounter any problems, please file an issue along with a detailed description.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A comprehensive Python utility library for data processing, file handling, database management, and automation tasks",
"version": "0.3.1",
"project_urls": {
"Bug Tracker": "https://github.com/qufe/qufe/issues",
"Documentation": "https://qufe.readthedocs.io",
"Homepage": "https://dev.qufe.net/",
"Repository": "https://github.com/qufe/qufe"
},
"split_keywords": [
"utilities",
" automation",
" data-processing",
" database",
" file-handling",
" web-scraping",
" image-processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "3f3f775f63c427beafaa57cf356e347a6a22dbed7928272063fe0b11bf6a8a5c",
"md5": "19dc97f45d7ec26dfa488aff03bc3eeb",
"sha256": "53330407c60050b630b9bdf8dbc294fb462ce93f6e32556f29aeecf10507a9d0"
},
"downloads": -1,
"filename": "qufe-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "19dc97f45d7ec26dfa488aff03bc3eeb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 29456,
"upload_time": "2025-09-02T05:10:16",
"upload_time_iso_8601": "2025-09-02T05:10:16.837643Z",
"url": "https://files.pythonhosted.org/packages/3f/3f/775f63c427beafaa57cf356e347a6a22dbed7928272063fe0b11bf6a8a5c/qufe-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "ce3000b0f2c0ca9c81a78c4fabba27764880ab63c5484dce3c0dce5e9738edf3",
"md5": "268ae2c26242e6380f7558aa6f9b7603",
"sha256": "7d9316f2a373ec920e44f7650f1dd4208a852f19b0c8e51e78fc764690632cbb"
},
"downloads": -1,
"filename": "qufe-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "268ae2c26242e6380f7558aa6f9b7603",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 32314,
"upload_time": "2025-09-02T05:10:17",
"upload_time_iso_8601": "2025-09-02T05:10:17.671311Z",
"url": "https://files.pythonhosted.org/packages/ce/30/00b0f2c0ca9c81a78c4fabba27764880ab63c5484dce3c0dce5e9738edf3/qufe-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-02 05:10:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "qufe",
"github_project": "qufe",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "qufe"
}
JSON
