# Inframe - Screen Context Recording and Querying SDK
A Python SDK for intelligent screen recording, context analysis, and real-time querying. Inframe captures screen activity, processes audio and visual content, and provides an AI-powered interface for understanding your digital workspace.
## Features
- **Real-time Screen Recording**: Native macOS recording with AVFoundation
- **Context-Aware Analysis**: Combines audio transcription with visual content analysis
- **Intelligent Querying**: Ask questions about your screen activity and get AI-powered answers
- **Rolling Buffer**: Maintains recent context for continuous analysis
- **Modular Architecture**: Separate recorders for different applications and contexts
- **Async Processing**: Non-blocking pipeline for smooth operation
- **Cython Optimized**: High-performance core components
## Quick Start
### 1. Install the Package
```bash
pip install inframe
```
### 2. Set Up Environment Variables
```bash
export OPENAI_API_KEY="your-openai-api-key-here"
```
### 3. Basic Usage
```python
import asyncio
from inframe import ContextRecorder, ContextQuery
import os
# Initialize recorder and query system
recorder = ContextRecorder(openai_api_key=os.getenv("OPENAI_API_KEY"))
query = ContextQuery(openai_api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
# Set up screen recording
screen_recorder = recorder.add_recorder(
include_apps=["VS Code", "PyCharm", "Cursor"],
recording_mode="full_screen",
visual_task="Describe visible code, terminal output, and development activity."
)
# Set up Slack monitoring
slack_recorder = recorder.add_recorder(
include_apps=["Slack"],
recording_mode="full_screen",
visual_task="Summarize recent DMs and workspace activity."
)
# Define a query to monitor for specific events
async def on_code_change_requested(result):
if "code change" in result.answer.lower():
print("Boss requested a code change!")
# Handle the request...
query.add_query(
prompt="Has my boss asked for a code change?",
recorder=slack_recorder,
callback=on_code_change_requested,
interval_seconds=5
)
# Start recording and monitoring
await recorder.start()
await query.start()
```
## Core Components
### ContextRecorder
Manages screen recording and context integration:
```python
from inframe import ContextRecorder
# Create recorder
recorder = ContextRecorder(openai_api_key="your-key")
# Add recorder with configuration
recorder_id = recorder.add_recorder(
buffer_duration=30, # seconds to keep in buffer
include_apps=["VS Code", "PyCharm"], # apps to monitor
recording_mode="full_screen",
visual_task="Describe code changes and development activity"
)
# Start recording
await recorder.start(recorder_id)
# Stop recording
await recorder.stop(recorder_id)
```
### ContextQuery
Provides intelligent querying capabilities over recorded content:
```python
from inframe import ContextQuery
query = ContextQuery(openai_api_key="your-key", model="gpt-4o-mini")
# Add a query to monitor the recorder
query_id = query.add_query(
prompt="What was I working on in the last 30 minutes?",
recorder=recorder,
interval_seconds=30
)
# Start monitoring
await query.start(query_id)
```
## Installation
### Prerequisites
- **macOS** (for native screen recording with AVFoundation)
- **Python 3.8+**
- **Screen Recording Permissions** - Grant in System Preferences > Security & Privacy > Privacy > Screen Recording
### Quick Install
```bash
pip install inframe
```
### Development Install
```bash
# For development access, contact the maintainer
# Private repository - requires access permissions
# Create conda environment (recommended)
conda env create -f environment.yml
conda activate inframe
```
## Dependencies
Core dependencies (automatically installed):
- `opencv-python>=4.5.0,<4.9.0` - Video processing
- `numpy>=1.21.0,<2.0.0` - Numerical computing
- `openai>=1.0.0` - AI analysis
- `faster-whisper>=0.7.0` - Speech recognition
- `pyobjc-framework-*` - macOS integration
- `mcp` and `fastmcp` - Model Context Protocol
## Advanced Usage
### Local Context Recording
Use the included CLI tool for quick testing:
```bash
# Record for 30 seconds and print context
python local-inframe/local_context_recorder.py --duration 30 --print-context
# Record with specific apps
python local-inframe/local_context_recorder.py --duration 60 --include-apps "Visual Studio Code" "Cursor" --print-context
```
### Custom Visual Tasks
Define specific analysis tasks for different applications:
```python
# Code review assistant
recorder_id = recorder.add_recorder(
include_apps=["VS Code", "GitHub"],
visual_task="Identify code changes, review comments, and pull request status"
)
# Meeting summarizer
recorder_id = recorder.add_recorder(
include_apps=["Zoom", "Teams"],
visual_task="Summarize meeting topics, participants, and action items"
)
```
### Real-time Monitoring
Set up continuous monitoring with callbacks:
```python
async def on_urgent_message(result):
if "urgent" in result.answer.lower():
print("Urgent message detected!")
# Handle the urgent message
query.add_query(
prompt="Is there an urgent or important message?",
recorder=recorder,
callback=on_urgent_message,
interval_seconds=10
)
```
## Project Structure
```
inframe/
├── inframe/ # Main package
│ ├── __init__.py # Package exports
│ ├── recorder.py # ContextRecorder class
│ └── query.py # ContextQuery class
├── inframe/_src/ # Cython-optimized core (compiled)
│ ├── video_stream.cpython-*.so
│ ├── transcription_pipeline.cpython-*.so
│ ├── context_integrator.cpython-*.so
│ ├── context_querier.cpython-*.so
│ └── tldw_utils.cpython-*.so
└── examples/ # Example implementations
└── simple_agent.py # Basic usage example
```
## Configuration
### Environment Variables
```bash
export OPENAI_API_KEY="your-api-key"
export KMP_DUPLICATE_LIB_OK="TRUE" # For macOS compatibility
```
### Recording Settings
- `buffer_duration`: Seconds to keep in rolling buffer (default: 30)
- `recording_mode`: "full_screen" or "window_only"
- `include_apps`: List of applications to monitor
- `visual_task`: Specific analysis instructions
- `interval_seconds`: Query frequency for monitoring
## Troubleshooting
### Common Issues
1. **Screen Recording Permission Error**
```
❌ Screen recording permission not granted
```
**Solution**: Go to System Preferences > Security & Privacy > Privacy > Screen Recording and add your terminal/IDE.
2. **Import Errors**
```
ModuleNotFoundError: No module named 'src'
```
**Solution**: Install the package properly with `pip install inframe` or `pip install -e .`
3. **OpenAI API Errors**
```
⚠️ No OpenAI API key provided
```
**Solution**: Set your OpenAI API key: `export OPENAI_API_KEY="your-key"`
4. **Recording Stops Unexpectedly**
```
❌ Recording error: Error Domain=AVFoundationErrorDomain
```
**Solution**: Restart your terminal/IDE after granting permissions.
### Debug Mode
Enable verbose logging for troubleshooting:
```python
import logging
logging.basicConfig(level=logging.DEBUG)
```
## Performance Considerations
- **Memory Usage**: Rolling buffers prevent memory accumulation
- **API Costs**: Configurable intervals control OpenAI usage
- **Processing**: Async pipeline ensures non-blocking operation
- **Storage**: Temporary files are automatically cleaned up
- **Cython**: Core components are compiled for performance
## License
This software is proprietary and not open source. For commercial licensing, please contact Ben Geist at bendgeist99@gmail.com.
Raw data
{
"_id": null,
"home_page": null,
"name": "inframe",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "screen recording, context, AI, transcription, video processing",
"author": null,
"author_email": "Inframe <bendgeist99@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/86/ca/01bc6694eedecb12713c475a36e0b904112e8cf2dd750d01aa45df0df8eb/inframe-0.1.4.tar.gz",
"platform": null,
"description": "# Inframe - Screen Context Recording and Querying SDK\n\nA Python SDK for intelligent screen recording, context analysis, and real-time querying. Inframe captures screen activity, processes audio and visual content, and provides an AI-powered interface for understanding your digital workspace.\n\n## Features\n\n- **Real-time Screen Recording**: Native macOS recording with AVFoundation\n- **Context-Aware Analysis**: Combines audio transcription with visual content analysis\n- **Intelligent Querying**: Ask questions about your screen activity and get AI-powered answers\n- **Rolling Buffer**: Maintains recent context for continuous analysis\n- **Modular Architecture**: Separate recorders for different applications and contexts\n- **Async Processing**: Non-blocking pipeline for smooth operation\n- **Cython Optimized**: High-performance core components\n\n## Quick Start\n\n### 1. Install the Package\n```bash\npip install inframe\n```\n\n### 2. Set Up Environment Variables\n```bash\nexport OPENAI_API_KEY=\"your-openai-api-key-here\"\n```\n\n### 3. Basic Usage\n```python\nimport asyncio\nfrom inframe import ContextRecorder, ContextQuery\nimport os\n\n# Initialize recorder and query system\nrecorder = ContextRecorder(openai_api_key=os.getenv(\"OPENAI_API_KEY\"))\nquery = ContextQuery(openai_api_key=os.getenv(\"OPENAI_API_KEY\"), model=\"gpt-4o\")\n\n# Set up screen recording\nscreen_recorder = recorder.add_recorder(\n include_apps=[\"VS Code\", \"PyCharm\", \"Cursor\"],\n recording_mode=\"full_screen\",\n visual_task=\"Describe visible code, terminal output, and development activity.\"\n)\n\n# Set up Slack monitoring\nslack_recorder = recorder.add_recorder(\n include_apps=[\"Slack\"],\n recording_mode=\"full_screen\", \n visual_task=\"Summarize recent DMs and workspace activity.\"\n)\n\n# Define a query to monitor for specific events\nasync def on_code_change_requested(result):\n if \"code change\" in result.answer.lower():\n print(\"Boss requested a code change!\")\n # Handle the request...\n\nquery.add_query(\n prompt=\"Has my boss asked for a code change?\",\n recorder=slack_recorder,\n callback=on_code_change_requested,\n interval_seconds=5\n)\n\n# Start recording and monitoring\nawait recorder.start()\nawait query.start()\n```\n\n## Core Components\n\n### ContextRecorder\nManages screen recording and context integration:\n\n```python\nfrom inframe import ContextRecorder\n\n# Create recorder\nrecorder = ContextRecorder(openai_api_key=\"your-key\")\n\n# Add recorder with configuration\nrecorder_id = recorder.add_recorder(\n buffer_duration=30, # seconds to keep in buffer\n include_apps=[\"VS Code\", \"PyCharm\"], # apps to monitor\n recording_mode=\"full_screen\", \n visual_task=\"Describe code changes and development activity\"\n)\n\n# Start recording\nawait recorder.start(recorder_id)\n\n# Stop recording\nawait recorder.stop(recorder_id)\n```\n\n### ContextQuery\nProvides intelligent querying capabilities over recorded content:\n\n```python\nfrom inframe import ContextQuery\n\nquery = ContextQuery(openai_api_key=\"your-key\", model=\"gpt-4o-mini\")\n\n# Add a query to monitor the recorder\nquery_id = query.add_query(\n prompt=\"What was I working on in the last 30 minutes?\",\n recorder=recorder,\n interval_seconds=30\n)\n\n# Start monitoring\nawait query.start(query_id)\n```\n\n## Installation\n\n### Prerequisites\n- **macOS** (for native screen recording with AVFoundation)\n- **Python 3.8+**\n- **Screen Recording Permissions** - Grant in System Preferences > Security & Privacy > Privacy > Screen Recording\n\n### Quick Install\n```bash\npip install inframe\n```\n\n### Development Install\n```bash\n# For development access, contact the maintainer\n# Private repository - requires access permissions\n\n# Create conda environment (recommended)\nconda env create -f environment.yml\nconda activate inframe\n```\n\n## Dependencies\nCore dependencies (automatically installed):\n- `opencv-python>=4.5.0,<4.9.0` - Video processing\n- `numpy>=1.21.0,<2.0.0` - Numerical computing\n- `openai>=1.0.0` - AI analysis\n- `faster-whisper>=0.7.0` - Speech recognition\n- `pyobjc-framework-*` - macOS integration\n- `mcp` and `fastmcp` - Model Context Protocol\n\n## Advanced Usage\n\n### Local Context Recording\nUse the included CLI tool for quick testing:\n\n```bash\n# Record for 30 seconds and print context\npython local-inframe/local_context_recorder.py --duration 30 --print-context\n\n# Record with specific apps\npython local-inframe/local_context_recorder.py --duration 60 --include-apps \"Visual Studio Code\" \"Cursor\" --print-context\n```\n\n### Custom Visual Tasks\nDefine specific analysis tasks for different applications:\n\n```python\n# Code review assistant\nrecorder_id = recorder.add_recorder(\n include_apps=[\"VS Code\", \"GitHub\"],\n visual_task=\"Identify code changes, review comments, and pull request status\"\n)\n\n# Meeting summarizer \nrecorder_id = recorder.add_recorder(\n include_apps=[\"Zoom\", \"Teams\"],\n visual_task=\"Summarize meeting topics, participants, and action items\"\n)\n```\n\n### Real-time Monitoring\nSet up continuous monitoring with callbacks:\n\n```python\nasync def on_urgent_message(result):\n if \"urgent\" in result.answer.lower():\n print(\"Urgent message detected!\")\n # Handle the urgent message\n\nquery.add_query(\n prompt=\"Is there an urgent or important message?\",\n recorder=recorder,\n callback=on_urgent_message,\n interval_seconds=10\n)\n```\n\n## Project Structure\n\n```\ninframe/\n\u251c\u2500\u2500 inframe/ # Main package\n\u2502 \u251c\u2500\u2500 __init__.py # Package exports\n\u2502 \u251c\u2500\u2500 recorder.py # ContextRecorder class\n\u2502 \u2514\u2500\u2500 query.py # ContextQuery class\n\u251c\u2500\u2500 inframe/_src/ # Cython-optimized core (compiled)\n\u2502 \u251c\u2500\u2500 video_stream.cpython-*.so\n\u2502 \u251c\u2500\u2500 transcription_pipeline.cpython-*.so\n\u2502 \u251c\u2500\u2500 context_integrator.cpython-*.so\n\u2502 \u251c\u2500\u2500 context_querier.cpython-*.so\n\u2502 \u2514\u2500\u2500 tldw_utils.cpython-*.so\n\u2514\u2500\u2500 examples/ # Example implementations\n \u2514\u2500\u2500 simple_agent.py # Basic usage example\n```\n\n## Configuration\n\n### Environment Variables\n```bash\nexport OPENAI_API_KEY=\"your-api-key\"\nexport KMP_DUPLICATE_LIB_OK=\"TRUE\" # For macOS compatibility\n```\n\n### Recording Settings\n- `buffer_duration`: Seconds to keep in rolling buffer (default: 30)\n- `recording_mode`: \"full_screen\" or \"window_only\"\n- `include_apps`: List of applications to monitor\n- `visual_task`: Specific analysis instructions\n- `interval_seconds`: Query frequency for monitoring\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Screen Recording Permission Error**\n ```\n \u274c Screen recording permission not granted\n ```\n **Solution**: Go to System Preferences > Security & Privacy > Privacy > Screen Recording and add your terminal/IDE.\n\n2. **Import Errors**\n ```\n ModuleNotFoundError: No module named 'src'\n ```\n **Solution**: Install the package properly with `pip install inframe` or `pip install -e .`\n\n3. **OpenAI API Errors**\n ```\n \u26a0\ufe0f No OpenAI API key provided\n ```\n **Solution**: Set your OpenAI API key: `export OPENAI_API_KEY=\"your-key\"`\n\n4. **Recording Stops Unexpectedly**\n ```\n \u274c Recording error: Error Domain=AVFoundationErrorDomain\n ```\n **Solution**: Restart your terminal/IDE after granting permissions.\n\n### Debug Mode\nEnable verbose logging for troubleshooting:\n\n```python\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n```\n\n## Performance Considerations\n\n- **Memory Usage**: Rolling buffers prevent memory accumulation\n- **API Costs**: Configurable intervals control OpenAI usage\n- **Processing**: Async pipeline ensures non-blocking operation\n- **Storage**: Temporary files are automatically cleaned up\n- **Cython**: Core components are compiled for performance\n\n## License\n\nThis software is proprietary and not open source. For commercial licensing, please contact Ben Geist at bendgeist99@gmail.com. \n",
"bugtrack_url": null,
"license": "Proprietary",
"summary": "Screen Context Recording and Querying SDK",
"version": "0.1.4",
"project_urls": {
"Documentation": "https://github.com/BenDGeist/inframe-public",
"Source": "https://github.com/BenDGeist/inframe-public"
},
"split_keywords": [
"screen recording",
" context",
" ai",
" transcription",
" video processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "760afc213f4c4e7e7e36c166c14f359f1b7eaec7f8543954a1168aa08cb1900c",
"md5": "b4f7a47434f2df8da3963ca44ddbd75e",
"sha256": "b89b08649e50cae832be927a3fe38f61f3929dd022655de732c506466440ff25"
},
"downloads": -1,
"filename": "inframe-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b4f7a47434f2df8da3963ca44ddbd75e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11900,
"upload_time": "2025-07-22T03:57:16",
"upload_time_iso_8601": "2025-07-22T03:57:16.603934Z",
"url": "https://files.pythonhosted.org/packages/76/0a/fc213f4c4e7e7e36c166c14f359f1b7eaec7f8543954a1168aa08cb1900c/inframe-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "86ca01bc6694eedecb12713c475a36e0b904112e8cf2dd750d01aa45df0df8eb",
"md5": "6280c340430b3774a3d35db9303fd145",
"sha256": "fdf8a530b9932938f0a22cbf546c7e12919bfa8ee440f2c0155745eb23cc51bc"
},
"downloads": -1,
"filename": "inframe-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "6280c340430b3774a3d35db9303fd145",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 30279,
"upload_time": "2025-07-22T03:57:17",
"upload_time_iso_8601": "2025-07-22T03:57:17.870123Z",
"url": "https://files.pythonhosted.org/packages/86/ca/01bc6694eedecb12713c475a36e0b904112e8cf2dd750d01aa45df0df8eb/inframe-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-22 03:57:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BenDGeist",
"github_project": "inframe-public",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "opencv-python",
"specs": [
[
"<",
"4.9.0"
],
[
">=",
"4.5.0"
]
]
},
{
"name": "mss",
"specs": [
[
">=",
"9.0.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"<",
"2.0.0"
],
[
">=",
"1.21.0"
]
]
},
{
"name": "openai",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "pillow",
"specs": [
[
">=",
"10.0.0"
]
]
},
{
"name": "faster-whisper",
"specs": [
[
">=",
"0.7.0"
]
]
},
{
"name": "transformers",
"specs": [
[
">=",
"4.20.0"
]
]
},
{
"name": "jinja2",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "torch",
"specs": [
[
">=",
"1.11.0"
]
]
},
{
"name": "fire",
"specs": [
[
">=",
"0.4.0"
]
]
},
{
"name": "pyobjc-core",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "pyobjc-framework-Cocoa",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "pyobjc-framework-AVFoundation",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "pyobjc-framework-Quartz",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "pyobjc-framework-ScreenCaptureKit",
"specs": [
[
">=",
"9.0"
]
]
},
{
"name": "mcp",
"specs": []
},
{
"name": "fastmcp",
"specs": []
},
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-asyncio",
"specs": []
},
{
"name": "inframe",
"specs": []
}
],
"lcname": "inframe"
}
JSON
