# Pipulate: Local First AI SEO Software
<!-- Don't forget to sync_ascii_art.py -->
> **Your data. Your AI. Your machine. Your control.**
> No subscriptions, no vendor lock-in, no cloud costs.
## 🚀 Quick Start for Impatient People
**Want to skip the philosophy and just see what this does?**
```bash
# 1. Install Nix (one-time setup)
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
# 2. Close and reopen your terminal, then:
curl -L https://pipulate.com/install.sh | sh
# 3. Launch it
cd ~/pipulate && nix develop
```
**What you get:** A local web app at `http://localhost:5001` with step-by-step workflows, integrated AI chat, and a JupyterLab instance at `http://localhost:8888`. No cloud required.
**Success looks like:** Two browser tabs auto-open showing the Pipulate interface and JupyterLab.
## 💡 What Can You Actually Build?
**Real examples of what people create with Pipulate:**
### 🔍 SEO Workflows
- **Keyword Research Pipeline**: Input seed keywords → AI expansion → competition analysis → export spreadsheet
- **Content Gap Analysis**: Compare your site vs competitors → identify missing topics → prioritized content calendar
- **Technical SEO Audits**: Crawl site → check Core Web Vitals → generate action items → track fixes
### 📊 Data Processing Workflows
- **CSV Data Cleaning**: Upload messy data → standardize formats → remove duplicates → validate results
- **API Data Collection**: Connect to APIs → fetch data in batches → transform to consistent format → store locally
- **Report Generation**: Combine multiple data sources → apply business rules → create branded reports
### 🤖 AI-Assisted Workflows
- **Content Creation Pipeline**: Research topics → generate outlines → write drafts → optimize for SEO
- **Data Analysis Helper**: Upload spreadsheet → AI suggests insights → create visualizations → export findings
**Key advantage:** Each workflow is a guided, step-by-step process that non-technical users can run repeatedly, while developers can customize the Python code behind each step.
### Meet Chip O'Theseus <!-- key: pipulate-welcome-banner -->
```
╔═════════════════════════════════════════════════════════════════════════╗ Chip O'What?
║ 🎭 PIPULATE: LOCAL-FIRST AI SEO SOFTWARE & DIGITAL WORKSHOP ║ , O
║ ────────────────────────────────────────────────────────────────────── ║ \\ . O
║ ║ |\\/| o
║ 💬 Chip O'Theseus: "Welcome to your sovereign computing environment!" ║ / " '\
║ ║ . . .
║ 🌟 Where Python functions become HTML elements... ║ / ) |
║ 🌟 Where workflows preserve your creative process... ║ ' _.' |
║ 🌟 Where AI integrates locally and globally... ║ '-'/ \
╚═════════════════════════════════════════════════════════════════════════╝
```
## AI On Rails: Structured Workflows for Any AI <!-- key: about-pipulate -->
**The Challenge with Agentic AI:** Powerful but unpredictable—you never know what you're gonna get.
**The Pipulate Approach:** Structured workflows that can leverage **any AI**—local, cloud, or hybrid—while maintaining complete visibility and control.
Think of it as putting guardrails on AI assistance. Instead of asking an AI to "figure it out," domain experts create step-by-step workflows that guide AI through proven processes. The AI gets structure, you get predictable results.
**Pipulate: Your AI Swiss Army Knife:** Whether you prefer local privacy, cloud power, or hybrid approaches, Pipulate provides the framework. Use local models for sensitive work, cloud APIs for heavy lifting, or both in the same workflow—your choice, your control.
```
🤖 AGENTIC MODE (Chaos) 🚂 AI ON RAILS (Pipulate)
═══════════════════════ ══════════════════════════
💥 GOES OFF 📊 LINEAR WORKFLOWS
HALF-COCKED! BY DOMAIN EXPERTS
│ │
▼ ▼
╔════════════════════╗ ┌─────────────────────┐
║ 🌪️ WILLY NILLY 🎲 ║ │ Step 1: Analyze▸ │
║ ║ VS │ Step 2: Process▸ │
║ Unpredictable ║ │ Step 3: Report▸ │
║ Results ║ │ Step 4: Export▸ │
╚════════════════════╝ └─────────────────────┘
│ │
▼ ▼
☁️ Trains Frontier Models 🏠 Keeps Domain Expertise Local
```
1. 🖥️ **Runs locally** like a desktop app using modern web technologies
2. 🐍 **Simple linear workflow** approach powered by HTMX for seamless interactivity
3. 📓 **Transforms Jupyter Notebooks** into production-ready, step-by-step workflows
4. 🤖 **Integrated AI assistance** using your own local models with complete privacy
5. 🔧 **Reproducible environments** with Nix that work identically across all platforms
6. 🎯 **Perfect for SEO practitioners** who want to turn technical expertise into guided, reusable workflows
--------------------------------------------------------------------------------
## What is Pipulate?
Pipulate is a **local-first, single-tenant desktop app framework** featuring AI-assisted, step-by-step workflows. Designed to feel like an Electron app, it uniquely runs a full, reproducible Linux environment within a project folder using Nix, ensuring consistency across macOS, Linux, and Windows (via WSL).
### Desktop App Architecture: Electron vs Pipulate <!-- key: desktop-app-architecture-comparison -->
<!-- START_ASCII_ART: desktop-app-architecture-comparison -->
```
🖥️ ELECTRON PATTERN 🌐 PIPULATE PATTERN
═══════════════════════ ═══════════════════════
┌─────────────────────────┐ ┌─────────────────────────┐
│ ELECTRON APP │ │ PIPULATE SETUP │
├─────────────────────────┤ ├─────────────────────────┤
│ ┌─────┐ ┌─────┐ ┌─────┐ │ │ ┌─────────────────────┐ │
│ │.exe │ │.dmg │ │.deb │ │ │ │ install.sh │ │
│ └─────┘ └─────┘ └─────┘ │ │ │ (Works on ALL OSes) │ │
│ Per-OS Installers │ │ └─────────────────────┘ │
└───────────┬─────────────┘ └───────────┬─────────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐ This is new.
│ 📱 Native Window │ │ 🖥️ Terminal Console │ , O
│ ┌─────────────────┐ │ │ ┌─────────────────────┐ │ \\ . O
│ │ Web Browser │ │ │ │ nix develop │ │ |\\/| o
│ │ (Bundled) │ │ │ │ Starting servers... │ │ / " '\
│ │ ┌───────────┐ │ │ │ │ ✓ JupyterLab ready │ │ . . .
│ │ │ │ │ │ │ │ ✓ Pipulate ready │ │ / ) |
│ │ │ HTML │ │ │ │ └─────────────────────┘ │ ' _.' |
│ │ │ CSS │ │ │ + └─────────────────────────┘ '-'/ \
│ │ │ JS │ │ │ │
│ │ │ │ │ │ ▼
│ │ └───────────┘ │ │ ┌─────────────────────────┐
│ └─────────────────┘ │ │ 🌐 Regular Browser │
│ │ │ │ ┌─────────────────────┐ │
│ ▼ │ │ │ localhost:5001 │ │
│ ┌─────────────────┐ │ │ │ ┌─────────────────┐ │ │
│ │ Node.js │ │ │ │ │ Python/HTMX │ │ │
│ │ Runtime │ │ │ │ │ Workflows │ │ │
│ └─────────────────┘ │ │ │ │ Local AI │ │ │
└─────────────────────────┘ │ │ └─────────────────┘ │ │
│ └─────────────────────┘ │
✅ Feels like native app └─────────────────────────┘
❌ Multiple installers needed
❌ Platform-specific builds ✅ Universal installer
❌ Update distribution complexity ✅ Auto-updates via git
✅ Same experience all OSes
✅ Complete reproducibility
```
<!-- END_ASCII_ART: desktop-app-architecture-comparison -->
### The Magnum Opus: Computing Sovereignty <!-- key: magnum-opus-computing-sovereignty -->
This isn't just another framework — it's a **deliberate culmination** of decades of tech evolution insights. Pipulate represents the "third act" approach to development (3rd time's the charm): choosing the **most durable and lovable** parts of the modern tech stack while rejecting the exhausting hamster wheel of framework churn.
If you are not an Empire builder and prefer craftsmanship over the rat race and want to build tools that last, then Pipulate may be for you. Pipulate embodies that philosophy — maximum creative freedom with minimum technical debt, recapturing *that old Webmaster feeling.*
### Core Philosophy: Local-First, WET, and AI-Augmented
#### Breaking Free: Durable Foundations for Any Approach <!-- key: breaking-free-framework-churn -->
<!-- START_ASCII_ART: breaking-free-framework-churn -->
```
🎡 THE FRAMEWORK CHURN CYCLE 🏰 COMPUTING SOVEREIGNTY
═══════════════════════════════ ═══════════════════════════
React → Vue → Angular → Svelte 🗽 Your Hardware
↑ ↓ 🗽 Your Data
Webpack ← Next.js ← Vite ← Remix 🗽 Your AI Choice
↑ ↓ 🗽 Your Code
Docker → K8s → Cloud → Serverless 🗽 Your Schedule
😵💫 Endless Learning 🗽 Your Hardware
💸 Migration Fatigue 🗽 Your Data
🔒 Platform Lock-in 🗽 Your AI Choice
📈 Growing Complexity 🗽 Your Code
🗽 Your Schedule
WITH
✨ Durable Tools:
🏃♂️ JUMP OFF THE WHEEL • Python (30+ years)
↓ • SQLite (built-in)
┌─────────────┐ • HTML/HTTP (timeless)
│ PIPULATE │ • Nix (reproducible)
│ Local-First │ • Cloud APIs (by choice)
│+ Any Cloud │
└─────────────┘ 🎯 Third Act Philosophy:
"Choose tools that will
outlast any framework"
```
<!-- END_ASCII_ART: breaking-free-framework-churn -->
- **Local-First Sovereignty:** Your data, code, and AI run on your hardware by default—extending to cloud services when you choose. This guarantees privacy, eliminates surprise costs, and gives you complete control over when and how to scale.
- **WET Workflows, DRY Framework:** Workflows are intentionally "WET" (explicit & step-by-step) for maximum clarity and customizability—perfectly mirroring Jupyter Notebooks. The underlying framework is "DRY" for efficiency.
- **The AI Advantage:** AI makes WET practical. Tedious code maintenance and refactoring, once a weakness of WET, is now an area where AI excels, turning repetition into a strength for rapid, context-aware development. Our breakthrough **Workflow Reconstruction System** exemplifies this: intelligent AST-based transplantation of workflow components eliminates traditional OOP inheritance complexity while maintaining perfect code precision.
- **Radical Transparency ("Know EVERYTHING!"):** We reject opaque enterprise patterns in favor of complete observability. State is managed in transparent SQLite tables and JSON blobs, making the entire system intuitive and debuggable. No black boxes, ever.
- **Reproducibility with Nix:** Nix Flakes provide a perfect, reproducible Linux environment on macOS, Linux, and Windows (WSL), solving the "works on my machine" problem.
- **Future-Proof Stack:** We rely on durable standards: Python, SQLite, HTML, and HTMX. This is a framework built to last.
### Primary Goals
1. **Empower End-Users (e.g., SEO Practitioners):** Enable non-programmers to run powerful, AI-guided workflows (often ported from Jupyter Notebooks) without needing to interact with Python code directly.
2. **Serve Developers:** Provide a simple, reproducible environment for building these workflows, leveraging integrated tooling like Jupyter, local LLMs, and a streamlined web framework.
--------------------------------------------------------------------------------
## The Technical Stack: Simple Yet Powerful
Pipulate's WET philosophy extends to its technology choices, favoring simple, durable tools over complex abstractions:
## *Not On My Machine* Problem Fixed <!-- key: not-on-my-machine-problem-fixed -->
> The Cloud's popularity has been driven in part by developers not wanting to maintain multiple codebases or installers per OS. Thanks to Nix, that's all fixed.
* **Nix Flakes:** Manages dependencies and creates reproducible environments, ensuring consistency across developers and operating systems, with optional CUDA support. E.g. Is this a Linux-thing you're reading about here? A Windows thing? A Mac thing? The answer is: YES!!! All of the above — and if you've got cool acceleration hardware, it will even take advantage and utilize that too. Best of all worlds.
```
____ _ _ .--. ___________
| _ \ __ _ _ ____ _(_)_ __ (_)_ __ ,--./,-. |o_o | | | |
| | | |/ _` | '__\ \ /\ / / | '_ \| \ \/ / / # \ |:_/ | | | |
| |_| | (_| | | \ V V /| | | | | |> < | | // \ \ |_____|_____|
|____/ \__,_|_| \_/\_/ |_|_| |_|_/_/\_\ \ / (| | ) | | |
`._,._,' /'\_ _/`\ | | |
Solving the "Not on my machine" problem well. \___)=(___/ |_____|_____|
```
**Nix serves as the "Noah's Ark"** — preserving this perfect focus in a reproducible environment that works identically across all platforms. Once you've locked in the focus, it lasts for years or decades, all bottled up in infrastructure-as-code.
## Other Key Technologies Used
Pipulate integrates a carefully selected set of tools aligned with its philosophy:
* **FastHTML:** A Python web framework prioritizing simplicity. It generates HTML directly from Python objects (no template language like Jinja2) and minimizes JavaScript by design, working closely with HTMX. It's distinct from API-focused frameworks like FastAPI. The Python function-naming *is the HTML-template language.*
### The New LAMP Stack: Evolution in Simplicity <!-- key: new-lamp-stack-comparison -->
<!-- START_ASCII_ART: new-lamp-stack-comparison -->
```
🏛️ ORIGINAL LAMP STACK (2000s) 🚀 NEW LAMP STACK (2025)
═══════════════════════════════ ═══════════════════════════
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ 🐧 L: Linux │ │ 🐧 L: Linux + Nix │
│ Single OS, manual setup │ │ Reproducible everywhere │
├─────────────────────────────┤ ├─────────────────────────────┤
│ 🌐 A: Apache │ │ ⚡ A: ASGI │
│ Static config, restarts │ │ Async, hot reload │
├─────────────────────────────┤ ├─────────────────────────────┤
│ 🗄️ M: MySQL │ │ 📊 M: MiniDataAPI │
│ Complex queries, joins │ │ Python-native simplicity│
├─────────────────────────────┤ ├─────────────────────────────┤
│ 🔧 P: PHP │ │ 🐍 P: Python + FastHTML │
│ Mix of HTML/logic │ │ + HTMX │
└─────────────────────────────┘ └─────────────────────────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────────┐
│ 🏢 Enterprise │ │ 🏠 Local-First │
│ Complexity │ │ Sovereignty │
│ │ │ │
│ • Multi-server │ │ • Single machine │
│ • Load balancers │ │ • Integrated AI │
│ • Database clusters │ VS │ • SQLite simplicity │
│ • DevOps overhead │ │ • Nix reproducibility │
│ • Cloud lock-in │ │ • Flexible deployment │
└─────────────────────┘ └─────────────────────────┘
🎯 One person understands 🎯 One person controls
part of the system the entire system
```
<!-- END_ASCII_ART: new-lamp-stack-comparison -->
The original LAMP stack was beautiful in its simplicity — one person could understand and manage the whole stack. But it got bloated with enterprise patterns, microservices, and distributed complexity.
Pipulate brings back that **"one person, full stack"** philosophy with modern tools:
- **L**inux + **N**ix: Reproducible environments across all platforms
- **A**SGI: Modern async server interface, future-proofed for performance
- **M**iniDataAPI: Universal SQL simplifier close to Python's core data structures
- **P**ython + FastHTML + HTMX: The new web development paradigm
This isn't just simpler — it's more powerful, giving you complete environment reproducibility, local AI integration, server-side state management, and future-proofed skills.
### The Lens Stack: Focused Architecture <!-- key: the-lens-stack -->
Pipulate's technology choices form **aligned lenses** that focus ideas from abstraction to actualization. Each lens must be **ground and polished** without misaligning the focus:
```
Universal Translator of Abstractions clarify into implementations
Spoken Language to Code by each lens being simple and transparent.
Idea --> Lens 1 --> Lens 2 --> Lens 3 -> Lens 4 -> Lens 5 -> Lens 6
-----> ,--.
---> ,' `.---------> ,--.
--> / \------> ,' `.-------> ,--. ,-.
o -> / Linux \----> / http \----> ,'_hx `.--->,' `. ,-.
/|\ ( HARDWARE )--> ( PROTOCOL )--> ( LINGUA )->( UI/UX )->(APP)->(git)
/ \ -> \ Nix /----> \ html /----> `..py ,'--->`. ,' `-'
--> \ /------> `. ,'-------> `--' `-' And so on
---> `. ,'---------> `--' AI Help
-----> `--' AI Help
AI Help
```
We keep lenses minimal, their material either thoroughly pre-trained into the model (Python 3.x, HTMX, etc.) or able to be included in the prompt and easily held in the context window. We've trimmed the cruft — the lens flashes and burrs, and all unnecessary extra lenses (Angular, React, Vue, etc.)
```yaml
HARDWARE:
install.sh: Published on Pipulate.com to initiate magic cookie install
flake.nix: Nix IaC creating a normalized Linux subsystem on any host OS
PROTOCOL:
http: Uvicorn fast Asynchronous Server Gateway Interface (ASGI) web server
html: Uvicorn talks to Python Starlette using anyio & httpx libraries
websocket: static/ws.js provides client bi-directional asynchronous communication
LINGUA:
htmx: static/htmx.js JavaScript library to eliminate most need for JavaScript
Python: .venv/bin/python3.12 latest version AIs are well trained on
UI/UX:
browser: Obviously, but I guess it needs to be said. Like a looser Electron.
fasthtml: static/fasthtml.js for FT Components, Python functions as templating
APP:
app: Flask-style Uvicorn factory instance instantiated by FastHTML fast_app
db: Dict-like DB providing transparent server-side state (server cookies)
pipulate: Pipeline state management, like db but with JSON blob for workflows
```
### Grinding Off the Burrs and Flashes <!-- key: grinding-off-burrs-flashes -->
In lens manufacturing, "flashes" are excess material that squeeze out of molds — unwanted projections that must be ground off. Steve Jobs famously did this twice: adopting Gorilla Glass (grinding off plastic flashes) and rejecting Flash Player (grinding off software bloat).
**Pipulate continues this tradition:**
- **FastHTML**: Grinds off Jinja2 template complexity
- **HTMX**: Grinds off virtual DOM overhead
- **Local AI**: Enables privacy by default, cloud power when desired
- **SQLite**: Grinds off enterprise database complexity
The result: clean, focused tools that do their job without unnecessary cruft.
--------------------------------------------------------------------------------
## From Flask to FastAPI to FastHTML
This is not your father's Python web framework. HTMX changes everything — a marriage made in heaven between Python and the Web, finally turning Python into a first-class citizen for web development. In many use cases such as this one, Python is even preferable to JavaScript in the way it blends Python's formidable ecosystem of packages with workflows.
### The Evolution: Flask → FastAPI → FastHTML <!-- key: the-evolution-flask-fastapi-fasthtml -->
The revolution isn't just another framework — it's eliminating the template layer entirely:
```
🍶 FLASK ERA 🚀 FASTAPI ERA 🌐 FASTHTML ERA
═══════════════ ═══════════════ ══════════════════
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Python │ │ Python │ │ Python │
│ Functions │ │ Functions │ │ Functions │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Jinja2 │ │ Pydantic │ │ HTMX │◄─ Over-the-wire
│ Templates │ │ Models │ │ Fragments │ HTML targeting
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ DOM elements
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ HTML │ │ JSON │ │ HTML │
│ Response │ │ Response │ │ Elements │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
▼ ▼ ▼
🌐 Full Page Reload 📱 Frontend Framework 🎯 DOM Element Updates
(React/Vue/Angular) def Div() = <div>
def Button() = <button>
Template files needed JSON ↔ HTML conversion Python functions ARE
Separate languages Client-side complexity the template language!
```
**The FastHTML Breakthrough:** Python function names directly become HTML elements, eliminating templates and making the server the single source of truth for UI state.
* **HTMX:** Enables dynamic, interactive UIs directly in HTML via attributes, minimizing the need for custom JavaScript. Pipulate uses it for server-rendered HTML updates — *over the wire HTML*-fragments targeting elements of the DOM directly instead of fragile, performance-reducing, framework-dependent JSON. *THIS* is where you *jump off the tech-churn hamsterwheel* and future-proof yourself.
* **MiniDataAPI:** A lightweight layer for interacting with SQLite and other databases. Uses Python dictionaries for schema definition, promoting type safety without the complexity of traditional ORMs — effectively future-proofing your SQL. You lose fancy *join* capabilities but in exchange get the *Python dict interface* as your main persistent database API forever-forward, enabiling instant swapability between SQLite and PostgreSQL (for example).
* **Ollama:** Facilitates running LLMs locally, enabling in-app chat, workflow guidance, and future automation capabilities while ensuring privacy and avoiding API costs. Your local AI (Chip O'Theseus) learns & grows with you, hopping from hardware to hardware as you upgrade — like a genie in a hermitcrab shell. And if that weren't kooky enough — it knows how to make MCP-calls!!! That's right, your friendly localhost AI Chip O'Theseus is also an *MCP client!* Your linear workflows ain't so linear anymore when a single-step can be: "Go out and do whatever."
### The Hybrid Advantage: Best of Both Worlds
**Pipulate isn't anti-cloud—it's pro-choice.** Each workflow step can choose the best tool for the job:
- **Step 1**: Use local AI for sensitive data analysis (privacy-first)
- **Step 2**: Call OpenAI's API for advanced reasoning (cloud power)
- **Step 3**: Process results locally and save to SQLite (data sovereignty)
- **Step 4**: Use Anthropic's API for final review (frontier capabilities)
**This is the Swiss Army knife approach:** Local by default, cloud by choice, with complete visibility into what's happening at each step. Whether you're processing confidential client data (local) or need cutting-edge AI capabilities (cloud), Pipulate gives you the framework to do both seamlessly.
* **SQLite & Jupyter Notebooks:** Foundational tools for data persistence and the workflow development process (porting from notebooks to Pipulate workflows). SQLite is built into Python and really all things — the *get-out-of-tech-liability free card* you didn't know you had. And a full JupyterLab instance is installed side-by-side with Pipulate sharing the same Python `.venv` virtual environment, which is also shared with your preferred AI code editor (Cursor, Windsurf, VSCode, Zed) so... well... uhm, there are no words for when 3 different portals-to-Python share the same environment. You can do such stupid AI-tricks as letting your local LLM and a frontier cloud model *inhabit* the same body (Pipulate) — controlling web browsers together and stuff.
--------------------------------------------------------------------------------
## How to Install Pipulate
### Installation Strategy: Universal First, PyPI Alternative <!-- key: installation-strategy-overview -->
We offer two installation paths that lead to the exact same robust, Nix-managed environment. Choose the path that best fits your experience level and preferences.
```
┌────────────────────────────┐
│ New User on macOS │
└─────────────┬──────────────┘
│
┌───────────────────────┴───────────────────────┐
│ │
▼ ▼
┌──────────────────────────────────┐ ┌───────────────────────────────────────────┐
│ PATH 1: Recommended for Everyone │ │ PATH 2: Alternative for Python Developers │
└──────────────────────────────────┘ └───────────────────────────────────────────┘
│ │
"I want the simplest, most "I prefer managing my command-line
direct way to get this running." tools with standard Python utilities."
│ │
▼ ▼
1. `curl ... [nix]` 1. `brew install pipx` (If needed)
2. `curl ... [pipulate]` 2. `pipx install pipulate`
3. `pipulate install`
│ │
└───────────────┐ ┌───────────────┘
│ │
▼ ▼
┌────────────────────────────┐
│ Nix-Managed Pipulate │
│ Environment │
└────────────────────────────┘
||
(Identical
Result)
```
### PATH 1: Quick Start — Universal Installation (Recommended) <!-- key: quick-start-universal-installation -->
This is the fastest and most universal way to install Pipulate. It has the fewest dependencies and works on any modern Mac, Linux system, or Windows with WSL.
```
📦 Your Machine 🔧 Add Foundation 🚀 Complete Environment
Today with Nix Ready to Go!
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Sad Computer│ Step 1 │ 🏗️ Nix │ Step 2 │ 🎯 Pipulate │
│ Without │ ──────────► │ Foundation │ ──────────► │ + AI + │
│ Nix😢 │ │ Installed │ │ Jupyter │
└─────────────┘ └─────────────┘ └─────────────┘
│
Step 3 │
▼
┌─────────────┐
│ 🌐 Browser │
│ Opens │
│Automatically│
└─────────────┘
Simple as 1-2-3! No Docker, no build steps, works with or without cloud services.
Everything runs locally with complete flexibility and control.
```
**Step 1: Install Nix (One-Time Setup)**
If you don't have it already, install the Nix package manager. It's the system that makes Pipulate's reproducible environment possible.
```bash
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
```
> **Important:** After the Nix installation finishes, you **must close and reopen** your Terminal window.
**Step 2: Run the Pipulate Installer**
Now, run the universal install script. You can give your project a custom name, too.
```bash
# To install with a custom name like "Botifython"
curl -L https://pipulate.com/install.sh | sh -s Botifython
# Or, to install with the default name "pipulate"
curl -L https://pipulate.com/install.sh | sh
```
**Step 3: Launch Pipulate**
Navigate into your new project directory and launch the environment with `nix develop`.
```bash
# cd into the directory you just created
cd ~/Botifython
# Launch Pipulate
nix develop
```
That's it! The server and JupyterLab will start, and the application will open in your browser.
**Running It Again:**
1. You can just forcibly exit out of that Terminal it's running from.
2. Open a new Terminal, and once again:
```bash
cd ~/Botifython
nix develop
```
**The Big Reset (If Necessary):**
Things sometimes go wrong. This is how you do a full Pipulate reset. This will also delete anything you downloaded with Pipulate. Adjust custom install name to what you used.
```bash
rm -rf ~/Botifython
curl -L https://pipulate.com/install.sh | sh -s Botifython
cd ~/Botifython
nix develop
```
Wait for ***BOTH TABS*** to auto-open in your browser.
### 🚨 Installation Troubleshooting
**Common Issues & Solutions:**
| Problem | Solution |
|---------|----------|
| `nix: command not found` | You didn't restart your terminal after Nix installation |
| Browser doesn't open automatically | Manually visit `http://localhost:5001` and `http://localhost:8888` |
| `Permission denied` errors | Make sure you can write to `~/pipulate` directory |
| Port conflicts | Kill processes on ports 5001/8888: `lsof -ti:5001 \| xargs kill -9` |
| Nix build fails | Clear Nix cache: `nix-collect-garbage` then retry |
**System Requirements:**
- **macOS**: 10.15+ (Intel/Apple Silicon)
- **Linux**: Any modern distribution with curl
- **Windows**: WSL2 with Ubuntu 20.04+
- **RAM**: 4GB minimum, 8GB recommended
- **Disk**: 2GB for installation + data storage
- **Network**: Internet connection for initial setup only
---
### PATH 2: Alternative Installation via PyPI (For Python Developers) <!-- key: alternative-installation-pypi -->
If you are a developer comfortable with tools like Homebrew and `pipx`, you can use our PyPI package as a gateway to the same robust installation.
**Step 1: Install `pipx`**
`pipx` is a tool for safely installing Python command-line applications. If you don't have it, you can install it with Homebrew.
```bash
brew install pipx
```
**Step 2: Install the Pipulate CLI**
Use `pipx` to install the `pipulate` command-line tool. This will not cause conflicts with your system Python.
```bash
pipx install pipulate
```
**Step 3: Run the Installer**
Use the command you just installed to set up the full Pipulate application.
```bash
pipulate install
```
This will trigger the same universal installation process, resulting in the exact same robust, Nix-managed environment. To run it in the future, just type `pipulate run`.
These few commands:
- ✅ Updates to the latest version automatically
- ✅ Starts JupyterLab and the Pipulate server
- ✅ Opens web interfaces in your browser
- ✅ Provides a complete, reproducible development environment
**That's it!** You now have a local-first development environment with AI integration, installed via your preferred Python toolchain.
### Installation Process Deep Dive <!-- key: installation-process-diagram -->
Here's what happens behind the scenes during the "magic cookie" installation:
```
User runs install.sh (via curl) Nix Flake Activation & Transformation
┌──────────────────────────────┐ ┌────────────────────────────────────────────┐
│ 1. Download install.sh │ │ 5. User runs 'nix develop' │
│ 2. Download ZIP from GitHub │ │ 6. Flake detects non-git directory │
│ 3. Extract ZIP to ~/AppName │ │ 7. Flake clones repo to temp dir │
│ 4. Download ROT13 SSH key │ │ 8. Preserves app_name.txt, .ssh, .venv │
│ to .ssh/rot │ │ 9. Moves git repo into place │
└─────────────┬────────────────┘ │10. Sets up SSH key for git │
│ │11. Transforms into git repo │
▼ │12. Enables auto-update via git pull │
┌─────────────────────────────────────────────────────────────────────────────┐
│ Result: Fully functional, auto-updating, git-based Pipulate installation │
└─────────────────────────────────────────────────────────────────────────────┘
```
--------------------------------------------------------------------------------
## Chef or Customer? <!-- key: target-audience -->
Are you a Developer or an End User? Chef or Customer? Understanding your audience is crucial for effective development. Pipulate serves two distinct but complementary audiences, much like a restaurant serves both chefs and customers
```
┌──────────────────────────────────────────────────────────┐
│ The Restaurant │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Kitchen (Dev) │ │ Dining Room │ │
│ │ │ │ (End Users) │ │
│ │ │ │ │ │
│ │ 👨🍳 Sous Chef │───recipes───►│ 🍽️ Customers │ │
│ │ 👩🍳 Head Chef │ │ 🏢 Restaurateur │ │
│ │ │ │ │ │
│ │ "How do we make │ │ "I want the best │ │
│ │ pasta you've │ │ pasta I've ever │ │
│ │ never had?" │ │ had in my life" │ │
│ └──────────────────┘ └──────────────────┘ │
└──────────────────────────────────────────────────────────┘
```
### 👨🍳 The Chef (Developer/Technical User)
* **🔧 Workflow Creators:** Build and customize AI-assisted workflows
* **📓 Jupyter Porters:** Convert notebook experiments into guided applications
* **🔍 Technical SEOs:** Create sophisticated, reusable SEO processes
* **⚙️ System Administrators:** Deploy consistent environments across teams
**What Chefs Get:**
- 🎛️ Complete control over the "recipe" (workflow logic)
- 🔄 Reproducible development environment via Nix
- 🏗️ Simple architecture that's easy to understand and modify
- 🧰 Integrated tooling (Jupyter, local LLM, SQLite)
### 🍽️ The Customer (End User/Non-Technical)
* **📈 SEO Practitioners:** Run powerful workflows without coding
* **✍️ Content Creators:** Use AI-assisted processes for optimization
* **📊 Marketing Teams:** Execute consistent SEO strategies
* **🏢 Business Owners:** Access enterprise-level SEO capabilities
**What Customers Get:**
- 🚶♂️ Guided, step-by-step workflow experiences
- 🤖 AI assistance at every step
- 🙈 No need to see or understand the underlying code
- 🎯 Consistent, repeatable results
### 🍝 The Restaurant Analogy
Just as a chef talks about knife techniques while a diner just wants amazing pasta, Pipulate separates the complexity of creation from the simplicity of consumption. Developers craft the workflows, end-users enjoy the results.
## 🎯 Your First 10 Minutes with Pipulate
**After installation succeeds, here's what to expect:**
### What You'll See
1. **Two browser tabs open automatically:**
- `localhost:5001` - Pipulate web interface with navigation menu
- `localhost:8888` - JupyterLab for development/experimentation
2. **In the Pipulate interface:**
- Left sidebar with workflow plugins (Introduction, Profiles, etc.)
- Main area showing step-by-step workflow interface
- Right panel with integrated AI chat (Chip O'Theseus)
3. **Terminal shows:**
```
🚀 Starting Pipulate servers...
✓ FastHTML server ready at http://localhost:5001
✓ JupyterLab ready at http://localhost:8888
✓ Local AI ready for chat assistance
```
### Your Next Steps Depend on Who You Are
**🔍 If you're an SEO practitioner:**
- Click "Introduction" in the left menu for a guided tour
- Try the built-in workflows to see the step-by-step pattern
- Use the AI chat to ask "How do I create a keyword research workflow?"
**👨💻 If you're a developer:**
- Open JupyterLab tab and run the introduction notebook
- Check out `plugins/010_introduction.py` to see workflow code structure
- Try creating a simple workflow: `python helpers/workflow/create_workflow.py`
**🤖 If you're an AI assistant:**
- Focus on the Quick Reference Card above
- Study the Critical Implementation Patterns section
- Review `mcp_tools.py` for MCP protocol capabilities
**🆕 If you're just exploring:**
- Click through the left menu items to see different workflow types
- Ask the AI chat: "What can I build with Pipulate?"
- Try the Introduction workflow to see the step-by-step experience
--------------------------------------------------------------------------------
## The WET Revolution: Why Explicit Code Wins in the AI Era
Pipulate is built on a radical philosophy that challenges programming orthodoxy: **WET (Write Everything Twice) is better than DRY (Don't Repeat Yourself)** when you have AI to help manage it.
### The Universal API Pattern: From Quarks to Code <!-- key: universal-api-pattern -->
At every scale of reality, we see the same pattern: **"lumps of stuff" with APIs** that enable interaction. Quarks combine into atoms, atoms into molecules, cells into organisms, individuals into societies. Each level requires the right **granularity** of interface — not so abstract that you lose control, not so granular that you drown in complexity.
**This is the 80/20 rule of existence:** Handle 80% of interactions gracefully with 20% of the API surface, then handle edge cases as needed. Pipulate applies this principle to code architecture.
### Durable vs. Ephemeral: Building on Bedrock <!-- key: durable-vs-ephemeral -->
The tech industry suffers from "hamster wheel syndrome" — constantly breaking APIs that force migration cycles. React (20+ versions), Node (frequent breaking changes), Angular (complete rewrites). This isn't progress; it's planned obsolescence.
**Pipulate chooses durable foundations:**
- **Linux Kernel**: Version 6 in 30 years
- **Python**: Version 3 in 30 years
- **HTML**: Version 5 and stable
- **HTTP**: Version 3 and backward compatible
These are the "laws of physics" for software — stable APIs that enable compound growth rather than constant rebuilding.
### Why WET Works Now <!-- key: why-wet-works-now -->
Traditional development follows DRY principles, creating abstract, complex systems that are hard to understand and modify. But the world has changed:
1. **🔬 Jupyter Notebooks** promote explicit, literate programming
2. **🤖 AI assistants** excel at managing repetitive code
3. **🏠 Local-first architectures** prioritize clarity over enterprise complexity
4. **🎯 Right Granularity**: WET provides the perfect abstraction level for human AND AI comprehension
```
________________________________
- Like Notebooks / \
- Linear Workflows | It runs proprietary private AI |
- Local & Cloud-free | Workflows from your Local PC?! |
- Chip O'Theseus included \________________________________/
()
HARDWARE PLATFORM LOCAL BROWSER O , Chip O'Theseus
_______________________ __________ _______ o \\ .
| | / Pipulate \Jupyter\__ |\\/|
| Windows, Mac or Linux | | __________________ | See! / " '\ - Radical transparency
| _____ ___ | | | App Name Menu⚙️| |<- - - - -. . . - MCP tool-call control
| _/ Nix \____\_____ | | |------------------| | / ) | - Browser as bot's body
| | | | | | Workflow | Local | | ' _.' |
| | Pipulate <---------> -Step #1 | AI🤖 | | '-'/ \
|__| localhost:5001 |_| | | -Step #2 | Chat | | What, no Docker?
| (AI on Rails🚂) | | | -Step #3 | Help▸ | | What, no React?
|__________________| | |__________|_______| | What, no Cloud?
|______________________|
```
**WET workflows are:**
- **🔍 Observable**: See exactly what's happening at every step
- **🔧 Customizable**: Modify workflows without breaking abstractions
- **🤖 AI-Friendly**: Clear code that AI assistants can easily understand and maintain
- **🚀 Future-Proof**: Built on durable web standards that won't become obsolete
--------------------------------------------------------------------------------
## Developer Setup & Environment Notes
**Nix Environment Activation:** Always run `nix develop` from the `~/pipulate` directory *before* running any project commands (`python server.py`, `pip install`, etc.) in a new terminal. This ensures you are using the correct dependencies defined in `flake.nix`.
**Interactive vs. Quiet Shell:**
**Standard Shell:** `nix develop` (or `nix develop .#default`) runs the startup script (`run-script` defined in `flake.nix`) with welcome messages and service startup. Ideal for general use.
**Quiet Shell:** `nix develop .#quiet` activates the Nix environment *without* running the full startup script or launching services automatically. It only sets up paths and installs pip requirements. Use this for:
- Running specific commands without starting the servers (e.g., `nix develop .#quiet --command python -c "import pandas"`).
- Debugging or interacting with AI assistants where verbose startup output is undesirable.
- Manually running `run-server` or `run-jupyter` (scripts placed in `.venv/bin` by the `shellHook`).
**Dependencies:** System-level dependencies (Python version, libraries like `gcc`, `zlib`) are managed by `flake.nix`. Python package dependencies are managed by `pip` using `requirements.txt` within the Nix-provided environment.
**Source of Truth:** The `flake.nix` file is the definitive source for the development environment setup.
--------------------------------------------------------------------------------
## Architecture & Key Concepts
Pipulate features a distinct architecture designed for its local-first, simple, and observable nature.
### Architecture Overview Diagram <!-- key: architecture-overview-diagram -->
This diagram illustrates the high-level components and their interactions:
```
┌─────────────┐ Like Electron, but full Linux subsystem
│ Browser │ in a folder for macOS and Windows (WSL)
└─────┬───────┘
│ HTTP/WS
▼
┌───────────────────────────────────────┐
│ Nix Flake Shell │ - In-app LLM (where it belongs)
│ ┌───────────────┐ ┌──────────────┐ │ - 100% reproducible
│ │ FastHTML │ │ Ollama │ │ - 100% local
│ │ HTMX App │ │ Local LLM │ │ - 100% multi-OS
│ └───────┬───────┘ └──────────────┘ │
│ │ │
│ ┌─────▼─────┐ ┌────────────┐ │
│ │MiniDataAPI│◄───►│ SQLite DB │ │
│ └───────────┘ └────────────┘ │
└───────────────────────────────────────┘
```
This complete, self-contained environment runs identically on any operating system, providing the foundation for all Pipulate workflows and AI interactions.
---
### Integrated Data Science Environment <!-- key: integrated-data-science-environment -->
Jupyter Notebooks run alongside the FastHTML server, allowing developers to prototype workflows in a familiar environment before porting them to Pipulate's step-based interface for end-users. The same Python virtual environment (`.venv`) is shared, and ad-hoc package installation is supported. If you're using Cursor, VSCode or Windsurf, set your `Ctrl`+`Shift`+`P` "Python: Set Interpreter" to "Enter Interpreter Path" `./pipulate/.venv/bin/python`. You might have to adjust based on the folder you use as your workspace. But then you'll have a Python environment unified between Cursor, JupyterLab and Pipulate.
```
┌──────────────────┐ ┌──────────────────┐
│ Jupyter Lab │ │ FastHTML │
│ Notebooks │ │ Server │
│ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Cell 1 │ │ │ │ Step 1 │ │
│ │ │ │--->│ │ │ │
│ └──────────┘ │ │ └──────────┘ │
│ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Cell 2 │ │ │ │ Step 2 │ │
│ │ │ │--->│ │ │ │
│ └──────────┘ │ │ └──────────┘ │
│ localhost:8888 │ │ localhost:5001 │
└──────────────────┘ └──────────────────┘
```
### Local-First & Single-Tenant Details <!-- key: local-first-single-tenant-details -->
Pipulate manages all state server-side within the local environment (think *local-server cookies*), with optional cloud integration as needed. This approach offers:
* **Privacy & Control:** Data stays local by default, cloud integration when you choose.
* **Full Resource Access:** Utilize local CPU/GPU freely for intensive tasks, plus cloud APIs for heavy lifting.
* **Simplicity:** Eliminates complexities of multi-tenancy while supporting both local and cloud workflows.
* **Observability:** State changes (via DictLikeDB/JSON) are transparent and easily logged (AI greps it there).
### Local-First State Management Benefits <!-- key: local-first-state-management-benefits -->
This detailed view shows how Pipulate's local-first architecture eliminates common web development complexities:
```
┌───────────────────────────────┐ # Benefits of Local-First Simplicity
│ Web Browser │
│ │ - No mysterious client-side state
│ ┌────────────────────┐ │ - No full-stack framework churn
│ │ Server Console │ │ - No complex ORM or SQL layers
│ │ & Web Logs │ │ - No external message queues
│ └─────────┬──────────┘ │ - No build step required
│ ▼ │ - Direct, observable state changes
│ ┌─────────────────────┐ │
│ │ Server-Side State │ │ - Conceptually like local-server-side cookies
│ │ DictLikeDB + JSON ◄──────── Enables the "Know EVERYTHING!" philosophy
│ └─────────────────────┘ │ - AI greps logs/server.log to see app state!
└───────────────────────────────┘
```
### Server-Rendered UI (HTMX) <!-- key: server-rendered-ui-htmx -->
The UI is constructed primarily with server-rendered HTML fragments delivered via HTMX. This minimizes client-side JavaScript complexity.
* FastHTML generates HTML components directly from Python.
* HTMX handles partial page updates based on user interactions, requesting new HTML snippets from the server.
* WebSockets and Server-Sent Events (SSE) provide real-time updates (e.g., for chat, live development reloading).
```
HTMX+Python enables a world-class
Python front-end Web Development environment.
┌─────────────────────┐
│ Navigation Bar │ - No template language (like Jinja2)
├─────────┬───────────┤ - HTML elements are Python functions
Simple Python back-end │ Main │ Chat │ - Minimal custom JavaScript / CSS
HTMX "paints" HTML into │ Area │ Interface │ - No React/Vue/Angular overhead
the DOM on demand ───────► │ │ │ - No "build" process like Svelte
└─────────┴───────────┘ - No virtual DOM, JSX, Redux, etc.
```
With such *minimal surface area* the AI code assistant *knows everything.* LLMs are either pre-trained on the stable, infrequently revved libraries used (Python 3.12, HTMX, or it's all small enough to fit in a 1-shot prompt — yes, the whole core code-base fits in one Gemini Web UI form submit.
--------------------------------------------------------------------------------
## Workflow Patterns & Development
### Pipeline Workflows <!-- key: pipeline-workflows -->
Designed for porting notebook-style processes, workflows are sequences of steps where the state is managed explicitly at each stage and stored persistently (typically as a JSON blob in the `pipeline` table).
* **Resumable & Interrupt-Safe:** Because each step's completion is recorded, workflows can be stopped and resumed.
* **Explicit State Flow:** Data typically passes from one step's output (`done` field) to the next via the `transform` function, simplifying debugging. Patterned on Unix pipes.
* **Good Training Data:** The structured input/output of each step creates valuable data for potentially fine-tuning models.
* **Proprietary Friendly:** Excellent for proprietary domain-experts and fields (competing academic, finances) who *resist* letting their data flow onto the Web for general AI training.
```
┌─────────┐ ┌─────────┐ ┌─────────┐ - Fully customizable steps
│ Step 01 │─piped─►│ Step 02 │─piped─►│ Step 03 │ - Interruption-safe & resumable
└─────────┘ └─────────┘ └─────────┘ - Easily ported from Notebooks
│ │ │ - One DB record per workflow run
▼ ▼ ▼ - Everything stays on your machine
State Saved State Saved Finalized? - Magnitudes simpler than celery
```
### Run All Cells Pattern <!-- key: run-all-cells-pattern -->
**The key insight**: Pipulate workflows use a `run_all_cells()` pattern that directly mirrors Jupyter's "Run All Cells" command. This creates an immediate mental model — each workflow step is like a notebook cell, and the system automatically progresses through them top-to-bottom, just like running all cells in a notebook.
```
📓 JUPYTER NOTEBOOK 🌐 PIPULATE WORKFLOW
═══════════════════ ══════════════════════
[ ] Cell 1: Import data ┌─────────────────────┐
│ │ Step 1: Data Input │
▼ └──────────┬──────────┘
[▶] Cell 2: Process data │ hx_trigger="load"
│ ▼
▼ ┌─────────────────────┐
[ ] Cell 3: Generate report │ Step 2: Processing │
│ └──────────┬──────────┘
▼ │ hx_trigger="load"
[ ] Cell 4: Export results ▼
┌─────────────────────┐
🎯 "Run All Cells" Button ═══► │ Step 3: Export │
Executes top-to-bottom └─────────────────────┘
Same mental model, same execution flow!
But with persistent state, a web UI and
not having to look at the Python code 🚫🐍.
```
### LLM Integration (Ollama) <!-- key: llm-integration-ollama -->
Integration with a local Ollama instance provides AI capabilities without external API calls:
* **Privacy:** Prompts and responses stay local.
* **Cost-Effective:** No per-token charges; run continuously using local resources.
* **Streaming Support:** Real-time interaction via WebSockets.
* **Bounded Context:** Manages conversation history effectively.
* **App State Awareness:** Grepping your server log reveals full application state.
* **Tool Calling:** Local LLM is an MCP client with a growing list of abilities
- Workflow assistance
- Browser automation
- Debugging
```
┌──────────────────┐
│ Local Ollama │ - No API keys needed
│ Server │ - Completely private processing
└────────┬─────────┘
│
│ Streaming via WebSocket
▼
┌──────────────────┐
│ Pipulate App │ - Monitors WS for MCP tool-call commands
│(WebSocket Client)│ - Parses responses in real-time
└────────┬─────────┘
│
│ In-memory or DB backed
▼
┌──────────────────┐
│ Bounded │ - Manages context window (~128k)
│ Chat History │ - Enables RAG / tool integration
└──────────────────┘
```
### Multi-OS & CUDA Support (Nix) <!-- key: multi-os-cuda-support-nix -->
Nix Flakes ensure a consistent environment across Linux, macOS, and Windows (via WSL), optionally leveraging CUDA GPUs if detected.
```
┌──────────────────┐
│ Linux / macOS │ - Write code once, run anywhere
│ Windows (WSL) │ - Consistent dev environment via Nix
└────────┬─────────┘ - As if Homebrew but across all OSes
│
│ Nix manages dependencies
▼
┌──────────────────┐
│ CUDA Support │ - Auto-detects NVIDIA GPU w/ CUDA
│ (if present) │ - Uses GPU for LLM acceleration
└──────────────────┘ - Falls back to CPU if no CUDA
```
### UI Layout <!-- key: ui-layout -->
The application interface is organized into distinct areas:
```
┌─────────────────────────────┐
│ Navigation ◄── Search, Profiles,
├───────────────┬─────────────┤ Apps, Settings
│ │ │
Workflow, ──► Main Area │ Chat │
App UI │ (Pipeline) │ Interface ◄── LLM Interaction
│ │ │
└─────────────────────────────┘
```
### UI Component Hierarchy: Complete DOM Structure with IDs & ARIA Labels
**Critical for AI assistants:** All UI components use semantic IDs and comprehensive ARIA labeling for accessibility and automation.
```
🏠 home (Root Component)
├── 📦 create_outer_container()
│ ├── 🧭 create_nav_group() [id='nav-group', role='navigation', aria-label='Main navigation']
│ │ ├── 🔍 nav_search_container [role='search', aria-label='Plugin search']
│ │ │ ├── Input [id='nav-plugin-search', role='searchbox', aria-label='Search plugins']
│ │ │ └── Div [id='search-results-dropdown', role='listbox', aria-label='Search results']
│ │ ├── 👤 create_profile_menu() [id='profile-dropdown-menu', aria-label='Profile management']
│ │ │ ├── Summary [id='profile-id', aria-label='Profile selection menu']
│ │ │ └── Ul [role='menu', aria-label='Profile options', aria-labelledby='profile-id']
│ │ ├── ⚡ create_app_menu() [id='app-dropdown-menu', aria-label='Application selection']
│ │ │ ├── Summary [id='app-id', aria-label='Application menu']
│ │ │ └── Ul [role='menu', aria-label='Application options', aria-labelledby='app-id']
│ │ ├── 🌍 create_env_menu() [id='env-dropdown-menu', data-testid='environment-dropdown-menu']
│ │ │ ├── Summary [id='env-id', aria-label='Environment selection menu']
│ │ │ └── Ul [role='menu', aria-label='Environment options', aria-labelledby='env-id']
│ │ └── ⚙️ poke_section [id='poke-dropdown-menu']
│ │ ├── Summary [id='poke-summary']
│ │ └── Div [id='nav-flyout-panel']
│ ├── 📱 main-grid
│ │ ├── 📋 create_grid_left() [id='grid-left-content'] → Workflow Steps/Cells Display
│ │ │ ├── content_to_render (Dynamic workflow content)
│ │ │ └── scroll_to_top [id='scroll-to-top-link']
│ │ └── 🤖 create_chat_interface() [id='chat-interface', role='complementary', aria-label='AI Assistant Chat']
│ │ ├── H2 [APP_NAME + ' Chatbot']
│ │ ├── Div [id='msg-list', role='log', aria-label='Chat conversation', aria-live='polite']
│ │ └── Form [role='form', aria-label='Chat input form']
│ │ ├── Textarea [id='msg', role='textbox', aria-label='Chat message input', aria-multiline='true']
│ │ ├── Button [id='send-btn', aria-label='Send message to AI assistant']
│ │ └── Button [id='stop-btn', aria-label='Stop AI response streaming']
│ └── 🔧 HTMX Refresh Listeners
│ ├── Div [id='profile-menu-refresh-listener', hx_target='#profile-dropdown-menu']
│ └── Div [id='app-menu-refresh-listener', hx_target='#app-dropdown-menu']
```
### 🎯 Key HTMX Targets for AI Browser Automation
**Navigation Updates:**
- `#profile-dropdown-menu` - Profile menu refresh target
- `#app-dropdown-menu` - App menu refresh target
- `#search-results-dropdown` - Live search results
- `#nav-flyout-panel` - Settings flyout panel
**Content Areas:**
- `#grid-left-content` - Main workflow display area
- `#msg-list` - Chat conversation history
- `body` - Full page navigation refreshes
**Interactive Elements:**
- `#nav-plugin-search` - Real-time plugin search (300ms delay)
- `#send-btn` / `#stop-btn` - Chat control buttons
- `#scroll-to-top-link` - Scroll navigation aid
This structure enables AI assistants to programmatically interact with all UI components using semantic selectors and ARIA landmarks.
### File Structure
```plaintext
.
├── .cursor/ # Bootstraps Radical Transparency (teaches AI to fish)
│ └── rules/ # Framework rules (01_CRITICAL_PATTERNS.mdc, etc.)
├── .venv/ # Common Python environment for FastHTML, Jupyter & Cursor
├── browser_automation/ # Selenium browser control & DOM capture
│ ├── looking_at/ # Current browser DOM state for AI visibility
│ └── *.py # Google search automation examples
├── cli.py # Command line interface for Pipulate operations
├── common.py # Base Class for DRY CRUD plugin app inheritance (todo)
├── data/
│ └── data.db # AI-accessible SQLite for application state (server cookies)
├── downloads/ # Default location for workflow outputs (e.g., CSVs)
├── helpers/
│ ├── botify
│ │ └── botify_api.ipynb # Git managed massive example notebook, produces docs
│ ├── workflow # Workflow workshop, lots of tools that make WET DRY
│ │ └── create_workflow.py # Example of what might be found there
│ └── prompt_foo.py # Bundles XML code payloads for massive 1-shot AI prompts
├── logs/
│ ├── server-1.log # N-rotations of server log per run per config
│ └── server.log # The server log of most recent run, contains app state
├── static/ # JS, CSS, images, icons
├── plugins/ # Workflow plugins (010_introduction.py, 400_trifecta.py, etc.)
├── pyproject.toml # Python packaging configuration and metadata
├── training/ # Markdown files for AI context/prompts
├── vulture_whitelist.py # Code analysis whitelist for unused code detection
├── flake.nix # Infrastructure as Code & all system-versions for AI
├── LICENSE # It's MIT
├── install.sh # "Magic cookie" installation script (curl | sh)
├── mcp_tools.py # MCP protocol tools - the AI assistant interface
├── notebook_introduction_local.ipynb # Editable (non-auto-updating) copy of botify_api.ipynb
├── README.md # This file
├── requirements.txt # Python dependencies (managed by Nix)
└── server.py # Main application entry point
```
--------------------------------------------------------------------------------
## Critical Implementation Patterns for LLMs
**These patterns are essential for LLMs working with Pipulate and are frequently missed:**
### 1. The Auto-Key Generation Pattern (MOST CRITICAL) <!-- key: auto-key-generation-pattern -->
<!-- START_ASCII_ART: auto-key-generation-pattern -->
```
📝 AUTO-KEY GENERATION FLOW
┌─────────────┐ POST ┌─────────────┐ HX-Refresh ┌─────────────┐
│ Empty Form │ ──────────► │ Server │ ──────────────► │ Page Reload │
│ Submit ⏎ │ /init │ Response │ Header │ Trigger │
└─────────────┘ └─────────────┘ └─────────────┘
▲ │
│ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ User Hits │ ◄─────────── │ Auto-Key │ ◄───────────── │ landing() │
│ Enter Again │ Ready! │ Populated │ Generates │ Method │
└─────────────┘ └─────────────┘ └─────────────┘
```
<!-- END_ASCII_ART: auto-key-generation-pattern -->
When a user hits Enter on an empty key field, this specific sequence occurs:
1. **Form Submission**: POSTs to `/{APP_NAME}/init` with empty `pipeline_id`
2. **Server Response**: The `init` method MUST return an `HX-Refresh` response:
```python
if not user_input:
from starlette.responses import Response
response = Response('')
response.headers['HX-Refresh'] = 'true'
return response
```
3. **Page Reload**: HTMX triggers a full page reload
4. **Auto-Key Population**: The `landing()` method calls `pip.generate_pipeline_key(self)` to populate the input field
5. **User Interaction**: User hits Enter again to start the workflow
### 2. The Chain Reaction Pattern: The `run_all_cells()` Breakthrough
Pipulate uses HTMX-driven step progression powered by the brilliantly named `run_all_cells()` method:
1. **Initial Trigger**: After `init`, the `run_all_cells()` method initializes the workflow just like Jupyter's "Run All Cells"
2. **Perfect Mental Model**: The method name creates immediate understanding — workflows execute top-to-bottom like notebook cells
3. **Step Handlers**: Each step has GET (display) and POST (submit) handlers
4. **Automatic Progression**: Completed steps trigger next step with `hx_trigger="load"`
5. **State Persistence**: Each step stores data in pipeline state
6. **Pedagogical Brilliance**: The naming makes the system instantly intuitive for developers and AI assistants
**Example: The `run_all_cells()` Pattern in Action**
```python
# ✅ CORRECT: Use the run_all_cells() method for workflow initialization
async def init(self, request):
"""Initialize workflow using the run_all_cells pattern"""
return pip.run_all_cells(app_name, steps)
# ❌ ANTI-PATTERN: Manual placeholder creation
async def init(self, request):
"""Manual approach — harder to understand and maintain"""
first_step_id = steps[0].id
return Div(
Div(id=first_step_id, hx_get=f'/{app_name}/{first_step_id}', hx_trigger='load'),
id=f"{app_name}-container"
)
```
The `run_all_cells()` method encapsulates the workflow initialization pattern and creates an immediate mental connection to Jupyter notebooks.
### 3. APP_NAME vs. Filename Distinction <!-- key: app-name-vs-filename -->
<!-- START_ASCII_ART: app-name-vs-filename -->
```
📂 FILENAME vs APP_NAME DISTINCTION
┌─────────────────────────────────────────────────────────────┐
│ CRITICAL SEPARATION │
├─────────────────────────────────────────────────────────────┤
│ │
│ 📁 FILENAME: 200_workflow_genesis.py │
│ ├── 🌐 Determines public URL: /workflow_genesis │
│ └── 📊 Controls menu order: 200 │
│ │
│ 🏷️ APP_NAME: "workflow_genesis_internal" │
│ ├── 💾 Database table identifier │
│ ├── 🔒 MUST REMAIN STABLE (data integrity) │
│ └── 🚫 NEVER change after deployment │
│ │
│ ⚠️ DANGER: Changing APP_NAME = Orphaned Data │
└─────────────────────────────────────────────────────────────┘
```
<!-- END_ASCII_ART: app-name-vs-filename -->
**Critical for data integrity:**
* **Filename** (e.g., `200_workflow_genesis.py`): Determines public URL endpoint and menu ordering
* **APP_NAME Constant** (e.g., `APP_NAME = "workflow_genesis_internal"`): Internal identifier that MUST REMAIN STABLE
### 4. State Management via DictLikeDB
* State stored as JSON blobs in pipeline table
* Accessed via `pip.get_step_data()` and `pip.set_step_data()`
* All state changes are transparent and observable
### 5. Plugin Discovery System <!-- key: plugin-discovery-system -->
<!-- START_ASCII_ART: plugin-discovery-system -->
```
📁 PLUGIN DISCOVERY SYSTEM
plugins/
├── 010_introduction.py ✅ Registered as "introduction" (menu order: 1)
├── 020_profiles.py ✅ Registered as "profiles" (menu order: 2)
├── hello_flow (Copy).py ❌ SKIPPED - Contains "()"
├── xx_experimental.py ❌ SKIPPED - "xx_" prefix
└── 200_workflow_genesis.py ✅ Registered as "workflow_genesis" (menu order: 20)
📊 AUTO-REGISTRATION RULES:
✅ Numeric prefix → Menu ordering + stripped for internal name
❌ Parentheses "()" → Development copies, skipped
❌ "xx_" prefix → Work-in-progress, skipped
🔧 Must have: landing() method + name attributes
💉 Auto dependency injection via __init__ signature
```
<!-- END_ASCII_ART: plugin-discovery-system -->
* Files in `plugins/` directory are auto-discovered
* Numeric prefixes control menu ordering
* Classes must have `landing` method and name attributes
* Automatic dependency injection based on `__init__` signature
## Workflow Development Helper Scripts
Pipulate includes sophisticated helper scripts for workflow development:
### `create_workflow.py`
Creates new workflows from templates:
```bash
python create_workflow.py workflow.py MyWorkflow my_workflow \
"My Workflow" "Welcome message" "Training prompt" \
--template trifecta --force
```
### `splice_workflow_step.py`
Adds steps to existing workflows:
```bash
python splice_workflow_step.py workflow.py --position top
python splice_workflow_step.py workflow.py --position bottom
```
### Template System <!-- key: workflow-template-system -->
<!-- START_ASCII_ART: workflow-template-system -->
```
🏗️ WORKFLOW TEMPLATE SYSTEM
┌─────────────────┐ ┌─────────────────┐
│ BLANK TEMPLATE │ │TRIFECTA TEMPLATE│
├─────────────────┤ ├─────────────────┤
│ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Step 1 │ │ │ │ Step 1 │ │
│ │ (Minimal) │ │ │ │ (Input) │ │
│ └─────────────┘ │ VS │ └──────┬──────┘ │
│ │ │ │ │
│ Quick Start │ │ ▼ │
│ Single Purpose │ │ ┌─────────────┐ │
└─────────────────┘ │ │ Step 2 │ │
│ │ (Process) │ │
create_workflow.py │ └──────┬──────┘ │
--template blank │ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Step 3 │ │
│ │ (Output) │ │
│ └─────────────┘ │
│ │
│ Full Pattern │
│ Complete Flow │
└─────────────────┘
create_workflow.py
--template trifecta
```
<!-- END_ASCII_ART: workflow-template-system -->
* `blank`: Minimal workflow with one step
* `trifecta`: Three-step workflow pattern
* Automatic method generation and insertion
### Workflow Reconstruction System <!-- key: workflow-reconstruction-system -->
**The Revolutionary Alternative to OOP Inheritance:** Atomic transplantation of workflow components using intelligent pattern matching and AST precision.
<!-- START_ASCII_ART: workflow-reconstruction-system -->
```
🧬 WORKFLOW RECONSTRUCTION: ATOMIC TRANSPLANTATION
═════════════════════════════════════════════════════════════════════════
OLD WORKFLOW WORKFLOW UPDATED WORKFLOW
(Atomic Source) RECONSTRUCTOR (Incremental Gen)
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ 🧬 Components: │ AST │ 🎯 Pattern │ AST │ ✨ Generated: │
│ │ ───► │ Matching │ ───► │ │
│ ┌─────────────┐ │ │ │ │ ┌─────────────┐ │
│ │step_params* │ │ │ Bundle Type 1: │ │ │step_params* │ │ ✅
│ │step_optim* │ │ │ Auto-Registered │ │ │step_optim* │ │ ✅
│ │parameter* │ │ │ Methods │ │ │parameter* │ │ ✅
│ └─────────────┘ │ │ │ │ └─────────────┘ │
│ │ │ Bundle Type 2: │ │ │
│ ┌─────────────┐ │ │ Custom Routes │ │ ┌─────────────┐ │
│ │_process │ │ │ (_process, │ │ │_process │ │ ✅
│ │preview │ │ │ preview) │ │ │preview │ │ ✅
│ └─────────────┘ │ │ │ │ └─────────────┘ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
🔄 COMPLETE LIFECYCLE: Test → Validate → Production → Cleanup
--suffix 5 --target new_name --target same_name git status
────────── ──────────────── ────────────────── ──────────
param_buster5 advanced_params param_buster (in-place) (shows cruft)
(safe testing) (new workflow) (git history preserved) (clean up!)
🎯 WHY IT WORKS: Lightning in a Bottle
┌─────────────────────────────────────────────────────────────────────────┐
│ ✨ Pattern Matching: No manual markers needed │
│ 🔧 AST Precision: Syntactically perfect code generation │
│ 🎭 Inheritance Alternative: Compose without complex super() chains │
│ 🧪 Safe Testing: Incremental validation without production risk │
│ 📚 Git Continuity: In-place updates preserve development history │
│ 🧹 Systematic Cleanup: Prevents file cruft accumulation │
└─────────────────────────────────────────────────────────────────────────┘
workflow_reconstructor.py --template botify_trifecta
--source parameter_buster
--suffix 5
```
<!-- END_ASCII_ART: workflow-reconstruction-system -->
**The System That Eliminates Bootstrap Paradox:**
* **Atomic Sources:** Battle-tested workflows become component libraries
* **Pattern Matching:** Intelligent detection via `_process`, `preview` patterns
* **AST Transplantation:** Surgical precision without syntax errors
* **Complete Lifecycle:** Development → Testing → Production → Cleanup
## 📋 Quick Reference Card
### Essential Commands
```bash
# Development workflow
cd ~/pipulate && nix develop # Start Pipulate
nix develop .#quiet # Start without auto-services
python server.py # Manual server start
git pull && nix develop # Update to latest
# Create new workflows
python helpers/workflow/create_workflow.py my_workflow.py MyClass my_internal_name
python helpers/workflow/splice_workflow_step.py my_workflow.py --position top
# Plugin naming conventions
010_my_plugin.py # Active plugin (menu order 1)
xx_my_plugin.py # Disabled during development
my_plugin (Copy).py # Ignored development copy
```
### Key URLs & Ports
- **Pipulate App**: `http://localhost:5001`
- **JupyterLab**: `http://localhost:8888`
- **Local AI Chat**: Built into the Pipulate interface
- **Logs**: `tail -f logs/server.log` for debugging
### Critical Patterns for AI Assistants
```python
# Auto-key generation flow
if not user_input:
response = Response('')
response.headers['HX-Refresh'] = 'true'
return response
# Workflow initialization
return pip.run_all_cells(app_name, steps)
# State management
data = pip.get_step_data(step_id)
pip.set_step_data(step_id, updated_data)
```
### File Structure Quick Reference
```
plugins/ # Your workflows (auto-discovered)
├── 010_introduction.py # Menu order 1
├── xx_draft.py # Disabled (xx_ prefix)
└── draft (Copy).py # Ignored (parentheses)
mcp_tools.py # AI assistant interface
common.py # Base classes for workflows
browser_automation/ # Selenium automation tools
logs/server.log # Debug everything here
data/data.db # SQLite application state
```
## Common LLM Implementation Mistakes <!-- key: llm-implementation-mistakes -->
<!-- START_ASCII_ART: llm-implementation-mistakes -->
```
🚨 LLM IMPLEMENTATION MISTAKE PREVENTION
┌────────────────────────────────────────────────────────────┐
│ COMMON PITFALLS │
├────────────────────────────────────────────────────────────┤
│ ❌ Missing HX-Refresh │ ✅ if not user_input: │
│ Response │ response.headers['HX- │
│ │ Refresh'] = 'true' │
├────────────────────────────┼───────────────────────────────┤
│ ❌ Wrong Key Generation │ ✅ pip.generate_pipeline_ │
│ Method │ key(self) │
├────────────────────────────┼───────────────────────────────┤
│ ❌ Broken Chain Reaction │ ✅ hx_trigger="load" → │
│ Pattern │ Automatic progression │
├────────────────────────────┼───────────────────────────────┤
│ ❌ APP_NAME Changes │ ✅ NEVER modify after │
│ (Data Orphaning) │ deployment │
└────────────────────────────────────────────────────────────┘
```
<!-- END_ASCII_ART: llm-implementation-mistakes -->
**LLMs frequently make these errors:**
1. **Missing HX-Refresh Response**: Forgetting to return the refresh response for empty keys
2. **Incorrect Key Generation**: Not using `pip.generate_pipeline_key(self)` properly
3. **Missing Cursor Positioning**: Forgetting the `_onfocus` attribute for user experience
4. **Wrong Route Handling**: Not understanding the difference between landing page and init routes
5. **State Inconsistency**: Not properly handling the key generation and storage flow
6. **APP_NAME Changes**: Modifying APP_NAME after deployment, orphaning existing data
7. **Chain Reaction Breaks**: Not properly implementing the HTMX step progression pattern
## Key Design Guidelines & Patterns
These "speedbumps" reinforce Pipulate's core philosophy:
* **Local vs. Enterprise Mindset:** Embrace local-first simplicity. Avoid patterns designed for distributed, multi-tenant systems.
* **JSON State Management (Workflows):** Keep workflow state in self-contained steps within a single JSON blob per run. Avoid complex state machines or external step tracking.
* **Database (MiniDataAPI):** Use the simple schema definition and access patterns provided. Avoid heavy ORMs.
* **Workflow Pattern:** Ensure workflows are linear and state is explicitly passed or saved at each step. Avoid complex async task chaining that obscures state.
* **UI Rendering Pattern:** Generate HTML directly from Python components via FastHTML. Avoid template engines.
* **WebSocket Pattern:** Use the dedicated `Chat` class for managing LLM interactions. Avoid raw WebSocket handling elsewhere.
* **Workflow Progression Pattern:** Workflows use an explicit chain reaction pattern with `hx_trigger="load"` to manage step progression. This pattern must be preserved exactly as implemented. See the workflow documentation for details.
## Internal Components <!-- key: core-concepts-internal-components -->
* **Monitoring:** A file system watchdog monitors code changes. Valid changes trigger an automatic, monitored server restart via Uvicorn, facilitating live development.
```
┌─────────────┐ ┌──────────────┐
│ File System │ Changes │ AST Syntax │ Checks Code
│ Watchdog │ Detects │ Checker │ Validity
└──────┬──────┘ └───────┬──────┘
│ Valid Change │
▼ ▼
┌───────────────────────────┐ ┌──────────┐
│ Uvicorn Server │◄─── │ Reload │ Triggers Restart
│ (Handles HTTP, WS, SSE) │ │ Process │
└───────────────────────────┘ └──────────┘
```
### Pipeline State Inspector & MCP Tools <!-- key: pipeline-state-inspector -->
The system provides comprehensive debugging and state inspection capabilities through MCP tools and real-time monitoring:
```
📊 PIPELINE STATE INSPECTOR
├─── 🔍 Discovering active workflows...
├─── ⚡ Found 3 running processes
├─── 🎯 Step 2/5: Processing data transformations
└─── ✨ Ready for next interaction!
🤖 MCP TOOL CALLS
├─── 📡 Connecting to Botify API...
├─── 🔐 Authentication successful
├─── 📊 Fetching schema (4,449 fields discovered!)
└─── 💾 Caching results for lightning-fast access
```
-----
## The Deeper Philosophy: Why This Matters <!-- key: deeper-philosophy -->
Pipulate represents more than technology choices — it's a statement about **computing sovereignty** and the future of human-AI collaboration.
### The Bootstrap Paradox Solved <!-- key: bootstrap-paradox-solved -->
How do you "lift yourself up by your bootstraps"? The answer: **you're not a closed system**. Humans interact with reality through interfaces, just like software components. We are "human apps with reality APIs."
This pattern repeats at every scale — from quantum fields to societies. The art lies in choosing the **right granularity** for these interfaces. Too abstract, and you lose control. Too granular, and you drown in complexity.
### Every API is a Compromise <!-- key: every-api-is-compromise -->
The most successful APIs are **durable** — they become the "laws of physics" for their domain. In reality: the laws of physics, chemistry, biology. In tech: logic gates, operating systems, protocols like HTTP and HTML.
Pipulate chooses these durable APIs deliberately, creating a **future-proof foundation** for AI-assisted development that will outlast the current framework churn.
### Writing for the Future <!-- key: writing-for-future -->
This documentation serves multiple audiences: humans seeking immediate productivity, and AI systems learning to collaborate with human creativity. The explicit, observable nature of Pipulate workflows creates **perfect training data** for the next generation of AI assistants.
## Final Thoughts
Pipulate offers a unique approach to building local, AI-assisted applications, prioritizing simplicity, user control, and reproducibility over conventional scaling patterns. By understanding its core concepts and embracing its philosophy, developers can create powerful workflows, and end-users can benefit from AI guidance with complete flexibility in deployment options.
Remember the guiding principles:
1. **Keep it simple.**
2. **Support local-first with cloud integration options.**
3. **Embrace the unconventional while remaining practical.**
4. **Choose durable foundations that work with any approach.**
5. **Build for both human creativity and AI collaboration—local or cloud.**
**The Bottom Line:** Pipulate doesn't reject the modern AI ecosystem—it provides a structured foundation that works with any AI service. Whether you're using Claude via API, ChatGPT for reasoning, or local models for privacy, Pipulate gives you the workflow framework to orchestrate them all effectively. It's not about choosing sides in the AI wars—it's about having the right tool for any job.
-----
## Developer's Notes
### The Pipulate Workshop
The repository includes not only polished plugins but also experimental scripts and notebooks under development (e.g., in the root directory or marked with `xx_` prefix in plugin directories). These represent ongoing work and exploration.
### Plugin Development Conventions
#### Auto-Registration Behavior
* **Numeric Prefixes:** Files like `workflows/10_hello_flow.py` are registered as `hello_flow` (number stripped for internal name, used for menu order).
* **Parentheses Skip:** Files with `()` in the name (e.g., `hello_flow (Copy).py`) are skipped – useful for temporary copies during development.
* **`xx_` Prefix Skip:** Files prefixed with `xx_` (e.g., `xx_experimental_flow.py`) are skipped – useful for keeping unfinished work in the plugin directories without activating it.
#### Workflow for Creating New Plugins
1. **Copy:** Copy a template to `my_flow (Copy).py`.
2. **Modify:** Develop your workflow. It won't auto-register yet.
3. **Test:** Rename to `xx_my_flow.py`. The server should auto-reload. Test thoroughly.
4. **Deploy:** Rename to `##_my_flow.py` to assign menu order and activate.
#### Git History Considerations
Use `git mv` for simple renames (like `xx_` to numbered prefix) to preserve history. Document more complex renames in commit messages.
```bash
git mv workflows/xx_my_flow.py workflows/##_my_flow.py
git commit -m "Feat: Promote workflow xx_my_flow.py to ##_my_flow.py"
```
--------------------------------------------------------------------------------
## About This README: Single Source of Truth Documentation <!-- key: about-this-readme -->
This README serves as the **upstream source of truth** for all Pipulate documentation across GitHub, Pipulate.com, and the built-in app documentation. Changes made here automatically cascade to all other documentation surfaces.
### The ASCII Art Synchronization System
```
🌊 THE UPSTREAM TRUTH CASCADE
═══════════════════════════════════════════════════════════════
📜 Source Code Reality (The Ultimate Truth)
│
▼
📄 README.md (Single Source of Truth)
├─ ASCII Art Blocks (Visual Truth)
├─ HTML Comment Keys (Metadata)
└─ 80-Hyphen Pagination (Structure)
│
┌──────────────┼──────────────────┐
▼ ▼ ▼
🌐 GitHub Page 📚 Pipulate.com 🔧 Built-in Docs
(Auto-display) (Jekyll Build) (Live Integration)
│ │ │
▼ ▼ ▼
📊 Screenshots 🎬 Demos 🧪 Tests
(Future) (Future) (Future)
```
**How it works:**
- **ASCII Art Blocks**: Visual diagrams are automatically extracted and distributed to other documentation files
- **HTML Comment Keys**: Headlines marked with `<!-- key: identifier -->` serve as reference anchors
- **80-Hyphen Pagination**: Section dividers enable automatic document structuring
- **Automatic Synchronization**: Running `python helpers/docs_sync/sync_ascii_art.py` updates all documentation
This creates **"ASCII art peer pressure"** — when visual diagrams change, they compel surrounding text to be updated for consistency, ensuring documentation accuracy across the entire ecosystem.
### Roadmap
**Core & Workflow Enhancements:**
* Dev, Test, and Prod database switching
* Saving source HTML and rendered DOM of any URL
* Botify data export CSV save (incorporating robust polling)
* Full web form field support (textarea, dropdown, checkboxes, radio buttons)
* Generic support for Anywidgets
* Utility for deleting garbage tables from plugin experimentation
**AI / LLM Integration:**
* LLM inspection of any local data object (RAG-style functionality)
* Various memory types for LLM context (vector embedding, graph, key/val-store)
* Enabling the local LLM to be an MCP Client
**Automation & External Interaction:**
* MCP Server for automated web browsing and similar tasks
---
## Included PrismJS Highlighting
THEMES
- Okaidia ocodia 1.77KB
LANGUAGES
- CSS1.71KB
- Markup + HTML + XML + SVG + MathML + SSML + Atom + RSS4.64KB
- C-like0.83KB
- JavaScript6.18KB
- Bash + Shell + Shell zeitgeist87 8.96KB
- Diff uranusjr 1.33KB
- JSON + Web App Manifest CupOfTea696 0.58KB
- JSON5 RunDevelopment 0.52KB
- JSONP RunDevelopment 0.23KB
- Liquid cinhtau 2.56KB
- Lua Golmote 0.74KB
- Markdown Golmote 10.43KB
- Markup templating
- Mermaid RunDevelopment 3.03KB
- Nix Golmote 1.47KB
- Python multipetros 2.45KB
- Regex RunDevelopment 2.33KB
- YAML hason 3.11KB
PLUGINS
- Line Highlight11.66KB
- Line Numbers kuba-kubula 7.23KB
- Toolbar mAAdhaTTah 5.63KB
## Contributing
Contributions are welcome\! Please adhere to the project's core philosophy:
* Maintain Local-First Simplicity (No multi-tenant patterns, complex ORMs, heavy client-side state).
* Respect Server-Side State (Use DictLikeDB/JSON for workflows, MiniDataAPI for CRUD).
* Preserve the Workflow Pipeline Pattern (Keep steps linear, state explicit).
* Honor Integrated Features (Don't disrupt core LLM/Jupyter integration unless enhancing local goals).
---
## License
This project is licensed under the MIT License. See the [LICENSE](https://github.com/miklevin/pipulate/blob/main/LICENSE) file for details.
## Resources
**Background Articles:** <a href="https://mikelev.in/">Mike Levin, AI SEO in NYC</a>
**Enhanced Documentation:** <a href="https://pipulate.com/">Pipulate AI SEO Software</a>
**On GitHub:** <a href="https://github.com/miklevin/pipulate">Pipulate on GitHub</a>
Raw data
{
"_id": null,
"home_page": null,
"name": "pipulate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "seo, ai, llm, automation, local-first, nix, fasthtml, htmx, agentic",
"author": null,
"author_email": "Mike Levin <pipulate@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/5d/35/1b92f28752d6b2cc54251deb1fcae0bd4c7fe00f44267e49e278e8a8a475/pipulate-1.1.1.tar.gz",
"platform": null,
"description": "# Pipulate: Local First AI SEO Software\n\n\n\n<!-- Don't forget to sync_ascii_art.py -->\n> **Your data. Your AI. Your machine. Your control.** \n> No subscriptions, no vendor lock-in, no cloud costs. \n\n## \ud83d\ude80 Quick Start for Impatient People\n\n**Want to skip the philosophy and just see what this does?**\n\n```bash\n# 1. Install Nix (one-time setup)\ncurl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install\n\n# 2. Close and reopen your terminal, then:\ncurl -L https://pipulate.com/install.sh | sh\n\n# 3. Launch it\ncd ~/pipulate && nix develop\n```\n\n**What you get:** A local web app at `http://localhost:5001` with step-by-step workflows, integrated AI chat, and a JupyterLab instance at `http://localhost:8888`. No cloud required.\n\n**Success looks like:** Two browser tabs auto-open showing the Pipulate interface and JupyterLab.\n\n## \ud83d\udca1 What Can You Actually Build?\n\n**Real examples of what people create with Pipulate:**\n\n### \ud83d\udd0d SEO Workflows\n- **Keyword Research Pipeline**: Input seed keywords \u2192 AI expansion \u2192 competition analysis \u2192 export spreadsheet\n- **Content Gap Analysis**: Compare your site vs competitors \u2192 identify missing topics \u2192 prioritized content calendar\n- **Technical SEO Audits**: Crawl site \u2192 check Core Web Vitals \u2192 generate action items \u2192 track fixes\n\n### \ud83d\udcca Data Processing Workflows \n- **CSV Data Cleaning**: Upload messy data \u2192 standardize formats \u2192 remove duplicates \u2192 validate results\n- **API Data Collection**: Connect to APIs \u2192 fetch data in batches \u2192 transform to consistent format \u2192 store locally\n- **Report Generation**: Combine multiple data sources \u2192 apply business rules \u2192 create branded reports\n\n### \ud83e\udd16 AI-Assisted Workflows\n- **Content Creation Pipeline**: Research topics \u2192 generate outlines \u2192 write drafts \u2192 optimize for SEO\n- **Data Analysis Helper**: Upload spreadsheet \u2192 AI suggests insights \u2192 create visualizations \u2192 export findings\n\n**Key advantage:** Each workflow is a guided, step-by-step process that non-technical users can run repeatedly, while developers can customize the Python code behind each step.\n\n### Meet Chip O'Theseus <!-- key: pipulate-welcome-banner -->\n\n```\n\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557 Chip O'What?\n\u2551 \ud83c\udfad PIPULATE: LOCAL-FIRST AI SEO SOFTWARE & DIGITAL WORKSHOP \u2551 , O\n\u2551 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2551 \\\\ . O\n\u2551 \u2551 |\\\\/| o\n\u2551 \ud83d\udcac Chip O'Theseus: \"Welcome to your sovereign computing environment!\" \u2551 / \" '\\\n\u2551 \u2551 . . .\n\u2551 \ud83c\udf1f Where Python functions become HTML elements... \u2551 / ) |\n\u2551 \ud83c\udf1f Where workflows preserve your creative process... \u2551 ' _.' |\n\u2551 \ud83c\udf1f Where AI integrates locally and globally... \u2551 '-'/ \\\n\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d\n```\n\n## AI On Rails: Structured Workflows for Any AI <!-- key: about-pipulate -->\n\n**The Challenge with Agentic AI:** Powerful but unpredictable\u2014you never know what you're gonna get.\n\n**The Pipulate Approach:** Structured workflows that can leverage **any AI**\u2014local, cloud, or hybrid\u2014while maintaining complete visibility and control.\n\nThink of it as putting guardrails on AI assistance. Instead of asking an AI to \"figure it out,\" domain experts create step-by-step workflows that guide AI through proven processes. The AI gets structure, you get predictable results.\n\n**Pipulate: Your AI Swiss Army Knife:** Whether you prefer local privacy, cloud power, or hybrid approaches, Pipulate provides the framework. Use local models for sensitive work, cloud APIs for heavy lifting, or both in the same workflow\u2014your choice, your control.\n\n```\n \ud83e\udd16 AGENTIC MODE (Chaos) \ud83d\ude82 AI ON RAILS (Pipulate)\n \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n \ud83d\udca5 GOES OFF \ud83d\udcca LINEAR WORKFLOWS\n HALF-COCKED! BY DOMAIN EXPERTS\n \u2502 \u2502\n \u25bc \u25bc\n \u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2551 \ud83c\udf2a\ufe0f WILLY NILLY \ud83c\udfb2 \u2551 \u2502 Step 1: Analyze\u25b8 \u2502\n \u2551 \u2551 VS \u2502 Step 2: Process\u25b8 \u2502\n \u2551 Unpredictable \u2551 \u2502 Step 3: Report\u25b8 \u2502\n \u2551 Results \u2551 \u2502 Step 4: Export\u25b8 \u2502\n \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u25bc \u25bc\n \u2601\ufe0f Trains Frontier Models \ud83c\udfe0 Keeps Domain Expertise Local\n```\n\n1. \ud83d\udda5\ufe0f **Runs locally** like a desktop app using modern web technologies\n2. \ud83d\udc0d **Simple linear workflow** approach powered by HTMX for seamless interactivity\n3. \ud83d\udcd3 **Transforms Jupyter Notebooks** into production-ready, step-by-step workflows\n4. \ud83e\udd16 **Integrated AI assistance** using your own local models with complete privacy\n5. \ud83d\udd27 **Reproducible environments** with Nix that work identically across all platforms\n6. \ud83c\udfaf **Perfect for SEO practitioners** who want to turn technical expertise into guided, reusable workflows\n\n--------------------------------------------------------------------------------\n\n## What is Pipulate?\n\nPipulate is a **local-first, single-tenant desktop app framework** featuring AI-assisted, step-by-step workflows. Designed to feel like an Electron app, it uniquely runs a full, reproducible Linux environment within a project folder using Nix, ensuring consistency across macOS, Linux, and Windows (via WSL).\n\n### Desktop App Architecture: Electron vs Pipulate <!-- key: desktop-app-architecture-comparison -->\n\n<!-- START_ASCII_ART: desktop-app-architecture-comparison -->\n```\n \ud83d\udda5\ufe0f ELECTRON PATTERN \ud83c\udf10 PIPULATE PATTERN\n \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 ELECTRON APP \u2502 \u2502 PIPULATE SETUP \u2502\n \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502.exe \u2502 \u2502.dmg \u2502 \u2502.deb \u2502 \u2502 \u2502 \u2502 install.sh \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 (Works on ALL OSes) \u2502 \u2502\n \u2502 Per-OS Installers \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 This is new.\n \u2502 \ud83d\udcf1 Native Window \u2502 \u2502 \ud83d\udda5\ufe0f Terminal Console \u2502 , O\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \\\\ . O\n \u2502 \u2502 Web Browser \u2502 \u2502 \u2502 \u2502 nix develop \u2502 \u2502 |\\\\/| o\n \u2502 \u2502 (Bundled) \u2502 \u2502 \u2502 \u2502 Starting servers... \u2502 \u2502 / \" '\\\n \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 \u2502 \u2713 JupyterLab ready \u2502 \u2502 . . .\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u2713 Pipulate ready \u2502 \u2502 / ) |\n \u2502 \u2502 \u2502 HTML \u2502 \u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 ' _.' |\n \u2502 \u2502 \u2502 CSS \u2502 \u2502 \u2502 + \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 '-'/ \\\n \u2502 \u2502 \u2502 JS \u2502 \u2502 \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u25bc\n \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \ud83c\udf10 Regular Browser \u2502\n \u2502 \u2502 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u25bc \u2502 \u2502 \u2502 localhost:5001 \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502\n \u2502 \u2502 Node.js \u2502 \u2502 \u2502 \u2502 \u2502 Python/HTMX \u2502 \u2502 \u2502\n \u2502 \u2502 Runtime \u2502 \u2502 \u2502 \u2502 \u2502 Workflows \u2502 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2502 Local AI \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2705 Feels like native app \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\u274c Multiple installers needed\n\u274c Platform-specific builds \u2705 Universal installer\n\u274c Update distribution complexity \u2705 Auto-updates via git\n \u2705 Same experience all OSes\n \u2705 Complete reproducibility\n```\n<!-- END_ASCII_ART: desktop-app-architecture-comparison -->\n\n### The Magnum Opus: Computing Sovereignty <!-- key: magnum-opus-computing-sovereignty -->\n\nThis isn't just another framework \u2014 it's a **deliberate culmination** of decades of tech evolution insights. Pipulate represents the \"third act\" approach to development (3rd time's the charm): choosing the **most durable and lovable** parts of the modern tech stack while rejecting the exhausting hamster wheel of framework churn.\n\nIf you are not an Empire builder and prefer craftsmanship over the rat race and want to build tools that last, then Pipulate may be for you. Pipulate embodies that philosophy \u2014 maximum creative freedom with minimum technical debt, recapturing *that old Webmaster feeling.*\n\n### Core Philosophy: Local-First, WET, and AI-Augmented\n\n#### Breaking Free: Durable Foundations for Any Approach <!-- key: breaking-free-framework-churn -->\n\n<!-- START_ASCII_ART: breaking-free-framework-churn -->\n```\n\ud83c\udfa1 THE FRAMEWORK CHURN CYCLE \ud83c\udff0 COMPUTING SOVEREIGNTY \n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n React \u2192 Vue \u2192 Angular \u2192 Svelte \ud83d\uddfd Your Hardware\n \u2191 \u2193 \ud83d\uddfd Your Data\n Webpack \u2190 Next.js \u2190 Vite \u2190 Remix \ud83d\uddfd Your AI Choice\n \u2191 \u2193 \ud83d\uddfd Your Code\n Docker \u2192 K8s \u2192 Cloud \u2192 Serverless \ud83d\uddfd Your Schedule\n\n \ud83d\ude35\u200d\ud83d\udcab Endless Learning \ud83d\uddfd Your Hardware\n \ud83d\udcb8 Migration Fatigue \ud83d\uddfd Your Data \n \ud83d\udd12 Platform Lock-in \ud83d\uddfd Your AI Choice\n \ud83d\udcc8 Growing Complexity \ud83d\uddfd Your Code\n \ud83d\uddfd Your Schedule\n WITH\n \u2728 Durable Tools:\n \ud83c\udfc3\u200d\u2642\ufe0f JUMP OFF THE WHEEL \u2022 Python (30+ years)\n \u2193 \u2022 SQLite (built-in)\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2022 HTML/HTTP (timeless)\n \u2502 PIPULATE \u2502 \u2022 Nix (reproducible)\n \u2502 Local-First \u2502 \u2022 Cloud APIs (by choice)\n \u2502+ Any Cloud \u2502 \n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \ud83c\udfaf Third Act Philosophy:\n \"Choose tools that will\n outlast any framework\"\n```\n<!-- END_ASCII_ART: breaking-free-framework-churn -->\n\n- **Local-First Sovereignty:** Your data, code, and AI run on your hardware by default\u2014extending to cloud services when you choose. This guarantees privacy, eliminates surprise costs, and gives you complete control over when and how to scale.\n- **WET Workflows, DRY Framework:** Workflows are intentionally \"WET\" (explicit & step-by-step) for maximum clarity and customizability\u2014perfectly mirroring Jupyter Notebooks. The underlying framework is \"DRY\" for efficiency.\n\n- **The AI Advantage:** AI makes WET practical. Tedious code maintenance and refactoring, once a weakness of WET, is now an area where AI excels, turning repetition into a strength for rapid, context-aware development. Our breakthrough **Workflow Reconstruction System** exemplifies this: intelligent AST-based transplantation of workflow components eliminates traditional OOP inheritance complexity while maintaining perfect code precision.\n- **Radical Transparency (\"Know EVERYTHING!\"):** We reject opaque enterprise patterns in favor of complete observability. State is managed in transparent SQLite tables and JSON blobs, making the entire system intuitive and debuggable. No black boxes, ever.\n- **Reproducibility with Nix:** Nix Flakes provide a perfect, reproducible Linux environment on macOS, Linux, and Windows (WSL), solving the \"works on my machine\" problem.\n- **Future-Proof Stack:** We rely on durable standards: Python, SQLite, HTML, and HTMX. This is a framework built to last.\n\n### Primary Goals\n\n1. **Empower End-Users (e.g., SEO Practitioners):** Enable non-programmers to run powerful, AI-guided workflows (often ported from Jupyter Notebooks) without needing to interact with Python code directly.\n2. **Serve Developers:** Provide a simple, reproducible environment for building these workflows, leveraging integrated tooling like Jupyter, local LLMs, and a streamlined web framework.\n\n--------------------------------------------------------------------------------\n\n## The Technical Stack: Simple Yet Powerful\n\nPipulate's WET philosophy extends to its technology choices, favoring simple, durable tools over complex abstractions:\n\n## *Not On My Machine* Problem Fixed <!-- key: not-on-my-machine-problem-fixed -->\n\n> The Cloud's popularity has been driven in part by developers not wanting to maintain multiple codebases or installers per OS. Thanks to Nix, that's all fixed.\n\n* **Nix Flakes:** Manages dependencies and creates reproducible environments, ensuring consistency across developers and operating systems, with optional CUDA support. E.g. Is this a Linux-thing you're reading about here? A Windows thing? A Mac thing? The answer is: YES!!! All of the above \u2014 and if you've got cool acceleration hardware, it will even take advantage and utilize that too. Best of all worlds.\n\n```\n ____ _ _ .--. ___________\n | _ \\ __ _ _ ____ _(_)_ __ (_)_ __ ,--./,-. |o_o | | | |\n | | | |/ _` | '__\\ \\ /\\ / / | '_ \\| \\ \\/ / / # \\ |:_/ | | | |\n | |_| | (_| | | \\ V V /| | | | | |> < | | // \\ \\ |_____|_____|\n |____/ \\__,_|_| \\_/\\_/ |_|_| |_|_/_/\\_\\ \\ / (| | ) | | |\n `._,._,' /'\\_ _/`\\ | | |\n Solving the \"Not on my machine\" problem well. \\___)=(___/ |_____|_____|\n```\n\n**Nix serves as the \"Noah's Ark\"** \u2014 preserving this perfect focus in a reproducible environment that works identically across all platforms. Once you've locked in the focus, it lasts for years or decades, all bottled up in infrastructure-as-code.\n\n## Other Key Technologies Used\n\nPipulate integrates a carefully selected set of tools aligned with its philosophy:\n\n* **FastHTML:** A Python web framework prioritizing simplicity. It generates HTML directly from Python objects (no template language like Jinja2) and minimizes JavaScript by design, working closely with HTMX. It's distinct from API-focused frameworks like FastAPI. The Python function-naming *is the HTML-template language.*\n\n### The New LAMP Stack: Evolution in Simplicity <!-- key: new-lamp-stack-comparison -->\n\n<!-- START_ASCII_ART: new-lamp-stack-comparison -->\n```\n\ud83c\udfdb\ufe0f ORIGINAL LAMP STACK (2000s) \ud83d\ude80 NEW LAMP STACK (2025)\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 \ud83d\udc27 L: Linux \u2502 \u2502 \ud83d\udc27 L: Linux + Nix \u2502\n\u2502 Single OS, manual setup \u2502 \u2502 Reproducible everywhere \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \ud83c\udf10 A: Apache \u2502 \u2502 \u26a1 A: ASGI \u2502\n\u2502 Static config, restarts \u2502 \u2502 Async, hot reload \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \ud83d\uddc4\ufe0f M: MySQL \u2502 \u2502 \ud83d\udcca M: MiniDataAPI \u2502\n\u2502 Complex queries, joins \u2502 \u2502 Python-native simplicity\u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \ud83d\udd27 P: PHP \u2502 \u2502 \ud83d\udc0d P: Python + FastHTML \u2502\n\u2502 Mix of HTML/logic \u2502 \u2502 + HTMX \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \ud83c\udfe2 Enterprise \u2502 \u2502 \ud83c\udfe0 Local-First \u2502\n \u2502 Complexity \u2502 \u2502 Sovereignty \u2502\n \u2502 \u2502 \u2502 \u2502\n \u2502 \u2022 Multi-server \u2502 \u2502 \u2022 Single machine \u2502\n \u2502 \u2022 Load balancers \u2502 \u2502 \u2022 Integrated AI \u2502\n \u2502 \u2022 Database clusters \u2502 VS \u2502 \u2022 SQLite simplicity \u2502\n \u2502 \u2022 DevOps overhead \u2502 \u2502 \u2022 Nix reproducibility \u2502\n \u2502 \u2022 Cloud lock-in \u2502 \u2502 \u2022 Flexible deployment \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n \ud83c\udfaf One person understands \ud83c\udfaf One person controls\n part of the system the entire system\n```\n<!-- END_ASCII_ART: new-lamp-stack-comparison -->\n\nThe original LAMP stack was beautiful in its simplicity \u2014 one person could understand and manage the whole stack. But it got bloated with enterprise patterns, microservices, and distributed complexity.\n\nPipulate brings back that **\"one person, full stack\"** philosophy with modern tools:\n\n- **L**inux + **N**ix: Reproducible environments across all platforms\n- **A**SGI: Modern async server interface, future-proofed for performance\n- **M**iniDataAPI: Universal SQL simplifier close to Python's core data structures\n- **P**ython + FastHTML + HTMX: The new web development paradigm\n\nThis isn't just simpler \u2014 it's more powerful, giving you complete environment reproducibility, local AI integration, server-side state management, and future-proofed skills.\n\n### The Lens Stack: Focused Architecture <!-- key: the-lens-stack -->\n\nPipulate's technology choices form **aligned lenses** that focus ideas from abstraction to actualization. Each lens must be **ground and polished** without misaligning the focus:\n\n```\n Universal Translator of Abstractions clarify into implementations\n Spoken Language to Code by each lens being simple and transparent.\n\n Idea --> Lens 1 --> Lens 2 --> Lens 3 -> Lens 4 -> Lens 5 -> Lens 6\n\n -----> ,--.\n ---> ,' `.---------> ,--.\n --> / \\------> ,' `.-------> ,--. ,-.\n o -> / Linux \\----> / http \\----> ,'_hx `.--->,' `. ,-.\n /|\\ ( HARDWARE )--> ( PROTOCOL )--> ( LINGUA )->( UI/UX )->(APP)->(git)\n / \\ -> \\ Nix /----> \\ html /----> `..py ,'--->`. ,' `-'\n --> \\ /------> `. ,'-------> `--' `-' And so on\n ---> `. ,'---------> `--' AI Help\n -----> `--' AI Help\n AI Help\n```\n\nWe keep lenses minimal, their material either thoroughly pre-trained into the model (Python 3.x, HTMX, etc.) or able to be included in the prompt and easily held in the context window. We've trimmed the cruft \u2014 the lens flashes and burrs, and all unnecessary extra lenses (Angular, React, Vue, etc.)\n\n```yaml\nHARDWARE:\n install.sh: Published on Pipulate.com to initiate magic cookie install \n flake.nix: Nix IaC creating a normalized Linux subsystem on any host OS\nPROTOCOL:\n http: Uvicorn fast Asynchronous Server Gateway Interface (ASGI) web server\n html: Uvicorn talks to Python Starlette using anyio & httpx libraries\n websocket: static/ws.js provides client bi-directional asynchronous communication\nLINGUA:\n htmx: static/htmx.js JavaScript library to eliminate most need for JavaScript\n Python: .venv/bin/python3.12 latest version AIs are well trained on\nUI/UX:\n browser: Obviously, but I guess it needs to be said. Like a looser Electron.\n fasthtml: static/fasthtml.js for FT Components, Python functions as templating\nAPP:\n app: Flask-style Uvicorn factory instance instantiated by FastHTML fast_app\n db: Dict-like DB providing transparent server-side state (server cookies)\n pipulate: Pipeline state management, like db but with JSON blob for workflows\n```\n\n### Grinding Off the Burrs and Flashes <!-- key: grinding-off-burrs-flashes -->\n\nIn lens manufacturing, \"flashes\" are excess material that squeeze out of molds \u2014 unwanted projections that must be ground off. Steve Jobs famously did this twice: adopting Gorilla Glass (grinding off plastic flashes) and rejecting Flash Player (grinding off software bloat).\n\n**Pipulate continues this tradition:**\n- **FastHTML**: Grinds off Jinja2 template complexity\n- **HTMX**: Grinds off virtual DOM overhead\n- **Local AI**: Enables privacy by default, cloud power when desired\n- **SQLite**: Grinds off enterprise database complexity\n\nThe result: clean, focused tools that do their job without unnecessary cruft.\n\n--------------------------------------------------------------------------------\n\n## From Flask to FastAPI to FastHTML\n\nThis is not your father's Python web framework. HTMX changes everything \u2014 a marriage made in heaven between Python and the Web, finally turning Python into a first-class citizen for web development. In many use cases such as this one, Python is even preferable to JavaScript in the way it blends Python's formidable ecosystem of packages with workflows.\n\n### The Evolution: Flask \u2192 FastAPI \u2192 FastHTML <!-- key: the-evolution-flask-fastapi-fasthtml -->\n\nThe revolution isn't just another framework \u2014 it's eliminating the template layer entirely:\n\n```\n \ud83c\udf76 FLASK ERA \ud83d\ude80 FASTAPI ERA \ud83c\udf10 FASTHTML ERA\n \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Python \u2502 \u2502 Python \u2502 \u2502 Python \u2502\n \u2502 Functions \u2502 \u2502 Functions \u2502 \u2502 Functions \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Jinja2 \u2502 \u2502 Pydantic \u2502 \u2502 HTMX \u2502\u25c4\u2500 Over-the-wire\n \u2502 Templates \u2502 \u2502 Models \u2502 \u2502 Fragments \u2502 HTML targeting\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 DOM elements\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 HTML \u2502 \u2502 JSON \u2502 \u2502 HTML \u2502\n \u2502 Response \u2502 \u2502 Response \u2502 \u2502 Elements \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n \ud83c\udf10 Full Page Reload \ud83d\udcf1 Frontend Framework \ud83c\udfaf DOM Element Updates\n (React/Vue/Angular) def Div() = <div>\n def Button() = <button>\n\n Template files needed JSON \u2194 HTML conversion Python functions ARE\n Separate languages Client-side complexity the template language!\n```\n\n**The FastHTML Breakthrough:** Python function names directly become HTML elements, eliminating templates and making the server the single source of truth for UI state.\n\n* **HTMX:** Enables dynamic, interactive UIs directly in HTML via attributes, minimizing the need for custom JavaScript. Pipulate uses it for server-rendered HTML updates \u2014 *over the wire HTML*-fragments targeting elements of the DOM directly instead of fragile, performance-reducing, framework-dependent JSON. *THIS* is where you *jump off the tech-churn hamsterwheel* and future-proof yourself.\n\n* **MiniDataAPI:** A lightweight layer for interacting with SQLite and other databases. Uses Python dictionaries for schema definition, promoting type safety without the complexity of traditional ORMs \u2014 effectively future-proofing your SQL. You lose fancy *join* capabilities but in exchange get the *Python dict interface* as your main persistent database API forever-forward, enabiling instant swapability between SQLite and PostgreSQL (for example).\n\n* **Ollama:** Facilitates running LLMs locally, enabling in-app chat, workflow guidance, and future automation capabilities while ensuring privacy and avoiding API costs. Your local AI (Chip O'Theseus) learns & grows with you, hopping from hardware to hardware as you upgrade \u2014 like a genie in a hermitcrab shell. And if that weren't kooky enough \u2014 it knows how to make MCP-calls!!! That's right, your friendly localhost AI Chip O'Theseus is also an *MCP client!* Your linear workflows ain't so linear anymore when a single-step can be: \"Go out and do whatever.\"\n\n### The Hybrid Advantage: Best of Both Worlds\n\n**Pipulate isn't anti-cloud\u2014it's pro-choice.** Each workflow step can choose the best tool for the job:\n\n- **Step 1**: Use local AI for sensitive data analysis (privacy-first)\n- **Step 2**: Call OpenAI's API for advanced reasoning (cloud power) \n- **Step 3**: Process results locally and save to SQLite (data sovereignty)\n- **Step 4**: Use Anthropic's API for final review (frontier capabilities)\n\n**This is the Swiss Army knife approach:** Local by default, cloud by choice, with complete visibility into what's happening at each step. Whether you're processing confidential client data (local) or need cutting-edge AI capabilities (cloud), Pipulate gives you the framework to do both seamlessly.\n\n* **SQLite & Jupyter Notebooks:** Foundational tools for data persistence and the workflow development process (porting from notebooks to Pipulate workflows). SQLite is built into Python and really all things \u2014 the *get-out-of-tech-liability free card* you didn't know you had. And a full JupyterLab instance is installed side-by-side with Pipulate sharing the same Python `.venv` virtual environment, which is also shared with your preferred AI code editor (Cursor, Windsurf, VSCode, Zed) so... well... uhm, there are no words for when 3 different portals-to-Python share the same environment. You can do such stupid AI-tricks as letting your local LLM and a frontier cloud model *inhabit* the same body (Pipulate) \u2014 controlling web browsers together and stuff.\n\n--------------------------------------------------------------------------------\n\n## How to Install Pipulate\n\n### Installation Strategy: Universal First, PyPI Alternative <!-- key: installation-strategy-overview -->\n\nWe offer two installation paths that lead to the exact same robust, Nix-managed environment. Choose the path that best fits your experience level and preferences.\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 New User on macOS \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 PATH 1: Recommended for Everyone \u2502 \u2502 PATH 2: Alternative for Python Developers \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \"I want the simplest, most \"I prefer managing my command-line\n direct way to get this running.\" tools with standard Python utilities.\"\n \u2502 \u2502\n \u25bc \u25bc\n 1. `curl ... [nix]` 1. `brew install pipx` (If needed)\n 2. `curl ... [pipulate]` 2. `pipx install pipulate`\n 3. `pipulate install`\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Nix-Managed Pipulate \u2502\n \u2502 Environment \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n ||\n (Identical\n Result)\n```\n\n### PATH 1: Quick Start \u2014 Universal Installation (Recommended) <!-- key: quick-start-universal-installation -->\n\nThis is the fastest and most universal way to install Pipulate. It has the fewest dependencies and works on any modern Mac, Linux system, or Windows with WSL.\n\n```\n\n \ud83d\udce6 Your Machine \ud83d\udd27 Add Foundation \ud83d\ude80 Complete Environment\n Today with Nix Ready to Go!\n\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Sad Computer\u2502 Step 1 \u2502 \ud83c\udfd7\ufe0f Nix \u2502 Step 2 \u2502 \ud83c\udfaf Pipulate \u2502\n \u2502 Without \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba \u2502 Foundation \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba \u2502 + AI + \u2502\n \u2502 Nix\ud83d\ude22 \u2502 \u2502 Installed \u2502 \u2502 Jupyter \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n Step 3 \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \ud83c\udf10 Browser \u2502\n \u2502 Opens \u2502\n \u2502Automatically\u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n Simple as 1-2-3! No Docker, no build steps, works with or without cloud services.\nEverything runs locally with complete flexibility and control.\n```\n\n**Step 1: Install Nix (One-Time Setup)**\n\nIf you don't have it already, install the Nix package manager. It's the system that makes Pipulate's reproducible environment possible.\n\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install\n```\n\n> **Important:** After the Nix installation finishes, you **must close and reopen** your Terminal window.\n\n**Step 2: Run the Pipulate Installer**\n\nNow, run the universal install script. You can give your project a custom name, too.\n\n```bash\n# To install with a custom name like \"Botifython\"\ncurl -L https://pipulate.com/install.sh | sh -s Botifython\n\n# Or, to install with the default name \"pipulate\"\ncurl -L https://pipulate.com/install.sh | sh\n```\n\n**Step 3: Launch Pipulate**\n\nNavigate into your new project directory and launch the environment with `nix develop`.\n\n```bash\n# cd into the directory you just created\ncd ~/Botifython\n\n# Launch Pipulate\nnix develop\n```\n\nThat's it! The server and JupyterLab will start, and the application will open in your browser.\n\n**Running It Again:**\n\n1. You can just forcibly exit out of that Terminal it's running from.\n2. Open a new Terminal, and once again:\n\n```bash\ncd ~/Botifython\nnix develop\n```\n\n**The Big Reset (If Necessary):**\n\nThings sometimes go wrong. This is how you do a full Pipulate reset. This will also delete anything you downloaded with Pipulate. Adjust custom install name to what you used.\n\n```bash\nrm -rf ~/Botifython\ncurl -L https://pipulate.com/install.sh | sh -s Botifython\ncd ~/Botifython\nnix develop\n```\n\nWait for ***BOTH TABS*** to auto-open in your browser.\n\n### \ud83d\udea8 Installation Troubleshooting\n\n**Common Issues & Solutions:**\n\n| Problem | Solution |\n|---------|----------|\n| `nix: command not found` | You didn't restart your terminal after Nix installation |\n| Browser doesn't open automatically | Manually visit `http://localhost:5001` and `http://localhost:8888` |\n| `Permission denied` errors | Make sure you can write to `~/pipulate` directory |\n| Port conflicts | Kill processes on ports 5001/8888: `lsof -ti:5001 \\| xargs kill -9` |\n| Nix build fails | Clear Nix cache: `nix-collect-garbage` then retry |\n\n**System Requirements:**\n- **macOS**: 10.15+ (Intel/Apple Silicon)\n- **Linux**: Any modern distribution with curl \n- **Windows**: WSL2 with Ubuntu 20.04+\n- **RAM**: 4GB minimum, 8GB recommended\n- **Disk**: 2GB for installation + data storage\n- **Network**: Internet connection for initial setup only\n\n---\n\n### PATH 2: Alternative Installation via PyPI (For Python Developers) <!-- key: alternative-installation-pypi -->\n\nIf you are a developer comfortable with tools like Homebrew and `pipx`, you can use our PyPI package as a gateway to the same robust installation.\n\n**Step 1: Install `pipx`**\n\n`pipx` is a tool for safely installing Python command-line applications. If you don't have it, you can install it with Homebrew.\n\n```bash\nbrew install pipx\n```\n\n**Step 2: Install the Pipulate CLI**\n\nUse `pipx` to install the `pipulate` command-line tool. This will not cause conflicts with your system Python.\n\n```bash\npipx install pipulate\n```\n\n**Step 3: Run the Installer**\n\nUse the command you just installed to set up the full Pipulate application.\n\n```bash\npipulate install\n```\n\nThis will trigger the same universal installation process, resulting in the exact same robust, Nix-managed environment. To run it in the future, just type `pipulate run`.\n\nThese few commands:\n- \u2705 Updates to the latest version automatically\n- \u2705 Starts JupyterLab and the Pipulate server\n- \u2705 Opens web interfaces in your browser\n- \u2705 Provides a complete, reproducible development environment\n\n**That's it!** You now have a local-first development environment with AI integration, installed via your preferred Python toolchain.\n\n### Installation Process Deep Dive <!-- key: installation-process-diagram -->\n\nHere's what happens behind the scenes during the \"magic cookie\" installation:\n\n```\nUser runs install.sh (via curl) Nix Flake Activation & Transformation\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 1. Download install.sh \u2502 \u2502 5. User runs 'nix develop' \u2502\n\u2502 2. Download ZIP from GitHub \u2502 \u2502 6. Flake detects non-git directory \u2502\n\u2502 3. Extract ZIP to ~/AppName \u2502 \u2502 7. Flake clones repo to temp dir \u2502\n\u2502 4. Download ROT13 SSH key \u2502 \u2502 8. Preserves app_name.txt, .ssh, .venv \u2502\n\u2502 to .ssh/rot \u2502 \u2502 9. Moves git repo into place \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u250210. Sets up SSH key for git \u2502\n \u2502 \u250211. Transforms into git repo \u2502\n \u25bc \u250212. Enables auto-update via git pull \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Result: Fully functional, auto-updating, git-based Pipulate installation \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n--------------------------------------------------------------------------------\n\n## Chef or Customer? <!-- key: target-audience -->\n\nAre you a Developer or an End User? Chef or Customer? Understanding your audience is crucial for effective development. Pipulate serves two distinct but complementary audiences, much like a restaurant serves both chefs and customers\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 The Restaurant \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Kitchen (Dev) \u2502 \u2502 Dining Room \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502 (End Users) \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n \u2502 \u2502 \ud83d\udc68\u200d\ud83c\udf73 Sous Chef \u2502\u2500\u2500\u2500recipes\u2500\u2500\u2500\u25ba\u2502 \ud83c\udf7d\ufe0f Customers \u2502 \u2502\n \u2502 \u2502 \ud83d\udc69\u200d\ud83c\udf73 Head Chef \u2502 \u2502 \ud83c\udfe2 Restaurateur \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n \u2502 \u2502 \"How do we make \u2502 \u2502 \"I want the best \u2502 \u2502\n \u2502 \u2502 pasta you've \u2502 \u2502 pasta I've ever \u2502 \u2502\n \u2502 \u2502 never had?\" \u2502 \u2502 had in my life\" \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### \ud83d\udc68\u200d\ud83c\udf73 The Chef (Developer/Technical User)\n* **\ud83d\udd27 Workflow Creators:** Build and customize AI-assisted workflows\n* **\ud83d\udcd3 Jupyter Porters:** Convert notebook experiments into guided applications\n* **\ud83d\udd0d Technical SEOs:** Create sophisticated, reusable SEO processes\n* **\u2699\ufe0f System Administrators:** Deploy consistent environments across teams\n\n**What Chefs Get:**\n- \ud83c\udf9b\ufe0f Complete control over the \"recipe\" (workflow logic)\n- \ud83d\udd04 Reproducible development environment via Nix\n- \ud83c\udfd7\ufe0f Simple architecture that's easy to understand and modify\n- \ud83e\uddf0 Integrated tooling (Jupyter, local LLM, SQLite)\n\n### \ud83c\udf7d\ufe0f The Customer (End User/Non-Technical)\n* **\ud83d\udcc8 SEO Practitioners:** Run powerful workflows without coding\n* **\u270d\ufe0f Content Creators:** Use AI-assisted processes for optimization\n* **\ud83d\udcca Marketing Teams:** Execute consistent SEO strategies\n* **\ud83c\udfe2 Business Owners:** Access enterprise-level SEO capabilities\n\n**What Customers Get:**\n- \ud83d\udeb6\u200d\u2642\ufe0f Guided, step-by-step workflow experiences\n- \ud83e\udd16 AI assistance at every step\n- \ud83d\ude48 No need to see or understand the underlying code\n- \ud83c\udfaf Consistent, repeatable results\n\n### \ud83c\udf5d The Restaurant Analogy\nJust as a chef talks about knife techniques while a diner just wants amazing pasta, Pipulate separates the complexity of creation from the simplicity of consumption. Developers craft the workflows, end-users enjoy the results.\n\n## \ud83c\udfaf Your First 10 Minutes with Pipulate\n\n**After installation succeeds, here's what to expect:**\n\n### What You'll See\n1. **Two browser tabs open automatically:**\n - `localhost:5001` - Pipulate web interface with navigation menu\n - `localhost:8888` - JupyterLab for development/experimentation\n\n2. **In the Pipulate interface:**\n - Left sidebar with workflow plugins (Introduction, Profiles, etc.)\n - Main area showing step-by-step workflow interface \n - Right panel with integrated AI chat (Chip O'Theseus)\n\n3. **Terminal shows:**\n ```\n \ud83d\ude80 Starting Pipulate servers...\n \u2713 FastHTML server ready at http://localhost:5001 \n \u2713 JupyterLab ready at http://localhost:8888\n \u2713 Local AI ready for chat assistance\n ```\n\n### Your Next Steps Depend on Who You Are\n\n**\ud83d\udd0d If you're an SEO practitioner:**\n- Click \"Introduction\" in the left menu for a guided tour\n- Try the built-in workflows to see the step-by-step pattern\n- Use the AI chat to ask \"How do I create a keyword research workflow?\"\n\n**\ud83d\udc68\u200d\ud83d\udcbb If you're a developer:**\n- Open JupyterLab tab and run the introduction notebook\n- Check out `plugins/010_introduction.py` to see workflow code structure\n- Try creating a simple workflow: `python helpers/workflow/create_workflow.py`\n\n**\ud83e\udd16 If you're an AI assistant:**\n- Focus on the Quick Reference Card above\n- Study the Critical Implementation Patterns section\n- Review `mcp_tools.py` for MCP protocol capabilities\n\n**\ud83c\udd95 If you're just exploring:**\n- Click through the left menu items to see different workflow types\n- Ask the AI chat: \"What can I build with Pipulate?\"\n- Try the Introduction workflow to see the step-by-step experience\n\n--------------------------------------------------------------------------------\n\n## The WET Revolution: Why Explicit Code Wins in the AI Era\n\nPipulate is built on a radical philosophy that challenges programming orthodoxy: **WET (Write Everything Twice) is better than DRY (Don't Repeat Yourself)** when you have AI to help manage it.\n\n### The Universal API Pattern: From Quarks to Code <!-- key: universal-api-pattern -->\n\nAt every scale of reality, we see the same pattern: **\"lumps of stuff\" with APIs** that enable interaction. Quarks combine into atoms, atoms into molecules, cells into organisms, individuals into societies. Each level requires the right **granularity** of interface \u2014 not so abstract that you lose control, not so granular that you drown in complexity.\n\n**This is the 80/20 rule of existence:** Handle 80% of interactions gracefully with 20% of the API surface, then handle edge cases as needed. Pipulate applies this principle to code architecture.\n\n### Durable vs. Ephemeral: Building on Bedrock <!-- key: durable-vs-ephemeral -->\n\nThe tech industry suffers from \"hamster wheel syndrome\" \u2014 constantly breaking APIs that force migration cycles. React (20+ versions), Node (frequent breaking changes), Angular (complete rewrites). This isn't progress; it's planned obsolescence.\n\n**Pipulate chooses durable foundations:**\n- **Linux Kernel**: Version 6 in 30 years\n- **Python**: Version 3 in 30 years\n- **HTML**: Version 5 and stable\n- **HTTP**: Version 3 and backward compatible\n\nThese are the \"laws of physics\" for software \u2014 stable APIs that enable compound growth rather than constant rebuilding.\n\n### Why WET Works Now <!-- key: why-wet-works-now -->\n\nTraditional development follows DRY principles, creating abstract, complex systems that are hard to understand and modify. But the world has changed:\n\n1. **\ud83d\udd2c Jupyter Notebooks** promote explicit, literate programming\n2. **\ud83e\udd16 AI assistants** excel at managing repetitive code\n3. **\ud83c\udfe0 Local-first architectures** prioritize clarity over enterprise complexity\n4. **\ud83c\udfaf Right Granularity**: WET provides the perfect abstraction level for human AND AI comprehension\n\n```\n ________________________________\n - Like Notebooks / \\\n - Linear Workflows | It runs proprietary private AI |\n - Local & Cloud-free | Workflows from your Local PC?! |\n - Chip O'Theseus included \\________________________________/\n ()\n HARDWARE PLATFORM LOCAL BROWSER O , Chip O'Theseus\n _______________________ __________ _______ o \\\\ .\n | | / Pipulate \\Jupyter\\__ |\\\\/|\n | Windows, Mac or Linux | | __________________ | See! / \" '\\ - Radical transparency\n | _____ ___ | | | App Name Menu\u2699\ufe0f| |<- - - - -. . . - MCP tool-call control\n | _/ Nix \\____\\_____ | | |------------------| | / ) | - Browser as bot's body\n | | | | | | Workflow | Local | | ' _.' |\n | | Pipulate <---------> -Step #1 | AI\ud83e\udd16 | | '-'/ \\\n |__| localhost:5001 |_| | | -Step #2 | Chat | | What, no Docker?\n | (AI on Rails\ud83d\ude82) | | | -Step #3 | Help\u25b8 | | What, no React?\n |__________________| | |__________|_______| | What, no Cloud?\n |______________________|\n```\n\n**WET workflows are:**\n- **\ud83d\udd0d Observable**: See exactly what's happening at every step\n- **\ud83d\udd27 Customizable**: Modify workflows without breaking abstractions\n- **\ud83e\udd16 AI-Friendly**: Clear code that AI assistants can easily understand and maintain\n- **\ud83d\ude80 Future-Proof**: Built on durable web standards that won't become obsolete\n\n--------------------------------------------------------------------------------\n\n## Developer Setup & Environment Notes\n\n**Nix Environment Activation:** Always run `nix develop` from the `~/pipulate` directory *before* running any project commands (`python server.py`, `pip install`, etc.) in a new terminal. This ensures you are using the correct dependencies defined in `flake.nix`.\n\n**Interactive vs. Quiet Shell:**\n\n**Standard Shell:** `nix develop` (or `nix develop .#default`) runs the startup script (`run-script` defined in `flake.nix`) with welcome messages and service startup. Ideal for general use.\n\n**Quiet Shell:** `nix develop .#quiet` activates the Nix environment *without* running the full startup script or launching services automatically. It only sets up paths and installs pip requirements. Use this for:\n- Running specific commands without starting the servers (e.g., `nix develop .#quiet --command python -c \"import pandas\"`).\n- Debugging or interacting with AI assistants where verbose startup output is undesirable.\n- Manually running `run-server` or `run-jupyter` (scripts placed in `.venv/bin` by the `shellHook`).\n\n**Dependencies:** System-level dependencies (Python version, libraries like `gcc`, `zlib`) are managed by `flake.nix`. Python package dependencies are managed by `pip` using `requirements.txt` within the Nix-provided environment.\n\n**Source of Truth:** The `flake.nix` file is the definitive source for the development environment setup.\n\n--------------------------------------------------------------------------------\n\n## Architecture & Key Concepts\n\nPipulate features a distinct architecture designed for its local-first, simple, and observable nature.\n\n### Architecture Overview Diagram <!-- key: architecture-overview-diagram -->\n\nThis diagram illustrates the high-level components and their interactions:\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 Like Electron, but full Linux subsystem\n \u2502 Browser \u2502 in a folder for macOS and Windows (WSL)\n \u2514\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 HTTP/WS\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Nix Flake Shell \u2502 - In-app LLM (where it belongs)\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 - 100% reproducible\n \u2502 \u2502 FastHTML \u2502 \u2502 Ollama \u2502 \u2502 - 100% local\n \u2502 \u2502 HTMX App \u2502 \u2502 Local LLM \u2502 \u2502 - 100% multi-OS\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502MiniDataAPI\u2502\u25c4\u2500\u2500\u2500\u25ba\u2502 SQLite DB \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\nThis complete, self-contained environment runs identically on any operating system, providing the foundation for all Pipulate workflows and AI interactions.\n\n---\n\n### Integrated Data Science Environment <!-- key: integrated-data-science-environment -->\n\nJupyter Notebooks run alongside the FastHTML server, allowing developers to prototype workflows in a familiar environment before porting them to Pipulate's step-based interface for end-users. The same Python virtual environment (`.venv`) is shared, and ad-hoc package installation is supported. If you're using Cursor, VSCode or Windsurf, set your `Ctrl`+`Shift`+`P` \"Python: Set Interpreter\" to \"Enter Interpreter Path\" `./pipulate/.venv/bin/python`. You might have to adjust based on the folder you use as your workspace. But then you'll have a Python environment unified between Cursor, JupyterLab and Pipulate.\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Jupyter Lab \u2502 \u2502 FastHTML \u2502\n \u2502 Notebooks \u2502 \u2502 Server \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Cell 1 \u2502 \u2502 \u2502 \u2502 Step 1 \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502--->\u2502 \u2502 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Cell 2 \u2502 \u2502 \u2502 \u2502 Step 2 \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502--->\u2502 \u2502 \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 localhost:8888 \u2502 \u2502 localhost:5001 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### Local-First & Single-Tenant Details <!-- key: local-first-single-tenant-details -->\n\nPipulate manages all state server-side within the local environment (think *local-server cookies*), with optional cloud integration as needed. This approach offers:\n* **Privacy & Control:** Data stays local by default, cloud integration when you choose.\n* **Full Resource Access:** Utilize local CPU/GPU freely for intensive tasks, plus cloud APIs for heavy lifting.\n* **Simplicity:** Eliminates complexities of multi-tenancy while supporting both local and cloud workflows.\n* **Observability:** State changes (via DictLikeDB/JSON) are transparent and easily logged (AI greps it there).\n\n### Local-First State Management Benefits <!-- key: local-first-state-management-benefits -->\n\nThis detailed view shows how Pipulate's local-first architecture eliminates common web development complexities:\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 # Benefits of Local-First Simplicity\n \u2502 Web Browser \u2502\n \u2502 \u2502 - No mysterious client-side state\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 - No full-stack framework churn\n \u2502 \u2502 Server Console \u2502 \u2502 - No complex ORM or SQL layers\n \u2502 \u2502 & Web Logs \u2502 \u2502 - No external message queues\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 - No build step required\n \u2502 \u25bc \u2502 - Direct, observable state changes\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Server-Side State \u2502 \u2502 - Conceptually like local-server-side cookies\n \u2502 \u2502 DictLikeDB + JSON \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Enables the \"Know EVERYTHING!\" philosophy\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 - AI greps logs/server.log to see app state!\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### Server-Rendered UI (HTMX) <!-- key: server-rendered-ui-htmx -->\n\nThe UI is constructed primarily with server-rendered HTML fragments delivered via HTMX. This minimizes client-side JavaScript complexity.\n* FastHTML generates HTML components directly from Python.\n* HTMX handles partial page updates based on user interactions, requesting new HTML snippets from the server.\n* WebSockets and Server-Sent Events (SSE) provide real-time updates (e.g., for chat, live development reloading).\n\n```\n HTMX+Python enables a world-class\n Python front-end Web Development environment.\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Navigation Bar \u2502 - No template language (like Jinja2)\n \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 - HTML elements are Python functions\n Simple Python back-end \u2502 Main \u2502 Chat \u2502 - Minimal custom JavaScript / CSS\n HTMX \"paints\" HTML into \u2502 Area \u2502 Interface \u2502 - No React/Vue/Angular overhead\n the DOM on demand \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba \u2502 \u2502 \u2502 - No \"build\" process like Svelte\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 - No virtual DOM, JSX, Redux, etc.\n```\n\nWith such *minimal surface area* the AI code assistant *knows everything.* LLMs are either pre-trained on the stable, infrequently revved libraries used (Python 3.12, HTMX, or it's all small enough to fit in a 1-shot prompt \u2014 yes, the whole core code-base fits in one Gemini Web UI form submit.\n\n--------------------------------------------------------------------------------\n\n## Workflow Patterns & Development\n\n### Pipeline Workflows <!-- key: pipeline-workflows -->\n\nDesigned for porting notebook-style processes, workflows are sequences of steps where the state is managed explicitly at each stage and stored persistently (typically as a JSON blob in the `pipeline` table).\n* **Resumable & Interrupt-Safe:** Because each step's completion is recorded, workflows can be stopped and resumed.\n* **Explicit State Flow:** Data typically passes from one step's output (`done` field) to the next via the `transform` function, simplifying debugging. Patterned on Unix pipes.\n* **Good Training Data:** The structured input/output of each step creates valuable data for potentially fine-tuning models.\n* **Proprietary Friendly:** Excellent for proprietary domain-experts and fields (competing academic, finances) who *resist* letting their data flow onto the Web for general AI training.\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 - Fully customizable steps\n \u2502 Step 01 \u2502\u2500piped\u2500\u25ba\u2502 Step 02 \u2502\u2500piped\u2500\u25ba\u2502 Step 03 \u2502 - Interruption-safe & resumable\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 - Easily ported from Notebooks\n \u2502 \u2502 \u2502 - One DB record per workflow run\n \u25bc \u25bc \u25bc - Everything stays on your machine\n State Saved State Saved Finalized? - Magnitudes simpler than celery\n```\n\n### Run All Cells Pattern <!-- key: run-all-cells-pattern -->\n\n**The key insight**: Pipulate workflows use a `run_all_cells()` pattern that directly mirrors Jupyter's \"Run All Cells\" command. This creates an immediate mental model \u2014 each workflow step is like a notebook cell, and the system automatically progresses through them top-to-bottom, just like running all cells in a notebook.\n\n```\n \ud83d\udcd3 JUPYTER NOTEBOOK \ud83c\udf10 PIPULATE WORKFLOW\n \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550 \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n [ ] Cell 1: Import data \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 Step 1: Data Input \u2502\n \u25bc \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n [\u25b6] Cell 2: Process data \u2502 hx_trigger=\"load\"\n \u2502 \u25bc\n \u25bc \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n [ ] Cell 3: Generate report \u2502 Step 2: Processing \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc \u2502 hx_trigger=\"load\"\n [ ] Cell 4: Export results \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \ud83c\udfaf \"Run All Cells\" Button \u2550\u2550\u2550\u25ba \u2502 Step 3: Export \u2502\n Executes top-to-bottom \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n Same mental model, same execution flow!\n But with persistent state, a web UI and\n not having to look at the Python code \ud83d\udeab\ud83d\udc0d.\n```\n\n### LLM Integration (Ollama) <!-- key: llm-integration-ollama -->\n\nIntegration with a local Ollama instance provides AI capabilities without external API calls:\n* **Privacy:** Prompts and responses stay local.\n* **Cost-Effective:** No per-token charges; run continuously using local resources.\n* **Streaming Support:** Real-time interaction via WebSockets.\n* **Bounded Context:** Manages conversation history effectively.\n* **App State Awareness:** Grepping your server log reveals full application state.\n* **Tool Calling:** Local LLM is an MCP client with a growing list of abilities\n - Workflow assistance\n - Browser automation\n - Debugging\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Local Ollama \u2502 - No API keys needed\n \u2502 Server \u2502 - Completely private processing\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u2502 Streaming via WebSocket\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Pipulate App \u2502 - Monitors WS for MCP tool-call commands\n \u2502(WebSocket Client)\u2502 - Parses responses in real-time\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u2502 In-memory or DB backed\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Bounded \u2502 - Manages context window (~128k)\n \u2502 Chat History \u2502 - Enables RAG / tool integration\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### Multi-OS & CUDA Support (Nix) <!-- key: multi-os-cuda-support-nix -->\n\nNix Flakes ensure a consistent environment across Linux, macOS, and Windows (via WSL), optionally leveraging CUDA GPUs if detected.\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Linux / macOS \u2502 - Write code once, run anywhere\n \u2502 Windows (WSL) \u2502 - Consistent dev environment via Nix\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 - As if Homebrew but across all OSes\n \u2502\n \u2502 Nix manages dependencies\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 CUDA Support \u2502 - Auto-detects NVIDIA GPU w/ CUDA\n \u2502 (if present) \u2502 - Uses GPU for LLM acceleration\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 - Falls back to CPU if no CUDA\n```\n\n### UI Layout <!-- key: ui-layout -->\n\nThe application interface is organized into distinct areas:\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Navigation \u25c4\u2500\u2500 Search, Profiles,\n \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 Apps, Settings\n \u2502 \u2502 \u2502\n Workflow, \u2500\u2500\u25ba Main Area \u2502 Chat \u2502\n App UI \u2502 (Pipeline) \u2502 Interface \u25c4\u2500\u2500 LLM Interaction\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### UI Component Hierarchy: Complete DOM Structure with IDs & ARIA Labels\n\n**Critical for AI assistants:** All UI components use semantic IDs and comprehensive ARIA labeling for accessibility and automation.\n\n```\n\ud83c\udfe0 home (Root Component)\n\u251c\u2500\u2500 \ud83d\udce6 create_outer_container()\n\u2502 \u251c\u2500\u2500 \ud83e\udded create_nav_group() [id='nav-group', role='navigation', aria-label='Main navigation']\n\u2502 \u2502 \u251c\u2500\u2500 \ud83d\udd0d nav_search_container [role='search', aria-label='Plugin search']\n\u2502 \u2502 \u2502 \u251c\u2500\u2500 Input [id='nav-plugin-search', role='searchbox', aria-label='Search plugins']\n\u2502 \u2502 \u2502 \u2514\u2500\u2500 Div [id='search-results-dropdown', role='listbox', aria-label='Search results']\n\u2502 \u2502 \u251c\u2500\u2500 \ud83d\udc64 create_profile_menu() [id='profile-dropdown-menu', aria-label='Profile management']\n\u2502 \u2502 \u2502 \u251c\u2500\u2500 Summary [id='profile-id', aria-label='Profile selection menu']\n\u2502 \u2502 \u2502 \u2514\u2500\u2500 Ul [role='menu', aria-label='Profile options', aria-labelledby='profile-id']\n\u2502 \u2502 \u251c\u2500\u2500 \u26a1 create_app_menu() [id='app-dropdown-menu', aria-label='Application selection']\n\u2502 \u2502 \u2502 \u251c\u2500\u2500 Summary [id='app-id', aria-label='Application menu']\n\u2502 \u2502 \u2502 \u2514\u2500\u2500 Ul [role='menu', aria-label='Application options', aria-labelledby='app-id']\n\u2502 \u2502 \u251c\u2500\u2500 \ud83c\udf0d create_env_menu() [id='env-dropdown-menu', data-testid='environment-dropdown-menu']\n\u2502 \u2502 \u2502 \u251c\u2500\u2500 Summary [id='env-id', aria-label='Environment selection menu']\n\u2502 \u2502 \u2502 \u2514\u2500\u2500 Ul [role='menu', aria-label='Environment options', aria-labelledby='env-id']\n\u2502 \u2502 \u2514\u2500\u2500 \u2699\ufe0f poke_section [id='poke-dropdown-menu']\n\u2502 \u2502 \u251c\u2500\u2500 Summary [id='poke-summary']\n\u2502 \u2502 \u2514\u2500\u2500 Div [id='nav-flyout-panel']\n\u2502 \u251c\u2500\u2500 \ud83d\udcf1 main-grid\n\u2502 \u2502 \u251c\u2500\u2500 \ud83d\udccb create_grid_left() [id='grid-left-content'] \u2192 Workflow Steps/Cells Display\n\u2502 \u2502 \u2502 \u251c\u2500\u2500 content_to_render (Dynamic workflow content)\n\u2502 \u2502 \u2502 \u2514\u2500\u2500 scroll_to_top [id='scroll-to-top-link']\n\u2502 \u2502 \u2514\u2500\u2500 \ud83e\udd16 create_chat_interface() [id='chat-interface', role='complementary', aria-label='AI Assistant Chat']\n\u2502 \u2502 \u251c\u2500\u2500 H2 [APP_NAME + ' Chatbot']\n\u2502 \u2502 \u251c\u2500\u2500 Div [id='msg-list', role='log', aria-label='Chat conversation', aria-live='polite']\n\u2502 \u2502 \u2514\u2500\u2500 Form [role='form', aria-label='Chat input form']\n\u2502 \u2502 \u251c\u2500\u2500 Textarea [id='msg', role='textbox', aria-label='Chat message input', aria-multiline='true']\n\u2502 \u2502 \u251c\u2500\u2500 Button [id='send-btn', aria-label='Send message to AI assistant']\n\u2502 \u2502 \u2514\u2500\u2500 Button [id='stop-btn', aria-label='Stop AI response streaming']\n\u2502 \u2514\u2500\u2500 \ud83d\udd27 HTMX Refresh Listeners\n\u2502 \u251c\u2500\u2500 Div [id='profile-menu-refresh-listener', hx_target='#profile-dropdown-menu']\n\u2502 \u2514\u2500\u2500 Div [id='app-menu-refresh-listener', hx_target='#app-dropdown-menu']\n```\n\n### \ud83c\udfaf Key HTMX Targets for AI Browser Automation\n\n**Navigation Updates:**\n- `#profile-dropdown-menu` - Profile menu refresh target\n- `#app-dropdown-menu` - App menu refresh target \n- `#search-results-dropdown` - Live search results\n- `#nav-flyout-panel` - Settings flyout panel\n\n**Content Areas:**\n- `#grid-left-content` - Main workflow display area\n- `#msg-list` - Chat conversation history\n- `body` - Full page navigation refreshes\n\n**Interactive Elements:**\n- `#nav-plugin-search` - Real-time plugin search (300ms delay)\n- `#send-btn` / `#stop-btn` - Chat control buttons\n- `#scroll-to-top-link` - Scroll navigation aid\n\nThis structure enables AI assistants to programmatically interact with all UI components using semantic selectors and ARIA landmarks.\n\n### File Structure\n\n```plaintext\n .\n \u251c\u2500\u2500 .cursor/ # Bootstraps Radical Transparency (teaches AI to fish)\n \u2502 \u2514\u2500\u2500 rules/ # Framework rules (01_CRITICAL_PATTERNS.mdc, etc.)\n \u251c\u2500\u2500 .venv/ # Common Python environment for FastHTML, Jupyter & Cursor\n \u251c\u2500\u2500 browser_automation/ # Selenium browser control & DOM capture\n \u2502 \u251c\u2500\u2500 looking_at/ # Current browser DOM state for AI visibility\n \u2502 \u2514\u2500\u2500 *.py # Google search automation examples\n \u251c\u2500\u2500 cli.py # Command line interface for Pipulate operations\n \u251c\u2500\u2500 common.py # Base Class for DRY CRUD plugin app inheritance (todo)\n \u251c\u2500\u2500 data/\n \u2502 \u2514\u2500\u2500 data.db # AI-accessible SQLite for application state (server cookies)\n \u251c\u2500\u2500 downloads/ # Default location for workflow outputs (e.g., CSVs)\n \u251c\u2500\u2500 helpers/\n \u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 botify\n \u00a0\u00a0 \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 botify_api.ipynb # Git managed massive example notebook, produces docs\n \u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 workflow # Workflow workshop, lots of tools that make WET DRY\n \u00a0\u00a0 \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 create_workflow.py # Example of what might be found there\n \u2502 \u2514\u2500\u2500 prompt_foo.py # Bundles XML code payloads for massive 1-shot AI prompts\n \u251c\u2500\u2500 logs/\n \u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 server-1.log # N-rotations of server log per run per config\n \u2502 \u2514\u2500\u2500 server.log # The server log of most recent run, contains app state\n \u251c\u2500\u2500 static/ # JS, CSS, images, icons\n \u251c\u2500\u2500 plugins/ # Workflow plugins (010_introduction.py, 400_trifecta.py, etc.)\n \u251c\u2500\u2500 pyproject.toml # Python packaging configuration and metadata\n \u251c\u2500\u2500 training/ # Markdown files for AI context/prompts\n \u251c\u2500\u2500 vulture_whitelist.py # Code analysis whitelist for unused code detection\n \u251c\u2500\u2500 flake.nix # Infrastructure as Code & all system-versions for AI\n \u251c\u2500\u2500 LICENSE # It's MIT\n \u251c\u2500\u2500 install.sh # \"Magic cookie\" installation script (curl | sh)\n \u251c\u2500\u2500 mcp_tools.py # MCP protocol tools - the AI assistant interface\n \u251c\u2500\u2500 notebook_introduction_local.ipynb # Editable (non-auto-updating) copy of botify_api.ipynb\n \u251c\u2500\u2500 README.md # This file\n \u251c\u2500\u2500 requirements.txt # Python dependencies (managed by Nix)\n \u2514\u2500\u2500 server.py # Main application entry point\n```\n\n--------------------------------------------------------------------------------\n\n## Critical Implementation Patterns for LLMs\n\n**These patterns are essential for LLMs working with Pipulate and are frequently missed:**\n\n### 1. The Auto-Key Generation Pattern (MOST CRITICAL) <!-- key: auto-key-generation-pattern -->\n\n<!-- START_ASCII_ART: auto-key-generation-pattern -->\n```\n\ud83d\udcdd AUTO-KEY GENERATION FLOW\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 POST \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 HX-Refresh \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Empty Form \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba \u2502 Server \u2502 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba \u2502 Page Reload \u2502\n\u2502 Submit \u23ce \u2502 /init \u2502 Response \u2502 Header \u2502 Trigger \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25b2 \u2502\n \u2502 \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 User Hits \u2502 \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2502 Auto-Key \u2502 \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2502 landing() \u2502\n\u2502 Enter Again \u2502 Ready! \u2502 Populated \u2502 Generates \u2502 Method \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n<!-- END_ASCII_ART: auto-key-generation-pattern -->\n\nWhen a user hits Enter on an empty key field, this specific sequence occurs:\n\n1. **Form Submission**: POSTs to `/{APP_NAME}/init` with empty `pipeline_id`\n2. **Server Response**: The `init` method MUST return an `HX-Refresh` response:\n ```python\n if not user_input:\n from starlette.responses import Response\n response = Response('')\n response.headers['HX-Refresh'] = 'true'\n return response\n ```\n3. **Page Reload**: HTMX triggers a full page reload\n4. **Auto-Key Population**: The `landing()` method calls `pip.generate_pipeline_key(self)` to populate the input field\n5. **User Interaction**: User hits Enter again to start the workflow\n\n### 2. The Chain Reaction Pattern: The `run_all_cells()` Breakthrough\n\nPipulate uses HTMX-driven step progression powered by the brilliantly named `run_all_cells()` method:\n\n1. **Initial Trigger**: After `init`, the `run_all_cells()` method initializes the workflow just like Jupyter's \"Run All Cells\"\n2. **Perfect Mental Model**: The method name creates immediate understanding \u2014 workflows execute top-to-bottom like notebook cells\n3. **Step Handlers**: Each step has GET (display) and POST (submit) handlers\n4. **Automatic Progression**: Completed steps trigger next step with `hx_trigger=\"load\"`\n5. **State Persistence**: Each step stores data in pipeline state\n6. **Pedagogical Brilliance**: The naming makes the system instantly intuitive for developers and AI assistants\n\n**Example: The `run_all_cells()` Pattern in Action**\n\n```python\n# \u2705 CORRECT: Use the run_all_cells() method for workflow initialization\nasync def init(self, request):\n \"\"\"Initialize workflow using the run_all_cells pattern\"\"\"\n return pip.run_all_cells(app_name, steps)\n\n# \u274c ANTI-PATTERN: Manual placeholder creation\nasync def init(self, request):\n \"\"\"Manual approach \u2014 harder to understand and maintain\"\"\"\n first_step_id = steps[0].id\n return Div(\n Div(id=first_step_id, hx_get=f'/{app_name}/{first_step_id}', hx_trigger='load'),\n id=f\"{app_name}-container\"\n )\n```\n\nThe `run_all_cells()` method encapsulates the workflow initialization pattern and creates an immediate mental connection to Jupyter notebooks.\n\n### 3. APP_NAME vs. Filename Distinction <!-- key: app-name-vs-filename -->\n\n<!-- START_ASCII_ART: app-name-vs-filename -->\n```\n\ud83d\udcc2 FILENAME vs APP_NAME DISTINCTION\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 CRITICAL SEPARATION \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u2502\n\u2502 \ud83d\udcc1 FILENAME: 200_workflow_genesis.py \u2502\n\u2502 \u251c\u2500\u2500 \ud83c\udf10 Determines public URL: /workflow_genesis \u2502\n\u2502 \u2514\u2500\u2500 \ud83d\udcca Controls menu order: 200 \u2502\n\u2502 \u2502\n\u2502 \ud83c\udff7\ufe0f APP_NAME: \"workflow_genesis_internal\" \u2502\n\u2502 \u251c\u2500\u2500 \ud83d\udcbe Database table identifier \u2502\n\u2502 \u251c\u2500\u2500 \ud83d\udd12 MUST REMAIN STABLE (data integrity) \u2502\n\u2502 \u2514\u2500\u2500 \ud83d\udeab NEVER change after deployment \u2502\n\u2502 \u2502\n\u2502 \u26a0\ufe0f DANGER: Changing APP_NAME = Orphaned Data \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n<!-- END_ASCII_ART: app-name-vs-filename -->\n\n**Critical for data integrity:**\n\n* **Filename** (e.g., `200_workflow_genesis.py`): Determines public URL endpoint and menu ordering\n* **APP_NAME Constant** (e.g., `APP_NAME = \"workflow_genesis_internal\"`): Internal identifier that MUST REMAIN STABLE\n\n### 4. State Management via DictLikeDB\n\n* State stored as JSON blobs in pipeline table\n* Accessed via `pip.get_step_data()` and `pip.set_step_data()`\n* All state changes are transparent and observable\n\n### 5. Plugin Discovery System <!-- key: plugin-discovery-system -->\n\n<!-- START_ASCII_ART: plugin-discovery-system -->\n```\n\ud83d\udcc1 PLUGIN DISCOVERY SYSTEM\nplugins/\n\u251c\u2500\u2500 010_introduction.py \u2705 Registered as \"introduction\" (menu order: 1)\n\u251c\u2500\u2500 020_profiles.py \u2705 Registered as \"profiles\" (menu order: 2)\n\u251c\u2500\u2500 hello_flow (Copy).py \u274c SKIPPED - Contains \"()\"\n\u251c\u2500\u2500 xx_experimental.py \u274c SKIPPED - \"xx_\" prefix\n\u2514\u2500\u2500 200_workflow_genesis.py \u2705 Registered as \"workflow_genesis\" (menu order: 20)\n\n \ud83d\udcca AUTO-REGISTRATION RULES:\n \u2705 Numeric prefix \u2192 Menu ordering + stripped for internal name\n \u274c Parentheses \"()\" \u2192 Development copies, skipped\n \u274c \"xx_\" prefix \u2192 Work-in-progress, skipped\n \ud83d\udd27 Must have: landing() method + name attributes\n \ud83d\udc89 Auto dependency injection via __init__ signature\n```\n<!-- END_ASCII_ART: plugin-discovery-system -->\n\n* Files in `plugins/` directory are auto-discovered\n* Numeric prefixes control menu ordering\n* Classes must have `landing` method and name attributes\n* Automatic dependency injection based on `__init__` signature\n\n## Workflow Development Helper Scripts\n\nPipulate includes sophisticated helper scripts for workflow development:\n\n### `create_workflow.py`\nCreates new workflows from templates:\n```bash\npython create_workflow.py workflow.py MyWorkflow my_workflow \\\n \"My Workflow\" \"Welcome message\" \"Training prompt\" \\\n --template trifecta --force\n```\n\n### `splice_workflow_step.py`\nAdds steps to existing workflows:\n```bash\npython splice_workflow_step.py workflow.py --position top\npython splice_workflow_step.py workflow.py --position bottom\n```\n\n### Template System <!-- key: workflow-template-system -->\n\n<!-- START_ASCII_ART: workflow-template-system -->\n```\n\ud83c\udfd7\ufe0f WORKFLOW TEMPLATE SYSTEM\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 BLANK TEMPLATE \u2502 \u2502TRIFECTA TEMPLATE\u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Step 1 \u2502 \u2502 \u2502 \u2502 Step 1 \u2502 \u2502\n\u2502 \u2502 (Minimal) \u2502 \u2502 \u2502 \u2502 (Input) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 VS \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u2502 \u2502 \u2502 \u2502\n\u2502 Quick Start \u2502 \u2502 \u25bc \u2502\n\u2502 Single Purpose \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 Step 2 \u2502 \u2502\n \u2502 \u2502 (Process) \u2502 \u2502\ncreate_workflow.py \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n--template blank \u2502 \u2502 \u2502\n \u2502 \u25bc \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Step 3 \u2502 \u2502\n \u2502 \u2502 (Output) \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502\n \u2502 Full Pattern \u2502\n \u2502 Complete Flow \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n create_workflow.py\n --template trifecta\n```\n<!-- END_ASCII_ART: workflow-template-system -->\n\n* `blank`: Minimal workflow with one step\n* `trifecta`: Three-step workflow pattern\n* Automatic method generation and insertion\n\n### Workflow Reconstruction System <!-- key: workflow-reconstruction-system -->\n\n**The Revolutionary Alternative to OOP Inheritance:** Atomic transplantation of workflow components using intelligent pattern matching and AST precision.\n\n<!-- START_ASCII_ART: workflow-reconstruction-system -->\n```\n\ud83e\uddec WORKFLOW RECONSTRUCTION: ATOMIC TRANSPLANTATION\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n OLD WORKFLOW WORKFLOW UPDATED WORKFLOW\n (Atomic Source) RECONSTRUCTOR (Incremental Gen)\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \ud83e\uddec Components: \u2502 AST \u2502 \ud83c\udfaf Pattern \u2502 AST \u2502 \u2728 Generated: \u2502\n \u2502 \u2502 \u2500\u2500\u2500\u25ba \u2502 Matching \u2502 \u2500\u2500\u2500\u25ba \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502step_params* \u2502 \u2502 \u2502 Bundle Type 1: \u2502 \u2502 \u2502step_params* \u2502 \u2502 \u2705\n \u2502 \u2502step_optim* \u2502 \u2502 \u2502 Auto-Registered \u2502 \u2502 \u2502step_optim* \u2502 \u2502 \u2705 \n \u2502 \u2502parameter* \u2502 \u2502 \u2502 Methods \u2502 \u2502 \u2502parameter* \u2502 \u2502 \u2705\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502 \u2502 Bundle Type 2: \u2502 \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 Custom Routes \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502_process \u2502 \u2502 \u2502 (_process, \u2502 \u2502 \u2502_process \u2502 \u2502 \u2705\n \u2502 \u2502preview \u2502 \u2502 \u2502 preview) \u2502 \u2502 \u2502preview \u2502 \u2502 \u2705\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n\ud83d\udd04 COMPLETE LIFECYCLE: Test \u2192 Validate \u2192 Production \u2192 Cleanup\n \n --suffix 5 --target new_name --target same_name git status\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n param_buster5 advanced_params param_buster (in-place) (shows cruft)\n (safe testing) (new workflow) (git history preserved) (clean up!)\n\n\ud83c\udfaf WHY IT WORKS: Lightning in a Bottle\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 \u2728 Pattern Matching: No manual markers needed \u2502\n\u2502 \ud83d\udd27 AST Precision: Syntactically perfect code generation \u2502 \n\u2502 \ud83c\udfad Inheritance Alternative: Compose without complex super() chains \u2502\n\u2502 \ud83e\uddea Safe Testing: Incremental validation without production risk \u2502\n\u2502 \ud83d\udcda Git Continuity: In-place updates preserve development history \u2502\n\u2502 \ud83e\uddf9 Systematic Cleanup: Prevents file cruft accumulation \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\n workflow_reconstructor.py --template botify_trifecta \n --source parameter_buster\n --suffix 5\n```\n<!-- END_ASCII_ART: workflow-reconstruction-system -->\n\n**The System That Eliminates Bootstrap Paradox:**\n* **Atomic Sources:** Battle-tested workflows become component libraries\n* **Pattern Matching:** Intelligent detection via `_process`, `preview` patterns\n* **AST Transplantation:** Surgical precision without syntax errors\n* **Complete Lifecycle:** Development \u2192 Testing \u2192 Production \u2192 Cleanup\n\n## \ud83d\udccb Quick Reference Card\n\n### Essential Commands\n```bash\n# Development workflow\ncd ~/pipulate && nix develop # Start Pipulate\nnix develop .#quiet # Start without auto-services\npython server.py # Manual server start\ngit pull && nix develop # Update to latest\n\n# Create new workflows \npython helpers/workflow/create_workflow.py my_workflow.py MyClass my_internal_name\npython helpers/workflow/splice_workflow_step.py my_workflow.py --position top\n\n# Plugin naming conventions\n010_my_plugin.py # Active plugin (menu order 1)\nxx_my_plugin.py # Disabled during development \nmy_plugin (Copy).py # Ignored development copy\n```\n\n### Key URLs & Ports\n- **Pipulate App**: `http://localhost:5001`\n- **JupyterLab**: `http://localhost:8888` \n- **Local AI Chat**: Built into the Pipulate interface\n- **Logs**: `tail -f logs/server.log` for debugging\n\n### Critical Patterns for AI Assistants\n```python\n# Auto-key generation flow\nif not user_input:\n response = Response('')\n response.headers['HX-Refresh'] = 'true'\n return response\n\n# Workflow initialization\nreturn pip.run_all_cells(app_name, steps)\n\n# State management\ndata = pip.get_step_data(step_id)\npip.set_step_data(step_id, updated_data)\n```\n\n### File Structure Quick Reference\n```\nplugins/ # Your workflows (auto-discovered)\n\u251c\u2500\u2500 010_introduction.py # Menu order 1\n\u251c\u2500\u2500 xx_draft.py # Disabled (xx_ prefix)\n\u2514\u2500\u2500 draft (Copy).py # Ignored (parentheses)\n\nmcp_tools.py # AI assistant interface \ncommon.py # Base classes for workflows\nbrowser_automation/ # Selenium automation tools\nlogs/server.log # Debug everything here\ndata/data.db # SQLite application state\n```\n\n## Common LLM Implementation Mistakes <!-- key: llm-implementation-mistakes -->\n\n<!-- START_ASCII_ART: llm-implementation-mistakes -->\n```\n\ud83d\udea8 LLM IMPLEMENTATION MISTAKE PREVENTION\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 COMMON PITFALLS \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u274c Missing HX-Refresh \u2502 \u2705 if not user_input: \u2502\n\u2502 Response \u2502 response.headers['HX- \u2502\n\u2502 \u2502 Refresh'] = 'true' \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u274c Wrong Key Generation \u2502 \u2705 pip.generate_pipeline_ \u2502\n\u2502 Method \u2502 key(self) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u274c Broken Chain Reaction \u2502 \u2705 hx_trigger=\"load\" \u2192 \u2502\n\u2502 Pattern \u2502 Automatic progression \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 \u274c APP_NAME Changes \u2502 \u2705 NEVER modify after \u2502\n\u2502 (Data Orphaning) \u2502 deployment \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n<!-- END_ASCII_ART: llm-implementation-mistakes -->\n\n**LLMs frequently make these errors:**\n\n1. **Missing HX-Refresh Response**: Forgetting to return the refresh response for empty keys\n2. **Incorrect Key Generation**: Not using `pip.generate_pipeline_key(self)` properly\n3. **Missing Cursor Positioning**: Forgetting the `_onfocus` attribute for user experience\n4. **Wrong Route Handling**: Not understanding the difference between landing page and init routes\n5. **State Inconsistency**: Not properly handling the key generation and storage flow\n6. **APP_NAME Changes**: Modifying APP_NAME after deployment, orphaning existing data\n7. **Chain Reaction Breaks**: Not properly implementing the HTMX step progression pattern\n\n## Key Design Guidelines & Patterns\n\nThese \"speedbumps\" reinforce Pipulate's core philosophy:\n\n * **Local vs. Enterprise Mindset:** Embrace local-first simplicity. Avoid patterns designed for distributed, multi-tenant systems.\n * **JSON State Management (Workflows):** Keep workflow state in self-contained steps within a single JSON blob per run. Avoid complex state machines or external step tracking.\n * **Database (MiniDataAPI):** Use the simple schema definition and access patterns provided. Avoid heavy ORMs.\n * **Workflow Pattern:** Ensure workflows are linear and state is explicitly passed or saved at each step. Avoid complex async task chaining that obscures state.\n * **UI Rendering Pattern:** Generate HTML directly from Python components via FastHTML. Avoid template engines.\n * **WebSocket Pattern:** Use the dedicated `Chat` class for managing LLM interactions. Avoid raw WebSocket handling elsewhere.\n * **Workflow Progression Pattern:** Workflows use an explicit chain reaction pattern with `hx_trigger=\"load\"` to manage step progression. This pattern must be preserved exactly as implemented. See the workflow documentation for details.\n\n## Internal Components <!-- key: core-concepts-internal-components -->\n\n * **Monitoring:** A file system watchdog monitors code changes. Valid changes trigger an automatic, monitored server restart via Uvicorn, facilitating live development.\n\n```\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 File System \u2502 Changes \u2502 AST Syntax \u2502 Checks Code\n \u2502 Watchdog \u2502 Detects \u2502 Checker \u2502 Validity\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 Valid Change \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Uvicorn Server \u2502\u25c4\u2500\u2500\u2500 \u2502 Reload \u2502 Triggers Restart\n \u2502 (Handles HTTP, WS, SSE) \u2502 \u2502 Process \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n### Pipeline State Inspector & MCP Tools <!-- key: pipeline-state-inspector -->\n\nThe system provides comprehensive debugging and state inspection capabilities through MCP tools and real-time monitoring:\n\n```\n\ud83d\udcca PIPELINE STATE INSPECTOR\n\u251c\u2500\u2500\u2500 \ud83d\udd0d Discovering active workflows...\n\u251c\u2500\u2500\u2500 \u26a1 Found 3 running processes\n\u251c\u2500\u2500\u2500 \ud83c\udfaf Step 2/5: Processing data transformations\n\u2514\u2500\u2500\u2500 \u2728 Ready for next interaction!\n\n\ud83e\udd16 MCP TOOL CALLS\n\u251c\u2500\u2500\u2500 \ud83d\udce1 Connecting to Botify API...\n\u251c\u2500\u2500\u2500 \ud83d\udd10 Authentication successful\n\u251c\u2500\u2500\u2500 \ud83d\udcca Fetching schema (4,449 fields discovered!)\n\u2514\u2500\u2500\u2500 \ud83d\udcbe Caching results for lightning-fast access\n```\n\n-----\n\n## The Deeper Philosophy: Why This Matters <!-- key: deeper-philosophy -->\n\nPipulate represents more than technology choices \u2014 it's a statement about **computing sovereignty** and the future of human-AI collaboration.\n\n### The Bootstrap Paradox Solved <!-- key: bootstrap-paradox-solved -->\n\nHow do you \"lift yourself up by your bootstraps\"? The answer: **you're not a closed system**. Humans interact with reality through interfaces, just like software components. We are \"human apps with reality APIs.\"\n\nThis pattern repeats at every scale \u2014 from quantum fields to societies. The art lies in choosing the **right granularity** for these interfaces. Too abstract, and you lose control. Too granular, and you drown in complexity.\n\n### Every API is a Compromise <!-- key: every-api-is-compromise -->\n\nThe most successful APIs are **durable** \u2014 they become the \"laws of physics\" for their domain. In reality: the laws of physics, chemistry, biology. In tech: logic gates, operating systems, protocols like HTTP and HTML.\n\nPipulate chooses these durable APIs deliberately, creating a **future-proof foundation** for AI-assisted development that will outlast the current framework churn.\n\n### Writing for the Future <!-- key: writing-for-future -->\n\nThis documentation serves multiple audiences: humans seeking immediate productivity, and AI systems learning to collaborate with human creativity. The explicit, observable nature of Pipulate workflows creates **perfect training data** for the next generation of AI assistants.\n\n## Final Thoughts\n\nPipulate offers a unique approach to building local, AI-assisted applications, prioritizing simplicity, user control, and reproducibility over conventional scaling patterns. By understanding its core concepts and embracing its philosophy, developers can create powerful workflows, and end-users can benefit from AI guidance with complete flexibility in deployment options.\n\nRemember the guiding principles:\n\n1. **Keep it simple.**\n2. **Support local-first with cloud integration options.**\n3. **Embrace the unconventional while remaining practical.**\n4. **Choose durable foundations that work with any approach.**\n5. **Build for both human creativity and AI collaboration\u2014local or cloud.**\n\n**The Bottom Line:** Pipulate doesn't reject the modern AI ecosystem\u2014it provides a structured foundation that works with any AI service. Whether you're using Claude via API, ChatGPT for reasoning, or local models for privacy, Pipulate gives you the workflow framework to orchestrate them all effectively. It's not about choosing sides in the AI wars\u2014it's about having the right tool for any job.\n\n-----\n\n## Developer's Notes\n\n### The Pipulate Workshop\n\nThe repository includes not only polished plugins but also experimental scripts and notebooks under development (e.g., in the root directory or marked with `xx_` prefix in plugin directories). These represent ongoing work and exploration.\n\n### Plugin Development Conventions\n\n#### Auto-Registration Behavior\n\n * **Numeric Prefixes:** Files like `workflows/10_hello_flow.py` are registered as `hello_flow` (number stripped for internal name, used for menu order).\n * **Parentheses Skip:** Files with `()` in the name (e.g., `hello_flow (Copy).py`) are skipped \u2013 useful for temporary copies during development.\n * **`xx_` Prefix Skip:** Files prefixed with `xx_` (e.g., `xx_experimental_flow.py`) are skipped \u2013 useful for keeping unfinished work in the plugin directories without activating it.\n\n#### Workflow for Creating New Plugins\n\n1. **Copy:** Copy a template to `my_flow (Copy).py`.\n2. **Modify:** Develop your workflow. It won't auto-register yet.\n3. **Test:** Rename to `xx_my_flow.py`. The server should auto-reload. Test thoroughly.\n4. **Deploy:** Rename to `##_my_flow.py` to assign menu order and activate.\n\n#### Git History Considerations\n\nUse `git mv` for simple renames (like `xx_` to numbered prefix) to preserve history. Document more complex renames in commit messages.\n\n```bash\ngit mv workflows/xx_my_flow.py workflows/##_my_flow.py\ngit commit -m \"Feat: Promote workflow xx_my_flow.py to ##_my_flow.py\"\n```\n\n--------------------------------------------------------------------------------\n\n## About This README: Single Source of Truth Documentation <!-- key: about-this-readme -->\n\nThis README serves as the **upstream source of truth** for all Pipulate documentation across GitHub, Pipulate.com, and the built-in app documentation. Changes made here automatically cascade to all other documentation surfaces.\n\n### The ASCII Art Synchronization System\n\n```\n\ud83c\udf0a THE UPSTREAM TRUTH CASCADE\n\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n\n \ud83d\udcdc Source Code Reality (The Ultimate Truth)\n \u2502\n \u25bc\n \ud83d\udcc4 README.md (Single Source of Truth)\n \u251c\u2500 ASCII Art Blocks (Visual Truth)\n \u251c\u2500 HTML Comment Keys (Metadata)\n \u2514\u2500 80-Hyphen Pagination (Structure)\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc \u25bc\n \ud83c\udf10 GitHub Page \ud83d\udcda Pipulate.com \ud83d\udd27 Built-in Docs\n (Auto-display) (Jekyll Build) (Live Integration)\n \u2502 \u2502 \u2502\n \u25bc \u25bc \u25bc\n \ud83d\udcca Screenshots \ud83c\udfac Demos \ud83e\uddea Tests\n (Future) (Future) (Future)\n```\n\n**How it works:**\n- **ASCII Art Blocks**: Visual diagrams are automatically extracted and distributed to other documentation files\n- **HTML Comment Keys**: Headlines marked with `<!-- key: identifier -->` serve as reference anchors\n- **80-Hyphen Pagination**: Section dividers enable automatic document structuring\n- **Automatic Synchronization**: Running `python helpers/docs_sync/sync_ascii_art.py` updates all documentation\n\nThis creates **\"ASCII art peer pressure\"** \u2014 when visual diagrams change, they compel surrounding text to be updated for consistency, ensuring documentation accuracy across the entire ecosystem.\n\n### Roadmap\n\n**Core & Workflow Enhancements:**\n\n * Dev, Test, and Prod database switching\n * Saving source HTML and rendered DOM of any URL\n * Botify data export CSV save (incorporating robust polling)\n * Full web form field support (textarea, dropdown, checkboxes, radio buttons)\n * Generic support for Anywidgets\n * Utility for deleting garbage tables from plugin experimentation\n\n**AI / LLM Integration:**\n\n * LLM inspection of any local data object (RAG-style functionality)\n * Various memory types for LLM context (vector embedding, graph, key/val-store)\n * Enabling the local LLM to be an MCP Client\n\n**Automation & External Interaction:**\n\n * MCP Server for automated web browsing and similar tasks\n\n---\n\n## Included PrismJS Highlighting\n\nTHEMES\n- Okaidia ocodia 1.77KB\n\nLANGUAGES\n- CSS1.71KB\n- Markup + HTML + XML + SVG + MathML + SSML + Atom + RSS4.64KB\n- C-like0.83KB\n- JavaScript6.18KB\n- Bash + Shell + Shell zeitgeist87 8.96KB\n- Diff uranusjr 1.33KB\n- JSON + Web App Manifest CupOfTea696 0.58KB\n- JSON5 RunDevelopment 0.52KB\n- JSONP RunDevelopment 0.23KB\n- Liquid cinhtau 2.56KB\n- Lua Golmote 0.74KB\n- Markdown Golmote 10.43KB\n- Markup templating\n- Mermaid RunDevelopment 3.03KB\n- Nix Golmote 1.47KB\n- Python multipetros 2.45KB\n- Regex RunDevelopment 2.33KB\n- YAML hason 3.11KB\n\nPLUGINS\n- Line Highlight11.66KB\n- Line Numbers kuba-kubula 7.23KB\n- Toolbar mAAdhaTTah 5.63KB\n\n## Contributing\n\nContributions are welcome\\! Please adhere to the project's core philosophy:\n\n * Maintain Local-First Simplicity (No multi-tenant patterns, complex ORMs, heavy client-side state).\n * Respect Server-Side State (Use DictLikeDB/JSON for workflows, MiniDataAPI for CRUD).\n * Preserve the Workflow Pipeline Pattern (Keep steps linear, state explicit).\n * Honor Integrated Features (Don't disrupt core LLM/Jupyter integration unless enhancing local goals).\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](https://github.com/miklevin/pipulate/blob/main/LICENSE) file for details.\n\n## Resources\n\n**Background Articles:** <a href=\"https://mikelev.in/\">Mike Levin, AI SEO in NYC</a>\n\n**Enhanced Documentation:** <a href=\"https://pipulate.com/\">Pipulate AI SEO Software</a>\n\n**On GitHub:** <a href=\"https://github.com/miklevin/pipulate\">Pipulate on GitHub</a>\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Pipulate: Local First AI SEO Software",
"version": "1.1.1",
"project_urls": {
"Bug Tracker": "https://github.com/miklevin/pipulate/issues",
"Documentation": "https://pipulate.com/guide/",
"Homepage": "https://github.com/miklevin/pipulate/"
},
"split_keywords": [
"seo",
" ai",
" llm",
" automation",
" local-first",
" nix",
" fasthtml",
" htmx",
" agentic"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ccfd8d2b3c837e151bb9df72276bc9725ec79e9124b6e69e3bece337cbafebd9",
"md5": "34dd6168b60f378fe9c031331e8a0fc7",
"sha256": "2aebccc2c998f544fb0546ac8c068dbed7241ac04891a3a98aec9445f21acadb"
},
"downloads": -1,
"filename": "pipulate-1.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "34dd6168b60f378fe9c031331e8a0fc7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 579022,
"upload_time": "2025-07-08T15:50:50",
"upload_time_iso_8601": "2025-07-08T15:50:50.847689Z",
"url": "https://files.pythonhosted.org/packages/cc/fd/8d2b3c837e151bb9df72276bc9725ec79e9124b6e69e3bece337cbafebd9/pipulate-1.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5d351b92f28752d6b2cc54251deb1fcae0bd4c7fe00f44267e49e278e8a8a475",
"md5": "246b9acfbb195434ac7929b0eb6130cb",
"sha256": "6cfcdef678b5c1a1017fdd301071285936aec3696309aa32055967b2c02a0804"
},
"downloads": -1,
"filename": "pipulate-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "246b9acfbb195434ac7929b0eb6130cb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 570844,
"upload_time": "2025-07-08T15:50:52",
"upload_time_iso_8601": "2025-07-08T15:50:52.199871Z",
"url": "https://files.pythonhosted.org/packages/5d/35/1b92f28752d6b2cc54251deb1fcae0bd4c7fe00f44267e49e278e8a8a475/pipulate-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-08 15:50:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "miklevin",
"github_project": "pipulate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "autopep8",
"specs": []
},
{
"name": "isort",
"specs": []
},
{
"name": "pylint",
"specs": []
},
{
"name": "python-minimizer",
"specs": []
},
{
"name": "strip-docs",
"specs": []
},
{
"name": "tiktoken",
"specs": []
},
{
"name": "vulture",
"specs": []
},
{
"name": "yapf",
"specs": []
},
{
"name": "aiohttp",
"specs": []
},
{
"name": "beautifulsoup4",
"specs": []
},
{
"name": "blinker",
"specs": [
[
"==",
"1.7.0"
]
]
},
{
"name": "build",
"specs": []
},
{
"name": "extruct",
"specs": []
},
{
"name": "httpx",
"specs": []
},
{
"name": "ipynbname",
"specs": []
},
{
"name": "itables",
"specs": []
},
{
"name": "jupyter-ai",
"specs": []
},
{
"name": "jupyterlab",
"specs": []
},
{
"name": "jupyterlab-spellchecker",
"specs": []
},
{
"name": "jupytext",
"specs": []
},
{
"name": "loguru",
"specs": []
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "markdown-it-py",
"specs": []
},
{
"name": "nbstripout",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "pyfiglet",
"specs": []
},
{
"name": "python_fasthtml",
"specs": []
},
{
"name": "requests",
"specs": []
},
{
"name": "rich",
"specs": []
},
{
"name": "selenium",
"specs": []
},
{
"name": "termcolor",
"specs": []
},
{
"name": "twine",
"specs": []
},
{
"name": "webdriver-manager",
"specs": []
},
{
"name": "selenium-wire",
"specs": []
},
{
"name": "undetected-chromedriver",
"specs": []
},
{
"name": "selenium-stealth",
"specs": []
},
{
"name": "watchdog",
"specs": []
},
{
"name": "google-api-python-client",
"specs": []
},
{
"name": "google-auth",
"specs": []
},
{
"name": "scikit-learn",
"specs": []
},
{
"name": "PyYAML",
"specs": []
}
],
"lcname": "pipulate"
}
JSON
