# Loppers
**Extract source file skeletons using tree-sitter queries.**
Removes function implementations while preserving structure, signatures, and docstrings. Supports 17 programming languages with a clean, fully-typed Python API and comprehensive CLI.
**Requires: tree-sitter >= 0.25**
## Features
- ✅ **17 Languages** - Python, JS/TS, Java, Kotlin, Go, Rust, C/C++, C#, Ruby, PHP, Swift, Lua, Scala, Groovy, Objective-C
- ✅ **Smart Extraction** - Functions, methods, constructors, arrow functions, getters/setters
- ✅ **Preserved Elements** - Signatures, class definitions, imports, docstrings, decorators
- ✅ **All File Types** - Process any non-binary text files (code, markdown, JSON, YAML, etc.)
- ✅ **Binary Detection** - Automatically skips binary files
- ✅ **Ignore Patterns** - Built-in + custom .gitignore support
- ✅ **Fully Typed** - Complete type hints throughout
- ✅ **CLI & Library** - Use as command-line tool or Python library
## Quick Start
### Installation
```bash
# With uv (recommended)
uv pip install loppers
# With pip
pip install loppers
```
## Python API
The public API consists of 5 core functions:
### 1. `extract_skeleton(source: str, language: str) -> str`
Extract skeleton from source code by language identifier.
```python
from loppers import extract_skeleton
code = """
def calculate(x: int, y: int) -> int:
'''Calculate sum.'''
result = x + y
return result
"""
skeleton = extract_skeleton(code, "python")
print(skeleton)
```
Output:
```python
def calculate(x: int, y: int) -> int:
'''Calculate sum.'''
```
### 2. `get_skeleton(file_path: Path | str, *, add_header: bool = False) -> str`
Extract skeleton from a file by auto-detecting language from extension.
```python
from loppers import get_skeleton
skeleton = get_skeleton("src/main.py")
print(skeleton)
# With header showing file path
skeleton = get_skeleton("src/main.py", add_header=True)
# Output: "--- /path/to/src/main.py\n..."
```
**Raises:**
- `FileNotFoundError` - If file doesn't exist
- `ValueError` - If file language is unsupported
### 3. `find_files(root: str | Path, *, recursive: bool = True, ignore_patterns: Sequence[str] | None = None, use_default_ignore: bool = True, respect_gitignore: bool = True) -> list[str]`
Collect all non-binary text files from a root directory.
```python
from loppers import find_files
# Find all text files in src/ recursively (default)
files = find_files("src/")
# Returns file paths relative to root:
# ['main.py', 'utils.py', 'config.yaml', 'README.md']
# Non-recursive
files = find_files("src/", recursive=False)
# Custom ignore patterns (gitignore syntax)
files = find_files(
"src/",
ignore_patterns=["*.test.py", "venv/"],
use_default_ignore=True, # Still applies built-in patterns
respect_gitignore=True, # Still respects .gitignore
)
```
**Features:**
- Takes single root directory (not multiple paths)
- Returns file paths relative to root
- Automatically excludes binary files (images, archives, etc.)
- Respects `.gitignore` by default
- Supports custom gitignore-style ignore patterns
- Built-in patterns exclude node_modules, .git, __pycache__, build artifacts, etc.
- Works with ALL non-binary text files (code, markdown, JSON, YAML, etc.)
### 4. `get_tree(root: str | Path, *, recursive: bool = True, ignore_patterns: Sequence[str] | None = None, use_default_ignore: bool = True, respect_gitignore: bool = True, collapse_single_dirs: bool = False, show_sizes: bool = False) -> str`
Display formatted directory tree from a root directory.
```python
from loppers import get_tree
# Display tree of src/ directory recursively
tree = get_tree("src/")
print(tree)
# Non-recursive tree
tree = get_tree("src/", recursive=False)
# With custom ignore patterns
tree = get_tree("src/", ignore_patterns=["*.test.py"])
# Collapse deep single-child directories (useful for Java packages)
tree = get_tree("src/", collapse_single_dirs=True)
# Show file sizes in human-friendly format
tree = get_tree("src/", show_sizes=True)
# Combine multiple options
tree = get_tree("src/", collapse_single_dirs=True, show_sizes=True)
```
Output (with `collapse_single_dirs=True` and `show_sizes=True`):
```
.
├─ main/java/com/example
│ ├─ Source.java (2.3KB)
│ └─ Util.java (1.8KB)
└─ resources/config.yaml (512B)
```
### 5. `concatenate_files(root: str | Path, file_paths: Sequence[str | Path], *, extract: bool = True, ignore_not_found: bool = False) -> str`
Concatenate files with optional skeleton extraction. Useful when you already have a list of file paths that you want to combine.
```python
from loppers import concatenate_files, find_files
# Get list of files to concatenate
root = "src/"
files = find_files(root)
# Concatenate with skeleton extraction (default)
result = concatenate_files(root, files)
print(result)
# Concatenate without extraction (include original content)
result = concatenate_files(root, files, extract=False)
# Ignore files that don't exist or can't be processed
result = concatenate_files(root, files, ignore_not_found=True)
```
Output format:
```
--- path/to/file.py
def calculate(x: int) -> int:
'''Calculate.'''
--- path/to/other.js
function process() {
}
```
**Features:**
- Files are concatenated with headers showing their relative paths
- Each file separated by newlines
- Automatically extracts skeletons from code files (unless `extract=False`)
- Falls back to original content for unsupported file types
- Can optionally ignore files that don't exist or can't be processed
**Parameters:**
- `root` - Root directory (file paths are relative to this)
- `file_paths` - List of file paths (relative to root) to concatenate
- `extract` - Extract skeletons from code files (default `True`)
- `ignore_not_found` - Ignore files that cannot be found or processed (default `False`)
**Raises:**
- `FileNotFoundError` - If root doesn't exist or a file is not found (when `ignore_not_found=False`)
- `ValueError` - If no file paths provided or no files could be processed
- `NotADirectoryError` - If root is not a directory
### Utility Function
**`get_language(extension: str) -> str | None`** - Get language identifier from file extension.
```python
from loppers import get_language
get_language(".py") # "python"
get_language(".js") # "javascript"
get_language(".json") # None (no extraction for data files)
```
## Command-Line Interface
Loppers provides 4 subcommands for common tasks.
### Basic Usage
```bash
loppers --version
loppers --help
```
### 1. `extract` - Extract skeleton from file or stdin
Extract a single file's skeleton:
```bash
# From file
loppers extract file.py
loppers extract file.py -o skeleton.py
# From stdin with explicit language
echo 'def foo(): pass' | loppers extract -l python
# Verbose output
loppers extract file.py -v
```
**Options:**
- `FILE` - File to extract (omit for stdin)
- `-l, --language` - Language identifier (auto-detected from extension if FILE provided, required for stdin)
- `-o, --output` - Output file (default: stdout)
- `-v, --verbose` - Print status to stderr
### 2. `concatenate` - Concatenate files with optional skeleton extraction
Process root directory with automatic skeleton extraction:
```bash
# Recursive (default)
loppers concatenate src/
# Non-recursive
loppers concatenate --no-recursive src/
# Save to file
loppers concatenate src/ -o combined.txt
# Verbose with progress
loppers concatenate -v src/
# Include original files without extraction
loppers concatenate --no-extract src/
# Custom ignore patterns
loppers concatenate -I "*.test.py" -I "venv/" src/
# Disable default ignores
loppers concatenate --no-default-ignore src/
# Don't respect .gitignore
loppers concatenate --no-gitignore src/
```
**Features:**
- Processes a single root directory (paths relative to root)
- Includes ALL non-binary text files (code, markdown, JSON, YAML, etc.)
- Automatically extracts skeletons for supported code files
- Includes original content for unsupported file types (graceful degradation)
- Each file prefixed with `--- filepath` header (relative path)
- Verbose mode shows extraction status for each file
**Options:**
- `root` - Root directory to process (required)
- `-o, --output` - Output file (default: stdout)
- `--no-extract` - Include original files without extraction
- `-I, --ignore-pattern` - Add custom ignore pattern (gitignore syntax, can be used multiple times)
- `--no-default-ignore` - Disable built-in ignore patterns
- `--no-gitignore` - Don't respect .gitignore
- `--no-recursive` - Don't recursively traverse directories
- `-v, --verbose` - Print status to stderr
### 3. `tree` - Show directory tree of discovered files
Display a formatted tree of all discovered files:
```bash
# Recursive tree (default)
loppers tree src/
# Non-recursive
loppers tree --no-recursive src/
# Save tree to file
loppers tree src/ -o tree.txt
# With ignore patterns
loppers tree -I "*.test.py" src/
# Collapse deep single-child directories (useful for Java packages)
loppers tree --collapse-single-dirs src/
# Show file sizes in human-friendly format
loppers tree --show-sizes src/
# Combine multiple options
loppers tree --collapse-single-dirs --show-sizes src/
```
**Options:**
- `root` - Root directory to process (required)
- `-o, --output` - Output file (default: stdout)
- `-I, --ignore-pattern` - Add custom ignore pattern
- `--no-default-ignore` - Disable built-in ignore patterns
- `--no-gitignore` - Don't respect .gitignore
- `--no-recursive` - Non-recursive tree
- `--collapse-single-dirs` - Collapse directories with single children (e.g., `java/com/example` becomes one line)
- `--show-sizes` - Show file sizes in human-friendly format (e.g., "1.2KB", "5.0MB")
- `-v, --verbose` - Print status to stderr
**Collapse Example:**
Without collapse:
```
.
└─ src
└─ main
└─ java
└─ com
└─ example
├─ Source.java
└─ Util.java
```
With `--collapse-single-dirs`:
```
.
└─ src/main/java/com/example
├─ Source.java
└─ Util.java
```
### 4. `files` - List all discovered files
Print one discovered file per line (relative to root):
```bash
# List all files recursively (default)
loppers files src/
# Save list to file
loppers files src/ -o file_list.txt
# Non-recursive
loppers files --no-recursive src/
# With custom ignores
loppers files -I "*.md" src/
```
**Options:**
- `root` - Root directory to process (required)
- `-o, --output` - Output file (default: stdout)
- `-I, --ignore-pattern` - Add custom ignore pattern
- `--no-default-ignore` - Disable built-in ignore patterns
- `--no-gitignore` - Don't respect .gitignore
- `--no-recursive` - Non-recursive listing
- `-v, --verbose` - Print status to stderr
## Examples: Before and After
### Python Example
**Before:**
```python
class Calculator:
def __init__(self, name: str):
"""Initialize calculator."""
self.name = name
self._setup()
def process(self, data):
"""Process data."""
result = []
for item in data:
result.append(item * 2)
return result
```
**After:**
```python
class Calculator:
def __init__(self, name: str):
"""Initialize calculator."""
def process(self, data):
"""Process data."""
```
### JavaScript/TypeScript Example
**Before:**
```typescript
class UserService {
constructor(baseUrl: string) {
this.baseUrl = baseUrl;
this.cache = {};
}
async getUser(id: string) {
if (this.cache[id]) return this.cache[id];
const user = await fetch(this.baseUrl + '/' + id);
return user.json();
}
}
```
**After:**
```typescript
class UserService {
constructor(baseUrl: string) {
}
async getUser(id: string) {
}
}
```
### Java Example
**Before:**
```java
public class UserService {
private String baseUrl;
public UserService(String baseUrl) {
this.baseUrl = baseUrl;
this.validate();
}
public User getUserById(String id) {
Database db = new Database();
return db.query(id);
}
private void validate() {
if (baseUrl == null) {
throw new IllegalArgumentException("BaseUrl required");
}
}
}
```
**After:**
```java
public class UserService {
private String baseUrl;
public UserService(String baseUrl) {
}
public User getUserById(String id) {
}
private void validate() {
}
}
```
## Supported Languages
| Language | Features |
|----------|----------|
| **Python** | Functions, methods, `__init__`, `@property`, docstrings |
| **JavaScript/TypeScript** | Functions, arrow functions, methods, async/await |
| **Java** | Methods, constructors, static methods, annotations |
| **Kotlin** | Functions, methods, properties (getters/setters) |
| **Go** | Functions, methods, closures |
| **Rust** | Functions, methods, closures |
| **C/C++** | Functions, methods, constructors |
| **C#** | Methods, properties (get/set), async/await |
| **Ruby** | Methods, singleton methods, blocks |
| **PHP** | Functions, methods, closures |
| **Swift** | Functions, methods, closures |
| **Lua** | Functions, local functions |
| **Scala** | Functions, methods, closures |
| **Groovy** | Functions, methods, closures |
| **Objective-C** | Methods, instance/class methods |
### What Gets Preserved
- ✅ Function/method signatures
- ✅ Parameter types and defaults
- ✅ Return types
- ✅ Class definitions
- ✅ Import statements
- ✅ Comments
- ✅ Python docstrings
- ✅ Decorators
- ✅ Access modifiers (public, private, protected)
### What Gets Removed
- ❌ Function/method bodies
- ❌ Local variable assignments
- ❌ Logic and implementation details
- ❌ Nested function implementations
### Known Limitations
- Concise arrow functions (`const f = x => x * 2`) - no body to remove
- Python lambdas - no body to remove
- Some edge cases with getters/setters in JavaScript/TypeScript
## How It Works
Loppers uses **tree-sitter** queries to parse source code into Abstract Syntax Trees (AST) and intelligently remove function/method bodies while preserving:
- Function/method signatures
- Class and interface definitions
- Import statements
- Python docstrings
- Comments
- Decorators
- Type hints
Each language has custom tree-sitter query patterns that capture function/method body nodes, which are then removed line-by-line.
## Development
### Setup
```bash
# Install with dev dependencies
uv sync
```
### Running Tests
```bash
# All tests
uv run pytest
# Verbose output
uv run pytest -v
# With coverage
uv run pytest --cov=loppers --cov-report=html
# Specific test
uv run pytest tests/test_loppers.py::test_python_extraction
```
### Code Quality
```bash
# Check and fix
uv run ruff check . --fix
# Format
uv run ruff format .
# All checks
uv run ruff check . --fix && uv run ruff format .
```
### Adding New Languages
To add support for a new language:
1. **Find the tree-sitter query** - Use the [tree-sitter playground](https://tree-sitter.github.io/tree-sitter/playground) to develop a query that captures function bodies
2. **Add to LANGUAGE_CONFIGS** in `src/loppers/loppers.py`:
```python
LANGUAGE_CONFIGS["mylang"] = LanguageConfig(
name="mylang",
body_query="(function_definition body: (block) @body)",
)
```
3. **Add file extensions** to `src/loppers/extensions.py`:
```python
EXTENSION_TO_LANGUAGE = {
".ml": "mylang",
".mli": "mylang",
}
```
4. **Write a test** in `tests/test_loppers.py`:
```python
def test_mylang_extraction(self):
code = "fun hello() { print('hi') }"
skeleton = extract_skeleton(code, "mylang")
assert "fun hello()" in skeleton
assert "print" not in skeleton
```
5. **Run tests** to verify everything works
## Project Structure
```
loppers/
├── src/loppers/
│ ├── __init__.py # Public API: extract_skeleton, get_skeleton, find_files, get_tree, concatenate_files
│ ├── loppers.py # Core extraction logic with SkeletonExtractor class
│ ├── source_utils.py # Convenience API and file operations
│ ├── extensions.py # Language extension mapping
│ ├── ignore_patterns.py # Default ignore patterns
│ ├── mapping.py # Backwards compatibility re-exports
│ └── cli.py # Command-line interface (4 subcommands)
├── tests/
│ └── test_loppers.py # Unit tests (38 tests)
├── pyproject.toml # Project configuration
├── README.md # This file
└── CLAUDE.md # Development guide for Claude Code
```
## Dependencies
**Runtime:**
- `tree-sitter>=0.25.0` - AST parsing library
- `tree-sitter-language-pack>=0.10.0` - Language grammars
- `binaryornot>=0.4.4` - Binary file detection
- `pathspec>=0.9.0` - .gitignore pattern matching
**Development:**
- `pytest>=7.0.0` - Testing framework
- `pytest-cov>=4.0.0` - Coverage reporting
- `ruff>=0.1.0` - Linting and formatting
## References
- [tree-sitter Documentation](https://tree-sitter.github.io/)
- [tree-sitter-language-pack](https://github.com/grantjenks/py-tree-sitter-language-pack)
- [binaryornot Library](https://github.com/audreyr/binaryornot)
- [pathspec Library](https://github.com/cpburnz/python-pathspec)
- [uv Package Manager](https://github.com/astral-sh/uv)
## License
MIT - See LICENSE file for details
Raw data
{
"_id": null,
"home_page": null,
"name": "loppers",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ast, code-skeleton, extraction, loppers, parsing, tree-sitter",
"author": null,
"author_email": "Your Name <manolo.santos@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/37/d8/7e66f0deba4f3597621375c3463787135e36121dd9d4dc7a4e38b5a9c9c1/loppers-2.3.0.tar.gz",
"platform": null,
"description": "# Loppers\n\n**Extract source file skeletons using tree-sitter queries.**\n\nRemoves function implementations while preserving structure, signatures, and docstrings. Supports 17 programming languages with a clean, fully-typed Python API and comprehensive CLI.\n\n**Requires: tree-sitter >= 0.25**\n\n## Features\n\n- \u2705 **17 Languages** - Python, JS/TS, Java, Kotlin, Go, Rust, C/C++, C#, Ruby, PHP, Swift, Lua, Scala, Groovy, Objective-C\n- \u2705 **Smart Extraction** - Functions, methods, constructors, arrow functions, getters/setters\n- \u2705 **Preserved Elements** - Signatures, class definitions, imports, docstrings, decorators\n- \u2705 **All File Types** - Process any non-binary text files (code, markdown, JSON, YAML, etc.)\n- \u2705 **Binary Detection** - Automatically skips binary files\n- \u2705 **Ignore Patterns** - Built-in + custom .gitignore support\n- \u2705 **Fully Typed** - Complete type hints throughout\n- \u2705 **CLI & Library** - Use as command-line tool or Python library\n\n## Quick Start\n\n### Installation\n\n```bash\n# With uv (recommended)\nuv pip install loppers\n\n# With pip\npip install loppers\n```\n\n## Python API\n\nThe public API consists of 5 core functions:\n\n### 1. `extract_skeleton(source: str, language: str) -> str`\n\nExtract skeleton from source code by language identifier.\n\n```python\nfrom loppers import extract_skeleton\n\ncode = \"\"\"\ndef calculate(x: int, y: int) -> int:\n '''Calculate sum.'''\n result = x + y\n return result\n\"\"\"\n\nskeleton = extract_skeleton(code, \"python\")\nprint(skeleton)\n```\n\nOutput:\n```python\ndef calculate(x: int, y: int) -> int:\n '''Calculate sum.'''\n```\n\n### 2. `get_skeleton(file_path: Path | str, *, add_header: bool = False) -> str`\n\nExtract skeleton from a file by auto-detecting language from extension.\n\n```python\nfrom loppers import get_skeleton\n\nskeleton = get_skeleton(\"src/main.py\")\nprint(skeleton)\n\n# With header showing file path\nskeleton = get_skeleton(\"src/main.py\", add_header=True)\n# Output: \"--- /path/to/src/main.py\\n...\"\n```\n\n**Raises:**\n- `FileNotFoundError` - If file doesn't exist\n- `ValueError` - If file language is unsupported\n\n### 3. `find_files(root: str | Path, *, recursive: bool = True, ignore_patterns: Sequence[str] | None = None, use_default_ignore: bool = True, respect_gitignore: bool = True) -> list[str]`\n\nCollect all non-binary text files from a root directory.\n\n```python\nfrom loppers import find_files\n\n# Find all text files in src/ recursively (default)\nfiles = find_files(\"src/\")\n\n# Returns file paths relative to root:\n# ['main.py', 'utils.py', 'config.yaml', 'README.md']\n\n# Non-recursive\nfiles = find_files(\"src/\", recursive=False)\n\n# Custom ignore patterns (gitignore syntax)\nfiles = find_files(\n \"src/\",\n ignore_patterns=[\"*.test.py\", \"venv/\"],\n use_default_ignore=True, # Still applies built-in patterns\n respect_gitignore=True, # Still respects .gitignore\n)\n```\n\n**Features:**\n- Takes single root directory (not multiple paths)\n- Returns file paths relative to root\n- Automatically excludes binary files (images, archives, etc.)\n- Respects `.gitignore` by default\n- Supports custom gitignore-style ignore patterns\n- Built-in patterns exclude node_modules, .git, __pycache__, build artifacts, etc.\n- Works with ALL non-binary text files (code, markdown, JSON, YAML, etc.)\n\n### 4. `get_tree(root: str | Path, *, recursive: bool = True, ignore_patterns: Sequence[str] | None = None, use_default_ignore: bool = True, respect_gitignore: bool = True, collapse_single_dirs: bool = False, show_sizes: bool = False) -> str`\n\nDisplay formatted directory tree from a root directory.\n\n```python\nfrom loppers import get_tree\n\n# Display tree of src/ directory recursively\ntree = get_tree(\"src/\")\nprint(tree)\n\n# Non-recursive tree\ntree = get_tree(\"src/\", recursive=False)\n\n# With custom ignore patterns\ntree = get_tree(\"src/\", ignore_patterns=[\"*.test.py\"])\n\n# Collapse deep single-child directories (useful for Java packages)\ntree = get_tree(\"src/\", collapse_single_dirs=True)\n\n# Show file sizes in human-friendly format\ntree = get_tree(\"src/\", show_sizes=True)\n\n# Combine multiple options\ntree = get_tree(\"src/\", collapse_single_dirs=True, show_sizes=True)\n```\n\nOutput (with `collapse_single_dirs=True` and `show_sizes=True`):\n```\n.\n\u251c\u2500 main/java/com/example\n\u2502 \u251c\u2500 Source.java (2.3KB)\n\u2502 \u2514\u2500 Util.java (1.8KB)\n\u2514\u2500 resources/config.yaml (512B)\n```\n\n### 5. `concatenate_files(root: str | Path, file_paths: Sequence[str | Path], *, extract: bool = True, ignore_not_found: bool = False) -> str`\n\nConcatenate files with optional skeleton extraction. Useful when you already have a list of file paths that you want to combine.\n\n```python\nfrom loppers import concatenate_files, find_files\n\n# Get list of files to concatenate\nroot = \"src/\"\nfiles = find_files(root)\n\n# Concatenate with skeleton extraction (default)\nresult = concatenate_files(root, files)\nprint(result)\n\n# Concatenate without extraction (include original content)\nresult = concatenate_files(root, files, extract=False)\n\n# Ignore files that don't exist or can't be processed\nresult = concatenate_files(root, files, ignore_not_found=True)\n```\n\nOutput format:\n```\n--- path/to/file.py\ndef calculate(x: int) -> int:\n '''Calculate.'''\n\n--- path/to/other.js\nfunction process() {\n}\n```\n\n**Features:**\n- Files are concatenated with headers showing their relative paths\n- Each file separated by newlines\n- Automatically extracts skeletons from code files (unless `extract=False`)\n- Falls back to original content for unsupported file types\n- Can optionally ignore files that don't exist or can't be processed\n\n**Parameters:**\n- `root` - Root directory (file paths are relative to this)\n- `file_paths` - List of file paths (relative to root) to concatenate\n- `extract` - Extract skeletons from code files (default `True`)\n- `ignore_not_found` - Ignore files that cannot be found or processed (default `False`)\n\n**Raises:**\n- `FileNotFoundError` - If root doesn't exist or a file is not found (when `ignore_not_found=False`)\n- `ValueError` - If no file paths provided or no files could be processed\n- `NotADirectoryError` - If root is not a directory\n\n### Utility Function\n\n**`get_language(extension: str) -> str | None`** - Get language identifier from file extension.\n\n```python\nfrom loppers import get_language\n\nget_language(\".py\") # \"python\"\nget_language(\".js\") # \"javascript\"\nget_language(\".json\") # None (no extraction for data files)\n```\n\n## Command-Line Interface\n\nLoppers provides 4 subcommands for common tasks.\n\n### Basic Usage\n\n```bash\nloppers --version\nloppers --help\n```\n\n### 1. `extract` - Extract skeleton from file or stdin\n\nExtract a single file's skeleton:\n```bash\n# From file\nloppers extract file.py\nloppers extract file.py -o skeleton.py\n\n# From stdin with explicit language\necho 'def foo(): pass' | loppers extract -l python\n\n# Verbose output\nloppers extract file.py -v\n```\n\n**Options:**\n- `FILE` - File to extract (omit for stdin)\n- `-l, --language` - Language identifier (auto-detected from extension if FILE provided, required for stdin)\n- `-o, --output` - Output file (default: stdout)\n- `-v, --verbose` - Print status to stderr\n\n### 2. `concatenate` - Concatenate files with optional skeleton extraction\n\nProcess root directory with automatic skeleton extraction:\n\n```bash\n# Recursive (default)\nloppers concatenate src/\n\n# Non-recursive\nloppers concatenate --no-recursive src/\n\n# Save to file\nloppers concatenate src/ -o combined.txt\n\n# Verbose with progress\nloppers concatenate -v src/\n\n# Include original files without extraction\nloppers concatenate --no-extract src/\n\n# Custom ignore patterns\nloppers concatenate -I \"*.test.py\" -I \"venv/\" src/\n\n# Disable default ignores\nloppers concatenate --no-default-ignore src/\n\n# Don't respect .gitignore\nloppers concatenate --no-gitignore src/\n```\n\n**Features:**\n- Processes a single root directory (paths relative to root)\n- Includes ALL non-binary text files (code, markdown, JSON, YAML, etc.)\n- Automatically extracts skeletons for supported code files\n- Includes original content for unsupported file types (graceful degradation)\n- Each file prefixed with `--- filepath` header (relative path)\n- Verbose mode shows extraction status for each file\n\n**Options:**\n- `root` - Root directory to process (required)\n- `-o, --output` - Output file (default: stdout)\n- `--no-extract` - Include original files without extraction\n- `-I, --ignore-pattern` - Add custom ignore pattern (gitignore syntax, can be used multiple times)\n- `--no-default-ignore` - Disable built-in ignore patterns\n- `--no-gitignore` - Don't respect .gitignore\n- `--no-recursive` - Don't recursively traverse directories\n- `-v, --verbose` - Print status to stderr\n\n### 3. `tree` - Show directory tree of discovered files\n\nDisplay a formatted tree of all discovered files:\n\n```bash\n# Recursive tree (default)\nloppers tree src/\n\n# Non-recursive\nloppers tree --no-recursive src/\n\n# Save tree to file\nloppers tree src/ -o tree.txt\n\n# With ignore patterns\nloppers tree -I \"*.test.py\" src/\n\n# Collapse deep single-child directories (useful for Java packages)\nloppers tree --collapse-single-dirs src/\n\n# Show file sizes in human-friendly format\nloppers tree --show-sizes src/\n\n# Combine multiple options\nloppers tree --collapse-single-dirs --show-sizes src/\n```\n\n**Options:**\n- `root` - Root directory to process (required)\n- `-o, --output` - Output file (default: stdout)\n- `-I, --ignore-pattern` - Add custom ignore pattern\n- `--no-default-ignore` - Disable built-in ignore patterns\n- `--no-gitignore` - Don't respect .gitignore\n- `--no-recursive` - Non-recursive tree\n- `--collapse-single-dirs` - Collapse directories with single children (e.g., `java/com/example` becomes one line)\n- `--show-sizes` - Show file sizes in human-friendly format (e.g., \"1.2KB\", \"5.0MB\")\n- `-v, --verbose` - Print status to stderr\n\n**Collapse Example:**\n\nWithout collapse:\n```\n.\n\u2514\u2500 src\n \u2514\u2500 main\n \u2514\u2500 java\n \u2514\u2500 com\n \u2514\u2500 example\n \u251c\u2500 Source.java\n \u2514\u2500 Util.java\n```\n\nWith `--collapse-single-dirs`:\n```\n.\n\u2514\u2500 src/main/java/com/example\n \u251c\u2500 Source.java\n \u2514\u2500 Util.java\n```\n\n### 4. `files` - List all discovered files\n\nPrint one discovered file per line (relative to root):\n\n```bash\n# List all files recursively (default)\nloppers files src/\n\n# Save list to file\nloppers files src/ -o file_list.txt\n\n# Non-recursive\nloppers files --no-recursive src/\n\n# With custom ignores\nloppers files -I \"*.md\" src/\n```\n\n**Options:**\n- `root` - Root directory to process (required)\n- `-o, --output` - Output file (default: stdout)\n- `-I, --ignore-pattern` - Add custom ignore pattern\n- `--no-default-ignore` - Disable built-in ignore patterns\n- `--no-gitignore` - Don't respect .gitignore\n- `--no-recursive` - Non-recursive listing\n- `-v, --verbose` - Print status to stderr\n\n## Examples: Before and After\n\n### Python Example\n\n**Before:**\n```python\nclass Calculator:\n def __init__(self, name: str):\n \"\"\"Initialize calculator.\"\"\"\n self.name = name\n self._setup()\n\n def process(self, data):\n \"\"\"Process data.\"\"\"\n result = []\n for item in data:\n result.append(item * 2)\n return result\n```\n\n**After:**\n```python\nclass Calculator:\n def __init__(self, name: str):\n \"\"\"Initialize calculator.\"\"\"\n\n def process(self, data):\n \"\"\"Process data.\"\"\"\n```\n\n### JavaScript/TypeScript Example\n\n**Before:**\n```typescript\nclass UserService {\n constructor(baseUrl: string) {\n this.baseUrl = baseUrl;\n this.cache = {};\n }\n\n async getUser(id: string) {\n if (this.cache[id]) return this.cache[id];\n const user = await fetch(this.baseUrl + '/' + id);\n return user.json();\n }\n}\n```\n\n**After:**\n```typescript\nclass UserService {\n constructor(baseUrl: string) {\n }\n\n async getUser(id: string) {\n }\n}\n```\n\n### Java Example\n\n**Before:**\n```java\npublic class UserService {\n private String baseUrl;\n\n public UserService(String baseUrl) {\n this.baseUrl = baseUrl;\n this.validate();\n }\n\n public User getUserById(String id) {\n Database db = new Database();\n return db.query(id);\n }\n\n private void validate() {\n if (baseUrl == null) {\n throw new IllegalArgumentException(\"BaseUrl required\");\n }\n }\n}\n```\n\n**After:**\n```java\npublic class UserService {\n private String baseUrl;\n\n public UserService(String baseUrl) {\n }\n\n public User getUserById(String id) {\n }\n\n private void validate() {\n }\n}\n```\n\n## Supported Languages\n\n| Language | Features |\n|----------|----------|\n| **Python** | Functions, methods, `__init__`, `@property`, docstrings |\n| **JavaScript/TypeScript** | Functions, arrow functions, methods, async/await |\n| **Java** | Methods, constructors, static methods, annotations |\n| **Kotlin** | Functions, methods, properties (getters/setters) |\n| **Go** | Functions, methods, closures |\n| **Rust** | Functions, methods, closures |\n| **C/C++** | Functions, methods, constructors |\n| **C#** | Methods, properties (get/set), async/await |\n| **Ruby** | Methods, singleton methods, blocks |\n| **PHP** | Functions, methods, closures |\n| **Swift** | Functions, methods, closures |\n| **Lua** | Functions, local functions |\n| **Scala** | Functions, methods, closures |\n| **Groovy** | Functions, methods, closures |\n| **Objective-C** | Methods, instance/class methods |\n\n### What Gets Preserved\n\n- \u2705 Function/method signatures\n- \u2705 Parameter types and defaults\n- \u2705 Return types\n- \u2705 Class definitions\n- \u2705 Import statements\n- \u2705 Comments\n- \u2705 Python docstrings\n- \u2705 Decorators\n- \u2705 Access modifiers (public, private, protected)\n\n### What Gets Removed\n\n- \u274c Function/method bodies\n- \u274c Local variable assignments\n- \u274c Logic and implementation details\n- \u274c Nested function implementations\n\n### Known Limitations\n\n- Concise arrow functions (`const f = x => x * 2`) - no body to remove\n- Python lambdas - no body to remove\n- Some edge cases with getters/setters in JavaScript/TypeScript\n\n## How It Works\n\nLoppers uses **tree-sitter** queries to parse source code into Abstract Syntax Trees (AST) and intelligently remove function/method bodies while preserving:\n\n- Function/method signatures\n- Class and interface definitions\n- Import statements\n- Python docstrings\n- Comments\n- Decorators\n- Type hints\n\nEach language has custom tree-sitter query patterns that capture function/method body nodes, which are then removed line-by-line.\n\n## Development\n\n### Setup\n\n```bash\n# Install with dev dependencies\nuv sync\n```\n\n### Running Tests\n\n```bash\n# All tests\nuv run pytest\n\n# Verbose output\nuv run pytest -v\n\n# With coverage\nuv run pytest --cov=loppers --cov-report=html\n\n# Specific test\nuv run pytest tests/test_loppers.py::test_python_extraction\n```\n\n### Code Quality\n\n```bash\n# Check and fix\nuv run ruff check . --fix\n\n# Format\nuv run ruff format .\n\n# All checks\nuv run ruff check . --fix && uv run ruff format .\n```\n\n### Adding New Languages\n\nTo add support for a new language:\n\n1. **Find the tree-sitter query** - Use the [tree-sitter playground](https://tree-sitter.github.io/tree-sitter/playground) to develop a query that captures function bodies\n\n2. **Add to LANGUAGE_CONFIGS** in `src/loppers/loppers.py`:\n ```python\n LANGUAGE_CONFIGS[\"mylang\"] = LanguageConfig(\n name=\"mylang\",\n body_query=\"(function_definition body: (block) @body)\",\n )\n ```\n\n3. **Add file extensions** to `src/loppers/extensions.py`:\n ```python\n EXTENSION_TO_LANGUAGE = {\n \".ml\": \"mylang\",\n \".mli\": \"mylang\",\n }\n ```\n\n4. **Write a test** in `tests/test_loppers.py`:\n ```python\n def test_mylang_extraction(self):\n code = \"fun hello() { print('hi') }\"\n skeleton = extract_skeleton(code, \"mylang\")\n assert \"fun hello()\" in skeleton\n assert \"print\" not in skeleton\n ```\n\n5. **Run tests** to verify everything works\n\n## Project Structure\n\n```\nloppers/\n\u251c\u2500\u2500 src/loppers/\n\u2502 \u251c\u2500\u2500 __init__.py # Public API: extract_skeleton, get_skeleton, find_files, get_tree, concatenate_files\n\u2502 \u251c\u2500\u2500 loppers.py # Core extraction logic with SkeletonExtractor class\n\u2502 \u251c\u2500\u2500 source_utils.py # Convenience API and file operations\n\u2502 \u251c\u2500\u2500 extensions.py # Language extension mapping\n\u2502 \u251c\u2500\u2500 ignore_patterns.py # Default ignore patterns\n\u2502 \u251c\u2500\u2500 mapping.py # Backwards compatibility re-exports\n\u2502 \u2514\u2500\u2500 cli.py # Command-line interface (4 subcommands)\n\u251c\u2500\u2500 tests/\n\u2502 \u2514\u2500\u2500 test_loppers.py # Unit tests (38 tests)\n\u251c\u2500\u2500 pyproject.toml # Project configuration\n\u251c\u2500\u2500 README.md # This file\n\u2514\u2500\u2500 CLAUDE.md # Development guide for Claude Code\n```\n\n## Dependencies\n\n**Runtime:**\n- `tree-sitter>=0.25.0` - AST parsing library\n- `tree-sitter-language-pack>=0.10.0` - Language grammars\n- `binaryornot>=0.4.4` - Binary file detection\n- `pathspec>=0.9.0` - .gitignore pattern matching\n\n**Development:**\n- `pytest>=7.0.0` - Testing framework\n- `pytest-cov>=4.0.0` - Coverage reporting\n- `ruff>=0.1.0` - Linting and formatting\n\n## References\n\n- [tree-sitter Documentation](https://tree-sitter.github.io/)\n- [tree-sitter-language-pack](https://github.com/grantjenks/py-tree-sitter-language-pack)\n- [binaryornot Library](https://github.com/audreyr/binaryornot)\n- [pathspec Library](https://github.com/cpburnz/python-pathspec)\n- [uv Package Manager](https://github.com/astral-sh/uv)\n\n## License\n\nMIT - See LICENSE file for details\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Extract source file skeletons using tree-sitter queries",
"version": "2.3.0",
"project_urls": {
"Documentation": "https://github.com/undo76/loppers#readme",
"Homepage": "https://github.com/undo76/loppers",
"Issues": "https://github.com/undo76/loppers/issues",
"Repository": "https://github.com/undo76/loppers"
},
"split_keywords": [
"ast",
" code-skeleton",
" extraction",
" loppers",
" parsing",
" tree-sitter"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "8b1998c17681c3ecd6b2969b2cc98e33b3ef61bf0de27c686e35f7883f45007d",
"md5": "5d12e33e85e27725619e1d4a5e3f150e",
"sha256": "cc166931db6d368bbf9c87033821cfc5b424aa317b8fb899e1bf79120857bd78"
},
"downloads": -1,
"filename": "loppers-2.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5d12e33e85e27725619e1d4a5e3f150e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 19524,
"upload_time": "2025-10-20T13:37:20",
"upload_time_iso_8601": "2025-10-20T13:37:20.112128Z",
"url": "https://files.pythonhosted.org/packages/8b/19/98c17681c3ecd6b2969b2cc98e33b3ef61bf0de27c686e35f7883f45007d/loppers-2.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "37d87e66f0deba4f3597621375c3463787135e36121dd9d4dc7a4e38b5a9c9c1",
"md5": "cec46d73ee43c8ec936386d87e82e385",
"sha256": "c461b8bb2c56eaf78a170f0c97c6011851dd321b19c5c82e63b4a92ce7e906d0"
},
"downloads": -1,
"filename": "loppers-2.3.0.tar.gz",
"has_sig": false,
"md5_digest": "cec46d73ee43c8ec936386d87e82e385",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 104944,
"upload_time": "2025-10-20T13:37:21",
"upload_time_iso_8601": "2025-10-20T13:37:21.416199Z",
"url": "https://files.pythonhosted.org/packages/37/d8/7e66f0deba4f3597621375c3463787135e36121dd9d4dc7a4e38b5a9c9c1/loppers-2.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-20 13:37:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "undo76",
"github_project": "loppers#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "loppers"
}
JSON
