| Name | lmprep JSON |
| Version |
0.4.1
JSON |
| download |
| home_page | None |
| Summary | A tool for preparing your codebase for use with LLMs |
| upload_time | 2024-12-13 16:50:13 |
| maintainer | None |
| docs_url | None |
| author | bcherb2 |
| requires_python | >=3.8 |
| license | MIT |
| keywords |
llm
code
preparation
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# LMPrep
[](https://opensource.org/licenses/MIT)
[](https://badge.fury.io/py/lmprep)
[](https://pypi.org/project/lmprep/)
[](https://www.rust-lang.org/)
[](https://github.com/bcherb2/lmprep)
A lightning-fast utility for preparing and organizing your code for use with LLMs like Claude Projects. LMPrep will collect and rename all of your project files to a flat directory, but preserving the structure within the filenames.
For example, a file at `src/models/user.py` will be renamed to `src^models^user.py` in the output directory. Be sure to tell the LLM that your files are structured this way!
https://github.com/user-attachments/assets/27d49b03-76a0-4742-9883-e361b73bc10e
## Features
- **Smart File Organization**: Automatically flattens complex directory structures while preserving path information in the filenames and in a file tree
- **Configurable Filtering**: Specify which file extensions to include in your dataset to limit context size
- **Path Preservation**: Uses customizable delimiters to maintain original path information in filenames
- **Git-Aware**: Respects `.gitignore` patterns to exclude unwanted files or secrets
- **Flexible Output**: Generate individual files or create a zip archive
- **Visual Tree View**: Visualize your source and output file structure, or send the file tree to the LLM
- **Fast & Efficient**: Written in Rust for maximum performance
## Quick Start
### Installation
The easiest way to install LMPrep is to get it from PyPi:
```bash
pip install lmprep
```
wheels are built for Windows, Linux, and MacOS.
#### Manaul / Install Script
1. Download the latest release for your platform from [Releases](https://github.com/bcherb2/lmprep/releases):
- Windows: `lm-x86_64-pc-windows-msvc.zip`
- Linux: `lm-x86_64-unknown-linux-gnu.tar.gz`
- macOS: `lm-x86_64-apple-darwin.tar.gz`
2. Install the binary:
**Linux/macOS**:
```bash
# Extract and copy binary
tar xzf lm-x86_64-*-*.tar.gz
sudo mv lm /usr/local/bin/
# Create config file
curl -O https://raw.githubusercontent.com/bcherb2/lmprep/main/src/config-example.yaml
mv config-example.yaml ~/.lmprep.yml
```
**Windows** (in PowerShell, run as Administrator):
```powershell
# Extract and copy binary
Expand-Archive lm-x86_64-pc-windows-msvc.zip
New-Item -ItemType Directory -Force -Path "C:\Program Files\lmprep"
Move-Item -Force lm.exe "C:\Program Files\lmprep\lm.exe"
$env:Path += ";C:\Program Files\lmprep"
# Create config file
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/bcherb2/lmprep/main/src/config-example.yaml" -OutFile "$env:USERPROFILE\.lmprep.yml"
```
3. Verify installation:
```bash
lm --help
```
#### Alternative: Build from Source
If you have Rust installed, you can build from source:
```bash
git clone https://github.com/bcherb2/lmprep
cd lmprep
cargo build --release
```
The binary will be in `target/release/lm` (or `lm.exe` on Windows). Follow step 2 above to set up the config file.
### Basic Usage
```bash
# Create a default config file in the current directory
lm --init-config
# Organize files in current directory
lm .
# Organize files from a specific directory
lm /path/to/source
# Use a custom config file
lm . -c /path/to/.lmprep.yml
# Create a zip archive instead of of individual files
lm . --zip
```
## Configuration
Create a `.lmprep.yml` file in your home directory to customize behavior, or create one in your project root directory. Here's an example:
```yaml
allowed_extensions:
- py
- rs
- md
- txt
ignored_directories:
- node_modules
delimiter: "^"
subfolder: context
zip: false
tree: true
respect_gitignore: true
```
>NOTE: The install script will create a default config file at `~/.lmprep.yml`
### Configuration Options
| Option | Description | Default |
|--------|-------------|---------|
| `allowed_extensions` | File extensions to include | `[]` (common extensions) |
| `ignored_directories` | Directories to ignore | `[]` (common directories) |
| `delimiter` | Character used to represent path hierarchy | `^` |
| `subfolder` | Output directory name within project | `context` |
| `zip` | Create zip archive instead of files | `false` |
| `tree` | Show file tree visualization | `true` |
| `respect_gitignore` | Honor .gitignore patterns | `true` |
## Command Line Options
```bash
lm [OPTIONS] [SOURCE]
Arguments:
[SOURCE] Source directory to organize files from [default: .]
Options:
-c, --config <FILE> Path to config file
-s, --subfolder <NAME> Override the subfolder name from config
-z, --zip Create a zip file instead of individual files
-t, --tree Show file tree of source and output
-v, --verbose Show more detailed output during processing
--init-config Create a default config file in the current directory
-h, --help Print help
-V, --version Print version
```
## Development
To set up for development:
1. Clone the repository
2. Run `./dev-setup.sh`
This will build the Rust binary, set up the correct directory structure, and install the package in development mode.
When you make changes to the Rust code, run `./dev-setup.sh` again to rebuild and reinstall.
Python changes will be picked up automatically due to the development install.
## Use Cases
- **Code Analysis**: Organize your code into a flat structure while preserving context (works especially well with Claude Projects)
- **Document Processing**: Organize and prepare document collections for processing, logs, etc.
- **Version Control**: Easily create clean snapshots of your codebase for archival in zip format
## Building from Source
1. Install Rust using [rustup](https://rustup.rs/)
2. Clone the repository:
```bash
git clone https://github.com/bcherb2/lmprep.git
cd lmprep
```
3. Build the project:
```bash
cargo build --release
```
4. The binary will be available at `target/release/lm`, copy it and add it to your PATH
5. Create the `.lmprep.yml` file in your home directory or project root
>
>NOTE: see [install/BUILD.md](https://github.com/bcherb2/lmprep/blob/main/install/BUILD.md) for more in depth building instructions.
>
## FAQ
**Q: Why use LMPrep instead of just copying files?**
A: LMPrep preserves directory structure information in filenames, making it easier for LLMs to understand file relationships and context. Sure, you can do this manually, but it gets tedious.
**Q: How does path flattening work?**
A: A file at `src/models/user.py` becomes `src^models^user.py` in the output directory (using default delimiter). Changing the delimiter to `+` would result in `src+models+user.py`.
**Q: Can I exclude certain files or directories?**
A: Yes! LMPrep respects `.gitignore` patterns and allows you to specify allowed file extensions.
**Q: Is it safe to use on large directories?**
A: Yes! LMPrep is written in Rust for performance and memory efficiency, making it suitable for large datasets.
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "lmprep",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "llm, code, preparation",
"author": "bcherb2",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# LMPrep \n\n[](https://opensource.org/licenses/MIT)\n[](https://badge.fury.io/py/lmprep)\n[](https://pypi.org/project/lmprep/)\n[](https://www.rust-lang.org/)\n[](https://github.com/bcherb2/lmprep)\n\nA lightning-fast utility for preparing and organizing your code for use with LLMs like Claude Projects. LMPrep will collect and rename all of your project files to a flat directory, but preserving the structure within the filenames. \n\nFor example, a file at `src/models/user.py` will be renamed to `src^models^user.py` in the output directory. Be sure to tell the LLM that your files are structured this way!\n\n\nhttps://github.com/user-attachments/assets/27d49b03-76a0-4742-9883-e361b73bc10e\n\n\n## Features\n\n- **Smart File Organization**: Automatically flattens complex directory structures while preserving path information in the filenames and in a file tree\n- **Configurable Filtering**: Specify which file extensions to include in your dataset to limit context size\n- **Path Preservation**: Uses customizable delimiters to maintain original path information in filenames\n- **Git-Aware**: Respects `.gitignore` patterns to exclude unwanted files or secrets\n- **Flexible Output**: Generate individual files or create a zip archive\n- **Visual Tree View**: Visualize your source and output file structure, or send the file tree to the LLM\n- **Fast & Efficient**: Written in Rust for maximum performance\n\n## Quick Start\n\n### Installation\n\nThe easiest way to install LMPrep is to get it from PyPi:\n\n```bash\npip install lmprep\n```\nwheels are built for Windows, Linux, and MacOS.\n\n#### Manaul / Install Script\n\n1. Download the latest release for your platform from [Releases](https://github.com/bcherb2/lmprep/releases):\n - Windows: `lm-x86_64-pc-windows-msvc.zip`\n - Linux: `lm-x86_64-unknown-linux-gnu.tar.gz`\n - macOS: `lm-x86_64-apple-darwin.tar.gz`\n\n2. Install the binary:\n\n**Linux/macOS**:\n```bash\n# Extract and copy binary\ntar xzf lm-x86_64-*-*.tar.gz\nsudo mv lm /usr/local/bin/\n\n# Create config file\ncurl -O https://raw.githubusercontent.com/bcherb2/lmprep/main/src/config-example.yaml\nmv config-example.yaml ~/.lmprep.yml\n```\n\n**Windows** (in PowerShell, run as Administrator):\n```powershell\n# Extract and copy binary\nExpand-Archive lm-x86_64-pc-windows-msvc.zip\nNew-Item -ItemType Directory -Force -Path \"C:\\Program Files\\lmprep\"\nMove-Item -Force lm.exe \"C:\\Program Files\\lmprep\\lm.exe\"\n$env:Path += \";C:\\Program Files\\lmprep\"\n\n# Create config file\nInvoke-WebRequest -Uri \"https://raw.githubusercontent.com/bcherb2/lmprep/main/src/config-example.yaml\" -OutFile \"$env:USERPROFILE\\.lmprep.yml\"\n```\n\n3. Verify installation:\n```bash\nlm --help\n```\n\n#### Alternative: Build from Source\n\nIf you have Rust installed, you can build from source:\n```bash\ngit clone https://github.com/bcherb2/lmprep\ncd lmprep\ncargo build --release\n```\nThe binary will be in `target/release/lm` (or `lm.exe` on Windows). Follow step 2 above to set up the config file.\n\n### Basic Usage\n\n```bash\n# Create a default config file in the current directory\nlm --init-config\n\n# Organize files in current directory\nlm .\n\n# Organize files from a specific directory\nlm /path/to/source\n\n# Use a custom config file\nlm . -c /path/to/.lmprep.yml\n\n# Create a zip archive instead of of individual files\nlm . --zip\n```\n\n## Configuration\n\nCreate a `.lmprep.yml` file in your home directory to customize behavior, or create one in your project root directory. Here's an example:\n\n```yaml\nallowed_extensions:\n - py\n - rs\n - md\n - txt\nignored_directories:\n - node_modules\ndelimiter: \"^\"\nsubfolder: context\nzip: false\ntree: true\nrespect_gitignore: true\n```\n\n>NOTE: The install script will create a default config file at `~/.lmprep.yml`\n\n### Configuration Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `allowed_extensions` | File extensions to include | `[]` (common extensions) |\n| `ignored_directories` | Directories to ignore | `[]` (common directories) |\n| `delimiter` | Character used to represent path hierarchy | `^` |\n| `subfolder` | Output directory name within project | `context` |\n| `zip` | Create zip archive instead of files | `false` |\n| `tree` | Show file tree visualization | `true` |\n| `respect_gitignore` | Honor .gitignore patterns | `true` |\n\n## Command Line Options\n\n```bash\nlm [OPTIONS] [SOURCE]\n\nArguments:\n [SOURCE] Source directory to organize files from [default: .]\n\nOptions:\n -c, --config <FILE> Path to config file\n -s, --subfolder <NAME> Override the subfolder name from config\n -z, --zip Create a zip file instead of individual files\n -t, --tree Show file tree of source and output\n -v, --verbose Show more detailed output during processing\n --init-config Create a default config file in the current directory\n -h, --help Print help\n -V, --version Print version\n```\n\n## Development\n\nTo set up for development:\n\n1. Clone the repository\n2. Run `./dev-setup.sh`\n\nThis will build the Rust binary, set up the correct directory structure, and install the package in development mode.\n\nWhen you make changes to the Rust code, run `./dev-setup.sh` again to rebuild and reinstall.\nPython changes will be picked up automatically due to the development install.\n\n## Use Cases\n\n- **Code Analysis**: Organize your code into a flat structure while preserving context (works especially well with Claude Projects)\n- **Document Processing**: Organize and prepare document collections for processing, logs, etc.\n- **Version Control**: Easily create clean snapshots of your codebase for archival in zip format\n\n## Building from Source\n\n1. Install Rust using [rustup](https://rustup.rs/)\n2. Clone the repository:\n ```bash\n git clone https://github.com/bcherb2/lmprep.git\n cd lmprep\n ```\n3. Build the project:\n ```bash\n cargo build --release\n ```\n4. The binary will be available at `target/release/lm`, copy it and add it to your PATH\n5. Create the `.lmprep.yml` file in your home directory or project root\n\n>\n>NOTE: see [install/BUILD.md](https://github.com/bcherb2/lmprep/blob/main/install/BUILD.md) for more in depth building instructions.\n>\n\n## FAQ\n\n**Q: Why use LMPrep instead of just copying files?**\nA: LMPrep preserves directory structure information in filenames, making it easier for LLMs to understand file relationships and context. Sure, you can do this manually, but it gets tedious.\n\n**Q: How does path flattening work?**\nA: A file at `src/models/user.py` becomes `src^models^user.py` in the output directory (using default delimiter). Changing the delimiter to `+` would result in `src+models+user.py`.\n\n**Q: Can I exclude certain files or directories?**\nA: Yes! LMPrep respects `.gitignore` patterns and allows you to specify allowed file extensions.\n\n**Q: Is it safe to use on large directories?**\nA: Yes! LMPrep is written in Rust for performance and memory efficiency, making it suitable for large datasets.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool for preparing your codebase for use with LLMs",
"version": "0.4.1",
"project_urls": {
"Homepage": "https://github.com/bcherb2/lmprep",
"Repository": "https://github.com/bcherb2/lmprep.git"
},
"split_keywords": [
"llm",
" code",
" preparation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3af1afc4226ade1f3c08d520f5208861d47586788c7c95e24a9b62dc8fef4235",
"md5": "a48f070be99a237289fb8ffb66984cd7",
"sha256": "0466b327d3af58e3a5c8c859a588d92a9527483047f9641d25353b7edb893592"
},
"downloads": -1,
"filename": "lmprep-0.4.1-py3-none-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "a48f070be99a237289fb8ffb66984cd7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 5466338,
"upload_time": "2024-12-13T16:50:13",
"upload_time_iso_8601": "2024-12-13T16:50:13.370708Z",
"url": "https://files.pythonhosted.org/packages/3a/f1/afc4226ade1f3c08d520f5208861d47586788c7c95e24a9b62dc8fef4235/lmprep-0.4.1-py3-none-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c8074e50749ac0f0bf64dace19b272db593fe900e4aabc9f23cbd5c300aa7c5f",
"md5": "47d63cce556f206260638f181926cf15",
"sha256": "85ea9bc41a30c305c56b30e7e391ebcb361f6fb35b419d6f4025947353f0d09f"
},
"downloads": -1,
"filename": "lmprep-0.4.1-py3-none-manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "47d63cce556f206260638f181926cf15",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 5466337,
"upload_time": "2024-12-13T16:50:16",
"upload_time_iso_8601": "2024-12-13T16:50:16.522586Z",
"url": "https://files.pythonhosted.org/packages/c8/07/4e50749ac0f0bf64dace19b272db593fe900e4aabc9f23cbd5c300aa7c5f/lmprep-0.4.1-py3-none-manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "85330d15fa35dfa1e8f7c6fbed69fb0f9f6e613e74031603d16790e0f346e88d",
"md5": "8114a9351aa2b891388f950ff4debbe4",
"sha256": "ed33bd89df2cdcfe3a54097b8524d72d606215b3971a06b9b36c3f133f65692f"
},
"downloads": -1,
"filename": "lmprep-0.4.1-py3-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "8114a9351aa2b891388f950ff4debbe4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 5466327,
"upload_time": "2024-12-13T16:50:19",
"upload_time_iso_8601": "2024-12-13T16:50:19.826182Z",
"url": "https://files.pythonhosted.org/packages/85/33/0d15fa35dfa1e8f7c6fbed69fb0f9f6e613e74031603d16790e0f346e88d/lmprep-0.4.1-py3-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 16:50:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bcherb2",
"github_project": "lmprep",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "lmprep"
}
