| Name | slimcontext JSON |
| Version |
0.1.2
JSON |
| download |
| home_page | None |
| Summary | A tool to turn a git repository into context for LLMs. |
| upload_time | 2024-12-09 05:30:51 |
| maintainer | None |
| docs_url | None |
| author | Neil Schneider |
| requires_python | >=3.10 |
| license | MIT |
| keywords |
llm
git
context
tool
chatbot
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# slimcontext
**slimcontext** is a Python package that transforms a Git repository into contextual information suitable for Large Language Models (LLMs). It extracts comprehensive information from Python source files, generates structured context for LLMs, and provides utilities for token counting and logging.
## Features
- **Context Extraction:** Extracts either full or succinct textual contexts based on the repository content, tailored for effective LLM interactions.
- **Token Counting:** Integrates with `tiktoken` to count tokens, ensuring compatibility with various LLM models.
- **Command-Line Interface (CLI):** User-friendly CLI built with `Click` for easy interaction and automation.
- **Git Integration:** Manages and interacts with Git repositories, handling file extraction and repository structure efficiently.
## Installation
You can install the package via PyPI:
```bash
pip install slimcontext
```
Or install directly from GitLab:
```bash
pip install git+https://gitlab.com/notorious-community/slimcontext.git
```
## Usage
### Command-Line Interface (CLI)
The package provides a CLI tool for easy interaction. Below are the available commands and options.
#### Basic Usage
```bash
slimcontext main --path /path/to/your/git/repo --context-level succinct
```
#### Options
- `--path`: Path to the Git repository (required).
- `--context-level`: Level of context generation (`full` or `succinct`). Defaults to `succinct`.
- `--output`: Path to the output file. If not specified, outputs to stdout.
- `--token-count`: Specify the model for token counting (e.g., `gpt-4`). Set to `None` to skip token counting.
- `--log-file`: Path to the log file. If not specified, logs are only output to the console.
- `--log-level`: Logging level (e.g., `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`). Defaults to `INFO`.
#### Example
Generate a full context and save the output to a file while logging to `app.log`:
```bash
slimcontext main \
--path /path/to/your/git/repo \
--context-level full \
--output /path/to/output/context.txt \
--token-count gpt-4 \
--log-file /path/to/logs/app.log \
--log-level DEBUG
```
### Python API
```python
from slimcontext import (
PyParser,
GitRepository,
TokenCounter,
generate_context,
count_tokens,
write_output,
setup_logger
)
from pathlib import Path
# Initialize logger
logger = setup_logger(__name__)
# Initialize Git repository
repo_path = Path('/path/to/your/git/repo')
git_repo = GitRepository(repo_dir=repo_path)
# Initialize Python parser
py_parser = PyParser(root_dir=repo_path)
# Extract repository information
python_files = git_repo.get_files_by_suffix(['.py'])
repo_info = py_parser.extract_repo_info(context_level='succinct')
# Generate context
context = generate_context(py_parser, python_files, context_level='succinct')
# Count tokens
token_counter = TokenCounter(model='gpt-4')
token_count = count_tokens(context, model='gpt-4')
print(f"Total tokens: {token_count}")
# Write output
output_path = Path('/path/to/output/context.txt')
write_output(context, output_path)
```
## Development
### Setting Up the Development Environment
1. **Clone the Repository:**
```bash
git clone https://gitlab.com/notorious-community/slimcontext.git
cd slimcontext
```
2. **Install Dependencies:**
Ensure you have `poetry` installed. Then run:
```bash
poetry install
```
### Running Tests
The package includes a comprehensive test suite using `pytest`.
```bash
poetry run pytest
```
### Linting and Formatting
Ensure code quality and consistency with `ruff`.
```bash
poetry run ruff check .
```
### Running Nox Sessions
Automate development tasks across multiple Python environments using `nox`.
```bash
nox -s tests
```
## Project Structure
```
slimcontext/
├── slimcontext/
│ ├── __init__.py
│ ├── main.py
│ ├── managers/ # Houses various managers for languages.
│ │ ├── __init__.py
│ │ └── python.py
│ ├── parsers/ # Houses the code to parse and extract key context by language.
│ │ ├── __init__.py
│ │ └── pyparse/
│ └── utils/ # Various utilities used across code base.
├── tests/
├── noxfile.py
└── README.md
```
## Contributing
Contributions are welcome! Whether it's reporting bugs, suggesting features, or submitting pull requests, your help is appreciated. Please follow these steps to contribute:
1. **Fork the Repository:**
Click the "Fork" button at the top right of the repository page.
2. **Clone Your Fork:**
```bash
git clone https://gitlab.com/your-username/slimcontext.git
cd slimcontext
```
3. **Create a New Branch:**
```bash
git checkout -b feature/your-feature-name
```
4. **Make Your Changes:**
Implement your feature or bug fix.
5. **Commit Your Changes:**
```bash
git commit -m "Title of your changes" -m "Describe **Why** you made this change."
```
6. **Push to Your Fork:**
```bash
git push origin feature/your-feature-name
```
7. **Create a Merge Request:**
Go to the original repository and create a merge request from your fork.
Please ensure all tests pass and adhere to the project's coding standards before submitting your merge request.
## License
This project is licensed under the [MIT License](LICENSE). See the [LICENSE](LICENSE) file for details.
---
**Copyright (c) 2024 Neil Schneider**
Raw data
{
"_id": null,
"home_page": null,
"name": "slimcontext",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "LLM, Git, Context, Tool, ChatBot",
"author": "Neil Schneider",
"author_email": "schneider.neil@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/5c/8e/8f951ffcd51a8ec5d4df2fccf2a2034549197a9362f9faf14da0cd812fd5/slimcontext-0.1.2.tar.gz",
"platform": null,
"description": "# slimcontext\n\n**slimcontext** is a Python package that transforms a Git repository into contextual information suitable for Large Language Models (LLMs). It extracts comprehensive information from Python source files, generates structured context for LLMs, and provides utilities for token counting and logging.\n\n## Features\n\n- **Context Extraction:** Extracts either full or succinct textual contexts based on the repository content, tailored for effective LLM interactions.\n- **Token Counting:** Integrates with `tiktoken` to count tokens, ensuring compatibility with various LLM models.\n- **Command-Line Interface (CLI):** User-friendly CLI built with `Click` for easy interaction and automation.\n- **Git Integration:** Manages and interacts with Git repositories, handling file extraction and repository structure efficiently.\n\n## Installation\n\nYou can install the package via PyPI:\n\n```bash\npip install slimcontext\n```\n\nOr install directly from GitLab:\n\n```bash\npip install git+https://gitlab.com/notorious-community/slimcontext.git\n```\n\n## Usage\n\n### Command-Line Interface (CLI)\n\nThe package provides a CLI tool for easy interaction. Below are the available commands and options.\n\n#### Basic Usage\n\n```bash\nslimcontext main --path /path/to/your/git/repo --context-level succinct\n```\n\n#### Options\n\n- `--path`: Path to the Git repository (required).\n- `--context-level`: Level of context generation (`full` or `succinct`). Defaults to `succinct`.\n- `--output`: Path to the output file. If not specified, outputs to stdout.\n- `--token-count`: Specify the model for token counting (e.g., `gpt-4`). Set to `None` to skip token counting.\n- `--log-file`: Path to the log file. If not specified, logs are only output to the console.\n- `--log-level`: Logging level (e.g., `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`). Defaults to `INFO`.\n\n#### Example\n\nGenerate a full context and save the output to a file while logging to `app.log`:\n\n```bash\nslimcontext main \\\n --path /path/to/your/git/repo \\\n --context-level full \\\n --output /path/to/output/context.txt \\\n --token-count gpt-4 \\\n --log-file /path/to/logs/app.log \\\n --log-level DEBUG\n```\n\n### Python API\n\n```python\nfrom slimcontext import (\n PyParser,\n GitRepository,\n TokenCounter,\n generate_context,\n count_tokens,\n write_output,\n setup_logger\n)\nfrom pathlib import Path\n\n# Initialize logger\nlogger = setup_logger(__name__)\n\n# Initialize Git repository\nrepo_path = Path('/path/to/your/git/repo')\ngit_repo = GitRepository(repo_dir=repo_path)\n\n# Initialize Python parser\npy_parser = PyParser(root_dir=repo_path)\n\n# Extract repository information\npython_files = git_repo.get_files_by_suffix(['.py'])\nrepo_info = py_parser.extract_repo_info(context_level='succinct')\n\n# Generate context\ncontext = generate_context(py_parser, python_files, context_level='succinct')\n\n# Count tokens\ntoken_counter = TokenCounter(model='gpt-4')\ntoken_count = count_tokens(context, model='gpt-4')\nprint(f\"Total tokens: {token_count}\")\n\n# Write output\noutput_path = Path('/path/to/output/context.txt')\nwrite_output(context, output_path)\n```\n\n## Development\n\n### Setting Up the Development Environment\n\n1. **Clone the Repository:**\n\n ```bash\n git clone https://gitlab.com/notorious-community/slimcontext.git\n cd slimcontext\n ```\n\n2. **Install Dependencies:**\n\n Ensure you have `poetry` installed. Then run:\n\n ```bash\n poetry install\n ```\n\n### Running Tests\n\nThe package includes a comprehensive test suite using `pytest`.\n\n```bash\npoetry run pytest\n```\n\n### Linting and Formatting\n\nEnsure code quality and consistency with `ruff`.\n\n```bash\npoetry run ruff check .\n```\n\n### Running Nox Sessions\n\nAutomate development tasks across multiple Python environments using `nox`.\n\n```bash\nnox -s tests\n```\n\n## Project Structure\n\n```\nslimcontext/\n\u251c\u2500\u2500 slimcontext/\n\u2502 \u251c\u2500\u2500 __init__.py\n\u2502 \u251c\u2500\u2500 main.py\n\u2502 \u251c\u2500\u2500 managers/ # Houses various managers for languages.\n\u2502 \u2502 \u251c\u2500\u2500 __init__.py\n\u2502 \u2502 \u2514\u2500\u2500 python.py\n\u2502 \u251c\u2500\u2500 parsers/ # Houses the code to parse and extract key context by language.\n\u2502 \u2502 \u251c\u2500\u2500 __init__.py\n\u2502 \u2502 \u2514\u2500\u2500 pyparse/\n\u2502 \u2514\u2500\u2500 utils/ # Various utilities used across code base.\n\u251c\u2500\u2500 tests/ \n\u251c\u2500\u2500 noxfile.py\n\u2514\u2500\u2500 README.md\n```\n\n## Contributing\n\nContributions are welcome! Whether it's reporting bugs, suggesting features, or submitting pull requests, your help is appreciated. Please follow these steps to contribute:\n\n1. **Fork the Repository:**\n\n Click the \"Fork\" button at the top right of the repository page.\n\n2. **Clone Your Fork:**\n\n ```bash\n git clone https://gitlab.com/your-username/slimcontext.git\n cd slimcontext\n ```\n\n3. **Create a New Branch:**\n\n ```bash\n git checkout -b feature/your-feature-name\n ```\n\n4. **Make Your Changes:**\n\n Implement your feature or bug fix.\n\n5. **Commit Your Changes:**\n\n ```bash\n git commit -m \"Title of your changes\" -m \"Describe **Why** you made this change.\"\n ```\n\n6. **Push to Your Fork:**\n\n ```bash\n git push origin feature/your-feature-name\n ```\n\n7. **Create a Merge Request:**\n\n Go to the original repository and create a merge request from your fork.\n\nPlease ensure all tests pass and adhere to the project's coding standards before submitting your merge request.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE). See the [LICENSE](LICENSE) file for details.\n\n---\n\n**Copyright (c) 2024 Neil Schneider**\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool to turn a git repository into context for LLMs.",
"version": "0.1.2",
"project_urls": null,
"split_keywords": [
"llm",
" git",
" context",
" tool",
" chatbot"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e42ad4fc178709bd3cdb594202d054ebcff16798cc4db92ccef502aa558f0416",
"md5": "cc18cd82d21e08bf3e0cf6746a7fc7df",
"sha256": "c62ea128a0a7fd85195555b63d473ff7ccba4c316a50a0388560ecec8ed844d0"
},
"downloads": -1,
"filename": "slimcontext-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cc18cd82d21e08bf3e0cf6746a7fc7df",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 22059,
"upload_time": "2024-12-09T05:30:49",
"upload_time_iso_8601": "2024-12-09T05:30:49.912056Z",
"url": "https://files.pythonhosted.org/packages/e4/2a/d4fc178709bd3cdb594202d054ebcff16798cc4db92ccef502aa558f0416/slimcontext-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5c8e8f951ffcd51a8ec5d4df2fccf2a2034549197a9362f9faf14da0cd812fd5",
"md5": "d3d76facabd42c69bccd702c8e32f534",
"sha256": "fb5638875cea728b8ac96bc3b2b55397ec5fca374606732abdfd50b66f6ec192"
},
"downloads": -1,
"filename": "slimcontext-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "d3d76facabd42c69bccd702c8e32f534",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 17246,
"upload_time": "2024-12-09T05:30:51",
"upload_time_iso_8601": "2024-12-09T05:30:51.631136Z",
"url": "https://files.pythonhosted.org/packages/5c/8e/8f951ffcd51a8ec5d4df2fccf2a2034549197a9362f9faf14da0cd812fd5/slimcontext-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-09 05:30:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "slimcontext"
}
