| Name | md-summarizer JSON |
| Version |
0.1.64
JSON |
| download |
| home_page | None |
| Summary | AI-powered Markdown summarizer that preserves structure and code blocks |
| upload_time | 2025-01-15 22:03:14 |
| maintainer | None |
| docs_url | None |
| author | Celtiberi |
| requires_python | >=3.11 |
| license | None |
| keywords |
ai
markdown
nlp
summarizer
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Markdown Summarizer
An AI-powered tool that reduces the token size of Markdown documents while preserving their essential structure, code blocks, and meaning. Designed to help fit documentation into AI context windows without losing critical information.
## Features
- **Token Reduction**: Intelligently reduces document size
- Preserves markdown heading hierarchy
- Protects code blocks and technical details
- Bottom-up concurrent processing
- Real-time progress updates
- Streaming API
- Multiple AI provider support
## Usage
```python
from md_summarizer import (
MarkdownSummarizer,
ProgressStatus
)
# Basic usage
summarizer = MarkdownSummarizer()
result = await summarizer.summarize(content)
# Streaming updates
async for update in summarizer.stream(content):
if update.status == ProgressStatus.STARTING:
print(f"Processing {update.total_sections} sections...")
elif update.status == ProgressStatus.SECTION_COMPLETE:
print(f"Completed section: {update.section_title}")
elif update.status == ProgressStatus.COMPLETE:
print("Done!")
print(update.content)
# Progress update types:
# - ProgressStatus.STARTING: total_sections count
# - ProgressStatus.SECTION_COMPLETE: section_title of completed section
# - ProgressStatus.COMPLETE: final content
# - ProgressStatus.ERROR: error details if something fails
# Customize prompts
summarizer.system_prompt = "Your custom system prompt"
summarizer.user_prompt = "Your custom user prompt"
```
Example output:
```
Status: ProgressStatus.STARTING, Total Sections: 4
Status: ProgressStatus.SECTION_COMPLETE, Section: Subsection 2.1
Status: ProgressStatus.SECTION_COMPLETE, Section: Section 1
Status: ProgressStatus.SECTION_COMPLETE, Section: Section 2
Status: ProgressStatus.SECTION_COMPLETE, Section: Test Document
Status: ProgressStatus.COMPLETE
```
## Installation
The module has not yet been published to PyPI. So you need to install it from the source code.
## Configuration
Set environment variables or use .env file:
```
OPENAI_API_KEY=your-key
MODEL=gpt-3.5-turbo
PROVIDER=openai
LOG_LEVEL=INFO
```
## How it Works
1. Parses markdown into hierarchical sections
2. Processes sections bottom-up (children before parents)
3. Preserves heading levels and structure
4. Provides real-time progress updates
5. Combines processed sections into final document
## Development
```bash
# Install development dependencies
pip install -e ".[test]"
# Run tests
make test
```
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first.
## License
This project is licensed under the GNU Affero General Public License v3.0 - see the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "md-summarizer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "ai, markdown, nlp, summarizer",
"author": "Celtiberi",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/bc/6f/79b85eaa842404a701c8fd6b5b09e6697599d3ebd7e23fbf358b3b765cb7/md_summarizer-0.1.64.tar.gz",
"platform": null,
"description": "# Markdown Summarizer\n\nAn AI-powered tool that reduces the token size of Markdown documents while preserving their essential structure, code blocks, and meaning. Designed to help fit documentation into AI context windows without losing critical information.\n\n## Features\n\n- **Token Reduction**: Intelligently reduces document size\n- Preserves markdown heading hierarchy\n- Protects code blocks and technical details\n- Bottom-up concurrent processing\n- Real-time progress updates\n- Streaming API\n- Multiple AI provider support\n\n## Usage\n\n```python\nfrom md_summarizer import (\n MarkdownSummarizer, \n ProgressStatus\n)\n\n# Basic usage\nsummarizer = MarkdownSummarizer()\nresult = await summarizer.summarize(content)\n\n# Streaming updates\nasync for update in summarizer.stream(content):\n if update.status == ProgressStatus.STARTING:\n print(f\"Processing {update.total_sections} sections...\")\n elif update.status == ProgressStatus.SECTION_COMPLETE:\n print(f\"Completed section: {update.section_title}\")\n elif update.status == ProgressStatus.COMPLETE:\n print(\"Done!\")\n print(update.content)\n\n# Progress update types:\n# - ProgressStatus.STARTING: total_sections count\n# - ProgressStatus.SECTION_COMPLETE: section_title of completed section\n# - ProgressStatus.COMPLETE: final content\n# - ProgressStatus.ERROR: error details if something fails\n\n# Customize prompts\nsummarizer.system_prompt = \"Your custom system prompt\"\nsummarizer.user_prompt = \"Your custom user prompt\"\n```\n\nExample output:\n```\nStatus: ProgressStatus.STARTING, Total Sections: 4\nStatus: ProgressStatus.SECTION_COMPLETE, Section: Subsection 2.1\nStatus: ProgressStatus.SECTION_COMPLETE, Section: Section 1\nStatus: ProgressStatus.SECTION_COMPLETE, Section: Section 2\nStatus: ProgressStatus.SECTION_COMPLETE, Section: Test Document\nStatus: ProgressStatus.COMPLETE\n```\n\n## Installation\n\nThe module has not yet been published to PyPI. So you need to install it from the source code.\n\n## Configuration\n\nSet environment variables or use .env file:\n```\nOPENAI_API_KEY=your-key\nMODEL=gpt-3.5-turbo\nPROVIDER=openai\nLOG_LEVEL=INFO\n```\n\n## How it Works\n\n1. Parses markdown into hierarchical sections\n2. Processes sections bottom-up (children before parents)\n3. Preserves heading levels and structure\n4. Provides real-time progress updates\n5. Combines processed sections into final document\n\n## Development\n\n```bash\n# Install development dependencies\npip install -e \".[test]\"\n\n# Run tests\nmake test\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first.\n\n## License\n\nThis project is licensed under the GNU Affero General Public License v3.0 - see the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": null,
"summary": "AI-powered Markdown summarizer that preserves structure and code blocks",
"version": "0.1.64",
"project_urls": {
"Bug Tracker": "https://github.com/yourusername/md-summarizer/issues",
"Homepage": "https://github.com/yourusername/md-summarizer",
"Source": "https://github.com/yourusername/md-summarizer"
},
"split_keywords": [
"ai",
" markdown",
" nlp",
" summarizer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "23ab1d989ee238013bdb635502fc14fb0d3798db1e461eed8513315fa5e5046c",
"md5": "d7e9af2a522396eee69609f2ff016bc5",
"sha256": "15e5645bd651be7fb8f91190c5375d0ed830705f303b5d21d65ae4825e7db3c2"
},
"downloads": -1,
"filename": "md_summarizer-0.1.64-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d7e9af2a522396eee69609f2ff016bc5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 24775,
"upload_time": "2025-01-15T22:03:13",
"upload_time_iso_8601": "2025-01-15T22:03:13.105080Z",
"url": "https://files.pythonhosted.org/packages/23/ab/1d989ee238013bdb635502fc14fb0d3798db1e461eed8513315fa5e5046c/md_summarizer-0.1.64-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bc6f79b85eaa842404a701c8fd6b5b09e6697599d3ebd7e23fbf358b3b765cb7",
"md5": "b9d4dcf0efa13c3f060d2a16f0e0669e",
"sha256": "8d2fbac4a5751c7c4620fbc6fdd70d2368852bf7af49a94ec6db7172b30cdb9f"
},
"downloads": -1,
"filename": "md_summarizer-0.1.64.tar.gz",
"has_sig": false,
"md5_digest": "b9d4dcf0efa13c3f060d2a16f0e0669e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 28176,
"upload_time": "2025-01-15T22:03:14",
"upload_time_iso_8601": "2025-01-15T22:03:14.962460Z",
"url": "https://files.pythonhosted.org/packages/bc/6f/79b85eaa842404a701c8fd6b5b09e6697599d3ebd7e23fbf358b3b765cb7/md_summarizer-0.1.64.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-15 22:03:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "md-summarizer",
"github_not_found": true,
"lcname": "md-summarizer"
}
