| Name | information-composer JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | None |
| Summary | A comprehensive toolkit for collecting, composing, and filtering information from various web resources with AI-powered markdown processing |
| upload_time | 2025-09-10 11:37:22 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.7 |
| license | MIT License
Copyright (c) 2024 Tao Zhang
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
|
| keywords |
web scraping
information collection
data composition
markdown
llm
filter
academic
paper
nlp
ai
dashscope
llama-index
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Information Composer
[](https://github.com/yourusername/information-composer/actions/workflows/code-quality.yaml)
[](https://www.python.org/)
[](LICENSE)
[](https://github.com/astral-sh/ruff)
A comprehensive toolkit for collecting, composing, and filtering information from various web resources with AI-powered markdown processing.
## Features
- **PDF Validation**: Validate PDF file formats and integrity
- **Markdown Processing**: Advanced markdown processing with LLM filtering
- **PubMed Integration**: Query and process PubMed data
- **DOI Management**: Download and manage DOI references
- **Code Quality**: Automated code quality checks with Ruff
- **Multi-format Support**: Support for various data formats and sources
## Installation
### Prerequisites
- Python 3.10, 3.11, 3.12, or 3.13
- Virtual environment (recommended)
### Setup
1. Clone the repository:
```bash
git clone https://github.com/yourusername/information-composer.git
cd information-composer
```
2. Create and activate virtual environment:
```bash
# Linux/macOS
python -m venv .venv
source .venv/bin/activate
# Windows
python -m venv .venv
.venv\Scripts\activate
```
3. Install dependencies:
```bash
pip install -e .
```
## Quick Start
### Activate Environment
```bash
# Linux/macOS
source activate.sh
# Windows
activate.bat
```
### Available Commands
- `md-llm-filter` - Run MD_LLM_Filter CLI
- `pdf-validator` - Run PDF validator CLI
- `python -m information_composer.core.doi_downloader` - Run DOI downloader
- `python -m information_composer.pubmed.pubmed` - Run PubMed tools
### Examples
```bash
# Validate PDF files
pdf-validator file.pdf
# Validate directory of PDFs
pdf-validator -d /path/to/directory -r
# Filter markdown with LLM
md-llm-filter input.md output.md
# Run code quality checks
python scripts/check_code.py --fix
```
## Development
### Code Quality
This project uses Ruff for code quality checks:
```bash
# Run all checks
python scripts/check_code.py
# Auto-fix issues
python scripts/check_code.py --fix
# With verbose output
python scripts/check_code.py --verbose
```
### Testing
```bash
# Run tests
python scripts/check_code.py --with-tests
# Or directly with pytest
pytest tests/ -v
```
## CI/CD
This project uses GitHub Actions for continuous integration:
- **Code Quality**: Automated Ruff checks on multiple Python versions
- **Testing**: Comprehensive test suite execution
- **Release**: Automated package building and publishing
See [.github/README.md](.github/README.md) for detailed CI/CD documentation.
## Documentation
- [๐ ๅฎๆดๆๆกฃ](docs/README.md) - ้กน็ฎๅฎๆดๆๆกฃ
- [๐ ๅฟซ้ๅผๅง](docs/quickstart.md) - 5ๅ้ๅฟซ้ไธๆ
- [โ๏ธ ้
็ฝฎ่ฏดๆ](docs/configuration.md) - ่ฏฆ็ป้
็ฝฎ้้กน
- [๐ ๅ่ฝๆๅ](docs/guides/) - ๅๅ่ฝ่ฏฆ็ป่ฏดๆ
- [๐ง ๅผๅๆๅ](docs/development/) - ๅผๅๅ่ดก็ฎๆๅ
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run code quality checks: `python scripts/check_code.py --fix`
5. Run tests: `python scripts/check_code.py --with-tests`
6. Submit a pull request
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Support
For questions and support, please open an issue on GitHub.
Raw data
{
"_id": null,
"home_page": null,
"name": "information-composer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "web scraping, information collection, data composition, markdown, llm, filter, academic, paper, nlp, ai, dashscope, llama-index",
"author": null,
"author_email": "Tao Zhang <forrest_zhang@163.com>",
"download_url": "https://files.pythonhosted.org/packages/ec/66/c39ade59d4f59e070f7eff0c9d36181f28cca86bfb43fb4291fc4d3f253f/information_composer-0.1.3.tar.gz",
"platform": null,
"description": "# Information Composer\n\n[](https://github.com/yourusername/information-composer/actions/workflows/code-quality.yaml)\n[](https://www.python.org/)\n[](LICENSE)\n[](https://github.com/astral-sh/ruff)\n\nA comprehensive toolkit for collecting, composing, and filtering information from various web resources with AI-powered markdown processing.\n\n## Features\n\n- **PDF Validation**: Validate PDF file formats and integrity\n- **Markdown Processing**: Advanced markdown processing with LLM filtering\n- **PubMed Integration**: Query and process PubMed data\n- **DOI Management**: Download and manage DOI references\n- **Code Quality**: Automated code quality checks with Ruff\n- **Multi-format Support**: Support for various data formats and sources\n\n## Installation\n\n### Prerequisites\n\n- Python 3.10, 3.11, 3.12, or 3.13\n- Virtual environment (recommended)\n\n### Setup\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/information-composer.git\ncd information-composer\n```\n\n2. Create and activate virtual environment:\n```bash\n# Linux/macOS\npython -m venv .venv\nsource .venv/bin/activate\n\n# Windows\npython -m venv .venv\n.venv\\Scripts\\activate\n```\n\n3. Install dependencies:\n```bash\npip install -e .\n```\n\n## Quick Start\n\n### Activate Environment\n```bash\n# Linux/macOS\nsource activate.sh\n\n# Windows\nactivate.bat\n```\n\n### Available Commands\n\n- `md-llm-filter` - Run MD_LLM_Filter CLI\n- `pdf-validator` - Run PDF validator CLI\n- `python -m information_composer.core.doi_downloader` - Run DOI downloader\n- `python -m information_composer.pubmed.pubmed` - Run PubMed tools\n\n### Examples\n\n```bash\n# Validate PDF files\npdf-validator file.pdf\n\n# Validate directory of PDFs\npdf-validator -d /path/to/directory -r\n\n# Filter markdown with LLM\nmd-llm-filter input.md output.md\n\n# Run code quality checks\npython scripts/check_code.py --fix\n```\n\n## Development\n\n### Code Quality\n\nThis project uses Ruff for code quality checks:\n\n```bash\n# Run all checks\npython scripts/check_code.py\n\n# Auto-fix issues\npython scripts/check_code.py --fix\n\n# With verbose output\npython scripts/check_code.py --verbose\n```\n\n### Testing\n\n```bash\n# Run tests\npython scripts/check_code.py --with-tests\n\n# Or directly with pytest\npytest tests/ -v\n```\n\n## CI/CD\n\nThis project uses GitHub Actions for continuous integration:\n\n- **Code Quality**: Automated Ruff checks on multiple Python versions\n- **Testing**: Comprehensive test suite execution\n- **Release**: Automated package building and publishing\n\nSee [.github/README.md](.github/README.md) for detailed CI/CD documentation.\n\n## Documentation\n\n- [\ud83d\udcda \u5b8c\u6574\u6587\u6863](docs/README.md) - \u9879\u76ee\u5b8c\u6574\u6587\u6863\n- [\ud83d\ude80 \u5feb\u901f\u5f00\u59cb](docs/quickstart.md) - 5\u5206\u949f\u5feb\u901f\u4e0a\u624b\n- [\u2699\ufe0f \u914d\u7f6e\u8bf4\u660e](docs/configuration.md) - \u8be6\u7ec6\u914d\u7f6e\u9009\u9879\n- [\ud83d\udcd6 \u529f\u80fd\u6307\u5357](docs/guides/) - \u5404\u529f\u80fd\u8be6\u7ec6\u8bf4\u660e\n- [\ud83d\udd27 \u5f00\u53d1\u6307\u5357](docs/development/) - \u5f00\u53d1\u548c\u8d21\u732e\u6307\u5357\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Run code quality checks: `python scripts/check_code.py --fix`\n5. Run tests: `python scripts/check_code.py --with-tests`\n6. Submit a pull request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\nFor questions and support, please open an issue on GitHub.\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2024 Tao Zhang\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.\n ",
"summary": "A comprehensive toolkit for collecting, composing, and filtering information from various web resources with AI-powered markdown processing",
"version": "0.1.3",
"project_urls": {
"Documentation": "https://information-composer.readthedocs.io/",
"Homepage": "https://github.com/yourusername/information-composer",
"Issues": "https://github.com/yourusername/information-composer/issues",
"Repository": "https://github.com/yourusername/information-composer.git"
},
"split_keywords": [
"web scraping",
" information collection",
" data composition",
" markdown",
" llm",
" filter",
" academic",
" paper",
" nlp",
" ai",
" dashscope",
" llama-index"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "56f8ec4f2ab8cca8faebefd2708e27817db2537283849b55d628566ca5457c82",
"md5": "6b600c214162b3910b134c83d6c27936",
"sha256": "0d378e95c0ec78794e405c7afff22b569fff63dc1d5056b066bb15067b386050"
},
"downloads": -1,
"filename": "information_composer-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6b600c214162b3910b134c83d6c27936",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 97633,
"upload_time": "2025-09-10T11:37:20",
"upload_time_iso_8601": "2025-09-10T11:37:20.758985Z",
"url": "https://files.pythonhosted.org/packages/56/f8/ec4f2ab8cca8faebefd2708e27817db2537283849b55d628566ca5457c82/information_composer-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "ec66c39ade59d4f59e070f7eff0c9d36181f28cca86bfb43fb4291fc4d3f253f",
"md5": "26031c5a59c84cbc63cf0527dc931f01",
"sha256": "6891307d2e277ea63f9ff7d4d16aebf01396f5b1902d742300b88bd5de115e2b"
},
"downloads": -1,
"filename": "information_composer-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "26031c5a59c84cbc63cf0527dc931f01",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 119348,
"upload_time": "2025-09-10T11:37:22",
"upload_time_iso_8601": "2025-09-10T11:37:22.828629Z",
"url": "https://files.pythonhosted.org/packages/ec/66/c39ade59d4f59e070f7eff0c9d36181f28cca86bfb43fb4291fc4d3f253f/information_composer-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-10 11:37:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "information-composer",
"github_not_found": true,
"lcname": "information-composer"
}
