# UpToDateAI
UpToDateAI is a Python package designed to fetch and provide the latest documentation about recently released programming frameworks to AI models. This package helps bridge the gap between AI model training cut-off dates and the latest developments in the programming world.
## Installation
You can install UpToDateAI using pip:
```bash
pip install uptodateai
```
## Usage
URL of the website you want to crawl:
```python
from uptodateai import process_docs
process_docs("https://docs.fastht.ml/")
```
This will crawl the specified website and save the content as Markdown files in a `docs` directory.
## Features
- Web crawling using Scrapy
- Content extraction using newspaper3k
- HTML to Markdown conversion
- Automatic directory structure creation based on URL paths
- Customizable crawling settings
## Development
To set up the development environment:
1. Clone the repository
2. Install dependencies: `pip install -r requirements.txt`
3. Run tests: `python -m unittest discover tests`
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
## Contributing
Contributions are welcome! Please open an issue or submit a pull request.
Raw data
{
"_id": null,
"home_page": "https://github.com/yourusername/uptodateai",
"name": "uptodateai",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": "Your Name",
"author_email": "your.email@example.com",
"download_url": "https://files.pythonhosted.org/packages/cf/02/8c42b1c8c1c2cd7c201df04ed9a7edc21baff24047f5cf8bf9b6ce7457bf/uptodateai-0.1.0.tar.gz",
"platform": null,
"description": "# UpToDateAI\n\nUpToDateAI is a Python package designed to fetch and provide the latest documentation about recently released programming frameworks to AI models. This package helps bridge the gap between AI model training cut-off dates and the latest developments in the programming world.\n\n## Installation\n\nYou can install UpToDateAI using pip:\n\n```bash\npip install uptodateai\n```\n\n## Usage\n\nURL of the website you want to crawl:\n\n```python\nfrom uptodateai import process_docs\n\nprocess_docs(\"https://docs.fastht.ml/\")\n```\n\nThis will crawl the specified website and save the content as Markdown files in a `docs` directory.\n\n## Features\n\n- Web crawling using Scrapy\n- Content extraction using newspaper3k\n- HTML to Markdown conversion\n- Automatic directory structure creation based on URL paths\n- Customizable crawling settings\n\n## Development\n\nTo set up the development environment:\n\n1. Clone the repository\n2. Install dependencies: `pip install -r requirements.txt`\n3. Run tests: `python -m unittest discover tests`\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request.\n",
"bugtrack_url": null,
"license": null,
"summary": "A package for crawling and converting web content to Markdown",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/yourusername/uptodateai"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "90759419f1ef4f70cb3a474f363842871c0b3860124e4007dd87bf375a88708e",
"md5": "c6c3413b8f2fd44334fc76e061c52d6c",
"sha256": "bc9cb768baf1b1fef4421fb4aba3034bf69a6977713c2e66be3183067486e76c"
},
"downloads": -1,
"filename": "uptodateai-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c6c3413b8f2fd44334fc76e061c52d6c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 4907,
"upload_time": "2024-09-08T20:53:12",
"upload_time_iso_8601": "2024-09-08T20:53:12.881634Z",
"url": "https://files.pythonhosted.org/packages/90/75/9419f1ef4f70cb3a474f363842871c0b3860124e4007dd87bf375a88708e/uptodateai-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cf028c42b1c8c1c2cd7c201df04ed9a7edc21baff24047f5cf8bf9b6ce7457bf",
"md5": "f5fb204a7fbf909897987d9df1bd40d9",
"sha256": "ff23e95ec4d176d0fbbdc0508a6b09aa0bfafbcf47a89591051b21e696096a0f"
},
"downloads": -1,
"filename": "uptodateai-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "f5fb204a7fbf909897987d9df1bd40d9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 4556,
"upload_time": "2024-09-08T20:53:14",
"upload_time_iso_8601": "2024-09-08T20:53:14.840070Z",
"url": "https://files.pythonhosted.org/packages/cf/02/8c42b1c8c1c2cd7c201df04ed9a7edc21baff24047f5cf8bf9b6ce7457bf/uptodateai-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-08 20:53:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "uptodateai",
"github_not_found": true,
"lcname": "uptodateai"
}
JSON
