## Spare Cores Crawler
[![Build](https://img.shields.io/github/actions/workflow/status/SpareCores/sc-crawler/tests.yaml)](https://github.com/SpareCores/sc-crawler/actions/workflows/tests.yaml)
<picture><source media="(prefers-color-scheme: dark)" srcset="https://img.shields.io/badge/status-alpha-blue"><source media="(prefers-color-scheme: light)" srcset="https://img.shields.io/badge/status-alpha-blue"><img alt="Project Status: Alpha" src="https://img.shields.io/badge/status-alpha-blue"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="https://img.shields.io/maintenance/yes/2024"><source media="(prefers-color-scheme: light)" srcset="https://img.shields.io/maintenance/yes/2024"><img alt="Maintenance Status: Active" src="https://img.shields.io/maintenance/yes/2024"></picture>
[![MPL 2.0 License](https://img.shields.io/github/license/SpareCores/sc-crawler)](https://github.com/SpareCores/sc-crawler/blob/main/LICENSE)
[![PyPI - Python Version](https://img.shields.io/pypi/v/sparecores-crawler?logo=python&logoColor=ffdd54)](https://pypi.org/project/sparecores-crawler/)
[![NGI Search Open Call 3 beneficiary](https://img.shields.io/badge/NGI%20Search-Open%20Call%20%233-blue?logo=data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4NCjwhLS0gR2VuZXJhdG9yOiBBZG9iZSBJbGx1c3RyYXRvciAyMi4xLjAsIFNWRyBFeHBvcnQgUGx1Zy1JbiAuIFNWRyBWZXJzaW9uOiA2LjAwIEJ1aWxkIDApICAtLT4NCjxzdmcgdmVyc2lvbj0iMS4xIiBpZD0iTGF5ZXJfMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgeD0iMHB4IiB5PSIwcHgiDQoJIHdpZHRoPSIxODBweCIgaGVpZ2h0PSIxODBweCIgdmlld0JveD0iMCAwIDE4MCAxODAiIHN0eWxlPSJlbmFibGUtYmFja2dyb3VuZDpuZXcgMCAwIDE4MCAxODA7IiB4bWw6c3BhY2U9InByZXNlcnZlIj4NCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+DQoJLnN0MHtjbGlwLXBhdGg6dXJsKCNTVkdJRF8yXyk7fQ0KCS5zdDF7Y2xpcC1wYXRoOnVybCgjU1ZHSURfNF8pO2ZpbGw6dXJsKCNTVkdJRF81Xyk7fQ0KCS5zdDJ7ZmlsbDp1cmwoI1NWR0lEXzZfKTt9DQoJLnN0M3tmaWxsOiNGRkZGRkY7fQ0KPC9zdHlsZT4NCjxnPg0KCTxnPg0KCQk8ZGVmcz4NCgkJCTxwYXRoIGlkPSJTVkdJRF8xXyIgZD0iTTkwLDVDNDMuMiw1LDUsNDMuMiw1LDkwdjBjMCw0Ni43LDM4LjIsODUsODUsODVoMGM0Ni43LDAsODUtMzguMiw4NS04NXYwQzE3NSw0My4yLDEzNi44LDUsOTAsNUw5MCw1eiINCgkJCQkvPg0KCQk8L2RlZnM+DQoJCTxjbGlwUGF0aCBpZD0iU1ZHSURfMl8iPg0KCQkJPHVzZSB4bGluazpocmVmPSIjU1ZHSURfMV8iICBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTsiLz4NCgkJPC9jbGlwUGF0aD4NCgkJPGcgY2xhc3M9InN0MCI+DQoJCQk8Zz4NCgkJCQk8ZGVmcz4NCgkJCQkJPHJlY3QgaWQ9IlNWR0lEXzNfIiB3aWR0aD0iMTgwIiBoZWlnaHQ9IjE4MCIvPg0KCQkJCTwvZGVmcz4NCgkJCQk8Y2xpcFBhdGggaWQ9IlNWR0lEXzRfIj4NCgkJCQkJPHVzZSB4bGluazpocmVmPSIjU1ZHSURfM18iICBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTsiLz4NCgkJCQk8L2NsaXBQYXRoPg0KCQkJCQ0KCQkJCQk8bGluZWFyR3JhZGllbnQgaWQ9IlNWR0lEXzVfIiBncmFkaWVudFVuaXRzPSJ1c2VyU3BhY2VPblVzZSIgeDE9Ii0zMjQuNTY0MyIgeTE9IjIxMi4yNTg1IiB4Mj0iLTMyMy4zOTAzIiB5Mj0iMjEyLjI1ODUiIGdyYWRpZW50VHJhbnNmb3JtPSJtYXRyaXgoLTE1NS40ODE5IDE1MS4wOTY2IC0xNTEuMDk2NiAtMTU1LjQ4MTkgLTE4MjExLjA1ODYgODIwNDQuMjI2NikiPg0KCQkJCQk8c3RvcCAgb2Zmc2V0PSIwIiBzdHlsZT0ic3RvcC1jb2xvcjojRTUzNTAwIi8+DQoJCQkJCTxzdG9wICBvZmZzZXQ9IjEiIHN0eWxlPSJzdG9wLWNvbG9yOiM1MDAwMkQiLz4NCgkJCQk8L2xpbmVhckdyYWRpZW50Pg0KCQkJCTxwb2x5Z29uIGNsYXNzPSJzdDEiIHBvaW50cz0iMjcwLDkyLjYgODcuNCwyNzAgLTkwLDg3LjQgOTIuNiwtOTAgCQkJCSIvPg0KCQkJPC9nPg0KCQk8L2c+DQoJCTxnIGNsYXNzPSJzdDAiPg0KCQkJDQoJCQkJPGxpbmVhckdyYWRpZW50IGlkPSJTVkdJRF82XyIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIHgxPSItMzM3Ljg2MDYiIHkxPSIxOTguNzY1NSIgeDI9Ii0zMzguMzY1MiIgeTI9IjE5OC43NzI3IiBncmFkaWVudFRyYW5zZm9ybT0ibWF0cml4KC0xMjEuNzc5NCAxMTguMzQ0NiAtMTE4LjM0NDYgLTEyMS43Nzk0IC0xNzU2MS45Mzc1IDY0MzA5LjgxMjUpIj4NCgkJCQk8c3RvcCAgb2Zmc2V0PSIwIiBzdHlsZT0ic3RvcC1jb2xvcjojNTAwMDJEIi8+DQoJCQkJPHN0b3AgIG9mZnNldD0iMSIgc3R5bGU9InN0b3AtY29sb3I6I0U1MzUwMCIvPg0KCQkJPC9saW5lYXJHcmFkaWVudD4NCgkJCTxwYXRoIGNsYXNzPSJzdDIiIGQ9Ik0xMDUuNiw4OC4yVjYzLjhjMC00LjMsMy41LTcuOCw3LjgtNy44aDBjNC4zLDAsNy44LDMuNSw3LjgsNy44djUyLjRjMCw0LjMtMy41LDcuOC03LjgsNy44aC0xLjUNCgkJCQljLTIuMywwLTQuNS0xLTYtMi44TDgwLjEsODkuN2MtMS45LTIuMy01LjctMS01LjcsMnYyNC41YzAsNC4zLTMuNSw3LjgtNy44LDcuOHMtNy44LTMuNS03LjgtNy44VjYzLjhjMC00LjMsMy41LTcuOCw3LjgtNy44DQoJCQkJaDEuNmMyLjMsMCw0LjUsMSw2LDIuOGwyNS43LDMxLjRDMTAxLjgsOTIuNiwxMDUuNiw5MS4zLDEwNS42LDg4LjJ6Ii8+DQoJCQk8cGF0aCBjbGFzcz0ic3QzIiBkPSJNNjguMiw1NmgtMS42Yy00LjMsMC03LjgsMy41LTcuOCw3Ljh2NTIuNGMwLDQuMywzLjUsNy44LDcuOCw3LjhzNy44LTMuNSw3LjgtNy44VjkxLjdjMC0zLDMuOC00LjQsNS43LTINCgkJCQlsMjUuOCwzMS41YzEuNSwxLjgsMy43LDIuOCw2LDIuOGgxLjVjNC4zLDAsNy44LTMuNSw3LjgtNy44VjYzLjhjMC00LjMtMy41LTcuOC03LjgtNy44aDBjLTQuMywwLTcuOCwzLjUtNy44LDcuOHYyNC41DQoJCQkJYzAsMy0zLjgsNC40LTUuNywyTDc0LjIsNTguOUM3Mi43LDU3LjEsNzAuNSw1Niw2OC4yLDU2eiIvPg0KCQk8L2c+DQoJPC9nPg0KPC9nPg0KPC9zdmc+DQo=)](https://www.ngisearch.eu/view/Events/OC3Searchers)
SC Crawler is a Python package to pull and standardize data on cloud
compute resources, with tooling to help organize and update the
collected data into databases.
## Installation
Stable version from PyPI:
```
pip install sparecores-crawler
```
Most recent version from GitHub:
```
pip install "sparecores-crawler @ git+https://git@github.com/SpareCores/sc-crawler.git"
```
## References
- [Package documentation](https://sparecores.github.io/sc-crawler/)
- [Database schemas](https://dbdocs.io/spare-cores/sc-crawler)
- [Latest SQLite database release](https://sc-data-public-40e9d310.s3.amazonaws.com/sc-data-all.db.bz2)
- [sparecores.com](https://sparecores.com)
Raw data
{
"_id": null,
"home_page": null,
"name": "sparecores-crawler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "Spare Cores team <pkg@sparecores.com>",
"keywords": "cloud, compute, etl, sqlite, spot-instances, cost-optimization",
"author": "Attila Nagy, Gergely Daroczi, Balazs Hodobay",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/98/5e/94d8f46fbb64adff7f3e68a11f5bfe96c1c981fe921878538e7a0cfb272b/sparecores_crawler-0.1.3.tar.gz",
"platform": null,
"description": "## Spare Cores Crawler\n\n[![Build](https://img.shields.io/github/actions/workflow/status/SpareCores/sc-crawler/tests.yaml)](https://github.com/SpareCores/sc-crawler/actions/workflows/tests.yaml)\n<picture><source media=\"(prefers-color-scheme: dark)\" srcset=\"https://img.shields.io/badge/status-alpha-blue\"><source media=\"(prefers-color-scheme: light)\" srcset=\"https://img.shields.io/badge/status-alpha-blue\"><img alt=\"Project Status: Alpha\" src=\"https://img.shields.io/badge/status-alpha-blue\"></picture>\n<picture><source media=\"(prefers-color-scheme: dark)\" srcset=\"https://img.shields.io/maintenance/yes/2024\"><source media=\"(prefers-color-scheme: light)\" srcset=\"https://img.shields.io/maintenance/yes/2024\"><img alt=\"Maintenance Status: Active\" src=\"https://img.shields.io/maintenance/yes/2024\"></picture>\n[![MPL 2.0 License](https://img.shields.io/github/license/SpareCores/sc-crawler)](https://github.com/SpareCores/sc-crawler/blob/main/LICENSE)\n[![PyPI - Python Version](https://img.shields.io/pypi/v/sparecores-crawler?logo=python&logoColor=ffdd54)](https://pypi.org/project/sparecores-crawler/)\n[![NGI Search Open Call 3 beneficiary](https://img.shields.io/badge/NGI%20Search-Open%20Call%20%233-blue?logo=data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4NCjwhLS0gR2VuZXJhdG9yOiBBZG9iZSBJbGx1c3RyYXRvciAyMi4xLjAsIFNWRyBFeHBvcnQgUGx1Zy1JbiAuIFNWRyBWZXJzaW9uOiA2LjAwIEJ1aWxkIDApICAtLT4NCjxzdmcgdmVyc2lvbj0iMS4xIiBpZD0iTGF5ZXJfMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgeD0iMHB4IiB5PSIwcHgiDQoJIHdpZHRoPSIxODBweCIgaGVpZ2h0PSIxODBweCIgdmlld0JveD0iMCAwIDE4MCAxODAiIHN0eWxlPSJlbmFibGUtYmFja2dyb3VuZDpuZXcgMCAwIDE4MCAxODA7IiB4bWw6c3BhY2U9InByZXNlcnZlIj4NCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+DQoJLnN0MHtjbGlwLXBhdGg6dXJsKCNTVkdJRF8yXyk7fQ0KCS5zdDF7Y2xpcC1wYXRoOnVybCgjU1ZHSURfNF8pO2ZpbGw6dXJsKCNTVkdJRF81Xyk7fQ0KCS5zdDJ7ZmlsbDp1cmwoI1NWR0lEXzZfKTt9DQoJLnN0M3tmaWxsOiNGRkZGRkY7fQ0KPC9zdHlsZT4NCjxnPg0KCTxnPg0KCQk8ZGVmcz4NCgkJCTxwYXRoIGlkPSJTVkdJRF8xXyIgZD0iTTkwLDVDNDMuMiw1LDUsNDMuMiw1LDkwdjBjMCw0Ni43LDM4LjIsODUsODUsODVoMGM0Ni43LDAsODUtMzguMiw4NS04NXYwQzE3NSw0My4yLDEzNi44LDUsOTAsNUw5MCw1eiINCgkJCQkvPg0KCQk8L2RlZnM+DQoJCTxjbGlwUGF0aCBpZD0iU1ZHSURfMl8iPg0KCQkJPHVzZSB4bGluazpocmVmPSIjU1ZHSURfMV8iICBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTsiLz4NCgkJPC9jbGlwUGF0aD4NCgkJPGcgY2xhc3M9InN0MCI+DQoJCQk8Zz4NCgkJCQk8ZGVmcz4NCgkJCQkJPHJlY3QgaWQ9IlNWR0lEXzNfIiB3aWR0aD0iMTgwIiBoZWlnaHQ9IjE4MCIvPg0KCQkJCTwvZGVmcz4NCgkJCQk8Y2xpcFBhdGggaWQ9IlNWR0lEXzRfIj4NCgkJCQkJPHVzZSB4bGluazpocmVmPSIjU1ZHSURfM18iICBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTsiLz4NCgkJCQk8L2NsaXBQYXRoPg0KCQkJCQ0KCQkJCQk8bGluZWFyR3JhZGllbnQgaWQ9IlNWR0lEXzVfIiBncmFkaWVudFVuaXRzPSJ1c2VyU3BhY2VPblVzZSIgeDE9Ii0zMjQuNTY0MyIgeTE9IjIxMi4yNTg1IiB4Mj0iLTMyMy4zOTAzIiB5Mj0iMjEyLjI1ODUiIGdyYWRpZW50VHJhbnNmb3JtPSJtYXRyaXgoLTE1NS40ODE5IDE1MS4wOTY2IC0xNTEuMDk2NiAtMTU1LjQ4MTkgLTE4MjExLjA1ODYgODIwNDQuMjI2NikiPg0KCQkJCQk8c3RvcCAgb2Zmc2V0PSIwIiBzdHlsZT0ic3RvcC1jb2xvcjojRTUzNTAwIi8+DQoJCQkJCTxzdG9wICBvZmZzZXQ9IjEiIHN0eWxlPSJzdG9wLWNvbG9yOiM1MDAwMkQiLz4NCgkJCQk8L2xpbmVhckdyYWRpZW50Pg0KCQkJCTxwb2x5Z29uIGNsYXNzPSJzdDEiIHBvaW50cz0iMjcwLDkyLjYgODcuNCwyNzAgLTkwLDg3LjQgOTIuNiwtOTAgCQkJCSIvPg0KCQkJPC9nPg0KCQk8L2c+DQoJCTxnIGNsYXNzPSJzdDAiPg0KCQkJDQoJCQkJPGxpbmVhckdyYWRpZW50IGlkPSJTVkdJRF82XyIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIHgxPSItMzM3Ljg2MDYiIHkxPSIxOTguNzY1NSIgeDI9Ii0zMzguMzY1MiIgeTI9IjE5OC43NzI3IiBncmFkaWVudFRyYW5zZm9ybT0ibWF0cml4KC0xMjEuNzc5NCAxMTguMzQ0NiAtMTE4LjM0NDYgLTEyMS43Nzk0IC0xNzU2MS45Mzc1IDY0MzA5LjgxMjUpIj4NCgkJCQk8c3RvcCAgb2Zmc2V0PSIwIiBzdHlsZT0ic3RvcC1jb2xvcjojNTAwMDJEIi8+DQoJCQkJPHN0b3AgIG9mZnNldD0iMSIgc3R5bGU9InN0b3AtY29sb3I6I0U1MzUwMCIvPg0KCQkJPC9saW5lYXJHcmFkaWVudD4NCgkJCTxwYXRoIGNsYXNzPSJzdDIiIGQ9Ik0xMDUuNiw4OC4yVjYzLjhjMC00LjMsMy41LTcuOCw3LjgtNy44aDBjNC4zLDAsNy44LDMuNSw3LjgsNy44djUyLjRjMCw0LjMtMy41LDcuOC03LjgsNy44aC0xLjUNCgkJCQljLTIuMywwLTQuNS0xLTYtMi44TDgwLjEsODkuN2MtMS45LTIuMy01LjctMS01LjcsMnYyNC41YzAsNC4zLTMuNSw3LjgtNy44LDcuOHMtNy44LTMuNS03LjgtNy44VjYzLjhjMC00LjMsMy41LTcuOCw3LjgtNy44DQoJCQkJaDEuNmMyLjMsMCw0LjUsMSw2LDIuOGwyNS43LDMxLjRDMTAxLjgsOTIuNiwxMDUuNiw5MS4zLDEwNS42LDg4LjJ6Ii8+DQoJCQk8cGF0aCBjbGFzcz0ic3QzIiBkPSJNNjguMiw1NmgtMS42Yy00LjMsMC03LjgsMy41LTcuOCw3Ljh2NTIuNGMwLDQuMywzLjUsNy44LDcuOCw3LjhzNy44LTMuNSw3LjgtNy44VjkxLjdjMC0zLDMuOC00LjQsNS43LTINCgkJCQlsMjUuOCwzMS41YzEuNSwxLjgsMy43LDIuOCw2LDIuOGgxLjVjNC4zLDAsNy44LTMuNSw3LjgtNy44VjYzLjhjMC00LjMtMy41LTcuOC03LjgtNy44aDBjLTQuMywwLTcuOCwzLjUtNy44LDcuOHYyNC41DQoJCQkJYzAsMy0zLjgsNC40LTUuNywyTDc0LjIsNTguOUM3Mi43LDU3LjEsNzAuNSw1Niw2OC4yLDU2eiIvPg0KCQk8L2c+DQoJPC9nPg0KPC9nPg0KPC9zdmc+DQo=)](https://www.ngisearch.eu/view/Events/OC3Searchers)\n\nSC Crawler is a Python package to pull and standardize data on cloud\ncompute resources, with tooling to help organize and update the\ncollected data into databases.\n\n## Installation\n\nStable version from PyPI:\n\n```\npip install sparecores-crawler\n```\n\nMost recent version from GitHub:\n\n```\npip install \"sparecores-crawler @ git+https://git@github.com/SpareCores/sc-crawler.git\"\n```\n\n## References\n\n- [Package documentation](https://sparecores.github.io/sc-crawler/)\n- [Database schemas](https://dbdocs.io/spare-cores/sc-crawler)\n- [Latest SQLite database release](https://sc-data-public-40e9d310.s3.amazonaws.com/sc-data-all.db.bz2)\n- [sparecores.com](https://sparecores.com)\n",
"bugtrack_url": null,
"license": null,
"summary": "Pull and standardize data on cloud compute resources.",
"version": "0.1.3",
"project_urls": {
"documentation": "https://sparecores.github.io/sc-crawler/",
"homepage": "https://sparecores.com",
"issues": "https://github.com/SpareCores/sc-crawler/issues",
"repository": "https://github.com/SpareCores/sc-crawler"
},
"split_keywords": [
"cloud",
" compute",
" etl",
" sqlite",
" spot-instances",
" cost-optimization"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ca9803e879268f196938416699c671ca7e2d8b630dfb82b4ce8aa10c203f284d",
"md5": "855914bd63ff61411c254b1587a85b89",
"sha256": "2fb73f8c8a4a179b87087aa82659efaad83f761cc45e9b5aba1f97dfb4cabd40"
},
"downloads": -1,
"filename": "sparecores_crawler-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "855914bd63ff61411c254b1587a85b89",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 72805,
"upload_time": "2024-05-09T22:03:23",
"upload_time_iso_8601": "2024-05-09T22:03:23.037922Z",
"url": "https://files.pythonhosted.org/packages/ca/98/03e879268f196938416699c671ca7e2d8b630dfb82b4ce8aa10c203f284d/sparecores_crawler-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "985e94d8f46fbb64adff7f3e68a11f5bfe96c1c981fe921878538e7a0cfb272b",
"md5": "94e834f8aa1c778f1c0990561308ed22",
"sha256": "6260c2089b43930b3e429707be0bab32a7a69b140fbd9124435cc0f625dcb9c1"
},
"downloads": -1,
"filename": "sparecores_crawler-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "94e834f8aa1c778f1c0990561308ed22",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 68843,
"upload_time": "2024-05-09T22:03:25",
"upload_time_iso_8601": "2024-05-09T22:03:25.067032Z",
"url": "https://files.pythonhosted.org/packages/98/5e/94d8f46fbb64adff7f3e68a11f5bfe96c1c981fe921878538e7a0cfb272b/sparecores_crawler-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-09 22:03:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SpareCores",
"github_project": "sc-crawler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "sparecores-crawler"
}