| Name | icrawler JSON |
| Version |
0.6.10
JSON |
| download |
| home_page | None |
| Summary | A multi-thread crawler framework with many builtin image crawlers provided. |
| upload_time | 2025-02-11 14:09:29 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.7 |
| license | The MIT License (MIT) Copyright (c) 2016 Kai Chen Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
crawler
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
beautifulsoup4
lxml
Pillow
requests
six
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
icrawler
========
.. image:: https://img.shields.io/pypi/v/icrawler.svg
:target: https://pypi.python.org/pypi/icrawler
:alt: PyPI Version
.. image:: https://anaconda.org/hellock/icrawler/badges/version.svg
:target: https://anaconda.org/hellock/icrawler
:alt: Anaconda Version
.. image:: https://img.shields.io/pypi/pyversions/icrawler.svg
:alt: Python Version
.. image:: https://img.shields.io/github/license/hellock/icrawler.svg
:alt: License
Introduction
------------
Documentation: http://icrawler.readthedocs.io/
Try it with ``pip install icrawler`` or ``conda install -c hellock icrawler``.
This package is a mini framework of web crawlers. With modularization design,
it is easy to use and extend. It supports media data like images and videos
very well, and can also be applied to texts and other type of files.
Scrapy is heavy and powerful, while icrawler is tiny and flexible.
With this package, you can write a multiple thread crawler easily by focusing on
the contents you want to crawl, keeping away from troublesome problems like
exception handling, thread scheduling and communication.
It also provides built-in crawlers for popular image sites like **Flickr** and
search engines such as **Google**, **Bing** and **Baidu**.
(Thank all the contributors and pull requests are always welcome!)
Requirements
------------
Python 3.5+ (recommended).
Examples
--------
Using built-in crawlers is very simple. A minimal example is shown as follows.
.. code:: python
from icrawler.builtin import GoogleImageCrawler
google_crawler = GoogleImageCrawler(storage={'root_dir': 'your_image_dir'})
google_crawler.crawl(keyword='cat', max_num=100)
You can also configurate number of threads and apply advanced search options.
(Note: compatible with 0.6.0 and later versions)
.. code:: python
from icrawler.builtin import GoogleImageCrawler
google_crawler = GoogleImageCrawler(
feeder_threads=1,
parser_threads=2,
downloader_threads=4,
storage={'root_dir': 'your_image_dir'})
filters = dict(
size='large',
color='orange',
license='commercial,modify',
date=((2017, 1, 1), (2017, 11, 30)))
google_crawler.crawl(keyword='cat', filters=filters, max_num=1000, file_idx_offset=0)
For more advanced usage about built-in crawlers, please refer to the
`documentation <http://icrawler.readthedocs.io/en/latest/builtin.html>`_.
Writing your own crawlers with this framework is also convenient, see the
`tutorials <http://icrawler.readthedocs.io/en/latest/extend.html>`_.
Architecture
------------
A crawler consists of 3 main components (Feeder, Parser and Downloader),
they are connected with each other with FIFO queues. The workflow is shown in
the following figure.
.. figure:: http://7xopqn.com1.z0.glb.clouddn.com/workflow.png
:alt:
- ``url_queue`` stores the url of pages which may contain images
- ``task_queue`` stores the image url as well as any meta data you
like, each element in the queue is a dictionary and must contain the
field ``img_url``
- Feeder puts page urls to ``url_queue``
- Parser requests and parses the page, then extracts the image urls and
puts them into ``task_queue``
- Downloader gets tasks from ``task_queue`` and requests the images,
then saves them in the given path.
Feeder, parser and downloader are all thread pools, so you can specify the
number of threads they use.
Raw data
{
"_id": null,
"home_page": null,
"name": "icrawler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "Kai Chen <chenkaidev@gmail.com>, Zhiyuan Chen <this@zyc.ai>",
"keywords": "Crawler",
"author": null,
"author_email": "Kai Chen <chenkaidev@gmail.com>, Zhiyuan Chen <this@zyc.ai>",
"download_url": "https://files.pythonhosted.org/packages/d5/44/3b1b91ec67f50000363d95871f1fc24a84c39c221c060b21db2a83f92fb3/icrawler-0.6.10.tar.gz",
"platform": null,
"description": "icrawler\n========\n\n.. image:: https://img.shields.io/pypi/v/icrawler.svg\n :target: https://pypi.python.org/pypi/icrawler\n :alt: PyPI Version\n\n.. image:: https://anaconda.org/hellock/icrawler/badges/version.svg\n :target: https://anaconda.org/hellock/icrawler\n :alt: Anaconda Version\n\n.. image:: https://img.shields.io/pypi/pyversions/icrawler.svg\n :alt: Python Version\n\n.. image:: \thttps://img.shields.io/github/license/hellock/icrawler.svg\n :alt: License\n\nIntroduction\n------------\n\nDocumentation: http://icrawler.readthedocs.io/\n\nTry it with ``pip install icrawler`` or ``conda install -c hellock icrawler``.\n\nThis package is a mini framework of web crawlers. With modularization design,\nit is easy to use and extend. It supports media data like images and videos\nvery well, and can also be applied to texts and other type of files.\nScrapy is heavy and powerful, while icrawler is tiny and flexible.\n\nWith this package, you can write a multiple thread crawler easily by focusing on\nthe contents you want to crawl, keeping away from troublesome problems like\nexception handling, thread scheduling and communication.\n\nIt also provides built-in crawlers for popular image sites like **Flickr** and\nsearch engines such as **Google**, **Bing** and **Baidu**.\n(Thank all the contributors and pull requests are always welcome!)\n\nRequirements\n------------\n\nPython 3.5+ (recommended).\n\nExamples\n--------\n\nUsing built-in crawlers is very simple. A minimal example is shown as follows.\n\n.. code:: python\n\n from icrawler.builtin import GoogleImageCrawler\n\n google_crawler = GoogleImageCrawler(storage={'root_dir': 'your_image_dir'})\n google_crawler.crawl(keyword='cat', max_num=100)\n\nYou can also configurate number of threads and apply advanced search options.\n(Note: compatible with 0.6.0 and later versions)\n\n.. code:: python\n\n from icrawler.builtin import GoogleImageCrawler\n\n google_crawler = GoogleImageCrawler(\n feeder_threads=1,\n parser_threads=2,\n downloader_threads=4,\n storage={'root_dir': 'your_image_dir'})\n filters = dict(\n size='large',\n color='orange',\n license='commercial,modify',\n date=((2017, 1, 1), (2017, 11, 30)))\n google_crawler.crawl(keyword='cat', filters=filters, max_num=1000, file_idx_offset=0)\n\nFor more advanced usage about built-in crawlers, please refer to the\n`documentation <http://icrawler.readthedocs.io/en/latest/builtin.html>`_.\n\nWriting your own crawlers with this framework is also convenient, see the\n`tutorials <http://icrawler.readthedocs.io/en/latest/extend.html>`_.\n\nArchitecture\n------------\n\nA crawler consists of 3 main components (Feeder, Parser and Downloader),\nthey are connected with each other with FIFO queues. The workflow is shown in\nthe following figure.\n\n.. figure:: http://7xopqn.com1.z0.glb.clouddn.com/workflow.png\n :alt:\n\n- ``url_queue`` stores the url of pages which may contain images\n- ``task_queue`` stores the image url as well as any meta data you\n like, each element in the queue is a dictionary and must contain the\n field ``img_url``\n- Feeder puts page urls to ``url_queue``\n- Parser requests and parses the page, then extracts the image urls and\n puts them into ``task_queue``\n- Downloader gets tasks from ``task_queue`` and requests the images,\n then saves them in the given path.\n\nFeeder, parser and downloader are all thread pools, so you can specify the\nnumber of threads they use.\n",
"bugtrack_url": null,
"license": "The MIT License (MIT) Copyright (c) 2016 Kai Chen Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A multi-thread crawler framework with many builtin image crawlers provided.",
"version": "0.6.10",
"project_urls": {
"documentation": "https://icrawler.readthedocs.io/",
"homepage": "https://icrawler.readthedocs.io/",
"repository": "https://github.com/hellock/icrawler"
},
"split_keywords": [
"crawler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c1141d68f9d2b01955f4c4c63d378e0a331497055b4b96ec1d3a175222411544",
"md5": "53b25112934652230c2fc53372075fc3",
"sha256": "159883cb06dea3c6b665b35045dcbea9922e6532d0b3d7eaee3029a2c3864940"
},
"downloads": -1,
"filename": "icrawler-0.6.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "53b25112934652230c2fc53372075fc3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 36226,
"upload_time": "2025-02-11T14:09:27",
"upload_time_iso_8601": "2025-02-11T14:09:27.481795Z",
"url": "https://files.pythonhosted.org/packages/c1/14/1d68f9d2b01955f4c4c63d378e0a331497055b4b96ec1d3a175222411544/icrawler-0.6.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d5443b1b91ec67f50000363d95871f1fc24a84c39c221c060b21db2a83f92fb3",
"md5": "7622ebc41a065e7fe0697048c2f03991",
"sha256": "45d2f47ab5f022cdfe73395175453eac2e8e8822659f6147ed3fb82146715727"
},
"downloads": -1,
"filename": "icrawler-0.6.10.tar.gz",
"has_sig": false,
"md5_digest": "7622ebc41a065e7fe0697048c2f03991",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 40420,
"upload_time": "2025-02-11T14:09:29",
"upload_time_iso_8601": "2025-02-11T14:09:29.787726Z",
"url": "https://files.pythonhosted.org/packages/d5/44/3b1b91ec67f50000363d95871f1fc24a84c39c221c060b21db2a83f92fb3/icrawler-0.6.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-11 14:09:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hellock",
"github_project": "icrawler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "beautifulsoup4",
"specs": [
[
">=",
"4.4.1"
]
]
},
{
"name": "lxml",
"specs": []
},
{
"name": "Pillow",
"specs": []
},
{
"name": "requests",
"specs": [
[
">=",
"2.9.1"
]
]
},
{
"name": "six",
"specs": [
[
">=",
"1.10.0"
]
]
}
],
"tox": true,
"lcname": "icrawler"
}
