| Name | fundus JSON |
| Version |
0.4.6
JSON |
| download |
| home_page | None |
| Summary | A very simple news crawler |
| upload_time | 2024-11-05 18:29:54 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT |
| keywords |
web scraping
web crawling
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_darkmode_with_font_and_clear_space.svg">
<source media="(prefers-color-scheme: light)" srcset="https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_lightmode_with_font_and_clear_space.svg">
<img src="https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_lightmode_with_font_and_clear_space.svg" alt="Logo" width="50%" height="50%">
</picture>
</p>
<p align="center">A very simple <b>news crawler</b> in Python.
Developed at <a href="https://www.informatik.hu-berlin.de/en/forschung-en/gebiete/ml-en/">Humboldt University of Berlin</a>.
</p>
<p align="center">
<a href="https://pypi.org/project/fundus/"><img alt="PyPi version" src="https://badge.fury.io/py/fundus.svg"></a>
<img alt="python" src="https://img.shields.io/badge/python-3.8-blue">
<img alt="Static Badge" src="https://img.shields.io/badge/license-MIT-green">
<img alt="Publisher Coverage" src="https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/dobbersc/ca0ae056b05cbfeaf30fa42f84ddf458/raw/fundus_publisher_coverage.json">
</p>
<div align="center">
<hr>
[Quick Start](#quick-start) | [Tutorials](#tutorials) | [News Sources](/docs/supported_publishers.md) | [Paper](https://aclanthology.org/2024.acl-demos.29/)
</div>
---
Fundus is:
* **A static news crawler.**
Fundus lets you crawl online news articles with only a few lines of Python code!
Be it from live websites or the CC-NEWS dataset.
* **An open-source Python package.**
Fundus is built on the idea of building something together.
We welcome your contribution to help Fundus [grow](docs/how_to_contribute.md)!
<hr>
## Quick Start
To install from pip, simply do:
```
pip install fundus
```
Fundus requires Python 3.8+.
## Example 1: Crawl a bunch of English-language news articles
Let's use Fundus to crawl 2 articles from publishers based in the US.
```python
from fundus import PublisherCollection, Crawler
# initialize the crawler for news publishers based in the US
crawler = Crawler(PublisherCollection.us)
# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
print(article)
```
That's already it!
If you run this code, it should print out something like this:
```console
Fundus-Article:
- Title: "Feinstein's Return Not Enough for Confirmation of Controversial New [...]"
- Text: "Democrats jammed three of President Joe Biden's controversial court nominees
through committee votes on Thursday thanks to a last-minute [...]"
- URL: https://freebeacon.com/politics/feinsteins-return-not-enough-for-confirmation-of-controversial-new-hampshire-judicial-nominee/
- From: FreeBeacon (2023-05-11 18:41)
Fundus-Article:
- Title: "Northwestern student government freezes College Republicans funding over [...]"
- Text: "Student government at Northwestern University in Illinois "indefinitely" froze
the funds of the university's chapter of College Republicans [...]"
- URL: https://www.foxnews.com/us/northwestern-student-government-freezes-college-republicans-funding-poster-critical-lgbtq-community
- From: FoxNews (2023-05-09 14:37)
```
This printout tells you that you successfully crawled two articles!
For each article, the printout details:
- the "Title" of the article, i.e. its headline
- the "Text", i.e. the main article body text
- the "URL" from which it was crawled
- the news source it is "From"
## Example 2: Crawl a specific news source
Maybe you want to crawl a specific news source instead. Let's crawl news articles from Washington Times only:
```python
from fundus import PublisherCollection, Crawler
# initialize the crawler for The New Yorker
crawler = Crawler(PublisherCollection.us.TheNewYorker)
# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
print(article)
```
## Example 3: Crawl 1 Million articles
To crawl such a vast amount of data, Fundus relies on the `CommonCrawl` web archive, in particular the news crawl `CC-NEWS`.
If you're not familiar with [`CommonCrawl`](https://commoncrawl.org/) or [`CC-NEWS`](https://commoncrawl.org/blog/news-dataset-available) check out their websites.
Simply import our `CCNewsCrawler` and make sure to check out our [tutorial](docs/2_crawl_from_cc_news.md) beforehand.
````python
from fundus import PublisherCollection, CCNewsCrawler
# initialize the crawler using all publishers supported by fundus
crawler = CCNewsCrawler(*PublisherCollection)
# crawl 1 million articles and print
for article in crawler.crawl(max_articles=1000000):
print(article)
````
**_Note_**: By default, the crawler utilizes all available CPU cores on your system.
For optimal performance, we recommend manually setting the number of processes using the `processes` parameter.
A good rule of thumb is to allocate `one process per 200 Mbps of bandwidth`.
This can vary depending on core speed.
**_Note_**: The crawl above took ~7 hours using the entire `PublisherCollection` on a machine with 1000 Mbps connection, Core i9-13905H, 64GB Ram, Windows 11 and without printing the articles.
The estimated time can vary substantially depending on the publisher used and the available bandwidth.
Additionally, not all publishers are included in the `CC-NEWS` crawl (especially US based publishers).
For large corpus creation, one can also use the regular crawler by utilizing only sitemaps, which requires significantly less bandwidth.
````python
from fundus import PublisherCollection, Crawler, Sitemap
# initialize a crawler for us/uk based publishers and restrict to Sitemaps only
crawler = Crawler(PublisherCollection.us, PublisherCollection.uk, restrict_sources_to=[Sitemap])
# crawl 1 million articles and print
for article in crawler.crawl(max_articles=1000000):
print(article)
````
## Tutorials
We provide **quick tutorials** to get you started with the library:
1. [**Tutorial 1: How to crawl news with Fundus**](docs/1_getting_started.md)
2. [**Tutorial 2: How to crawl articles from CC-NEWS**](docs/2_crawl_from_cc_news.md)
3. [**Tutorial 3: The Article Class**](docs/3_the_article_class.md)
4. [**Tutorial 4: How to filter articles**](docs/4_how_to_filter_articles.md)
5. [**Tutorial 5: Advanced topics**](docs/5_advanced_topics.md)
6. [**Tutorial 6: Logging**](docs/6_logging.md)
If you wish to contribute check out these tutorials:
1. [**How to contribute**](docs/how_to_contribute.md)
2. [**How to add a publisher**](docs/how_to_add_a_publisher.md)
## Currently Supported News Sources
You can find the publishers currently supported [**here**](/docs/supported_publishers.md).
Also: **Adding a new publisher is easy - consider contributing to the project!**
## Evaluation Benchmark
Check out our evaluation [benchmark](https://github.com/dobbersc/fundus-evaluation).
The following table summarizes the overall performance of Fundus and evaluated scrapers in terms of averaged ROUGE-LSum precision, recall and F1-score and their standard deviation. The table is sorted in descending order over the F1-score:
| **Scraper** | **Precision** | **Recall** | **F1-Score** | **Version** |
|-----------------------------------------------------------------------------------------------------------------|:--------------------------|---------------------------|---------------------------|-------------|
| [Fundus](https://github.com/flairNLP/fundus) | **99.89**<sub>±0.57</sub> | 96.75<sub>±12.75</sub> | **97.69**<sub>±9.75</sub> | 0.4.1 |
| [Trafilatura](https://github.com/adbar/trafilatura) | 93.91<sub>±12.89</sub> | 96.85<sub>±15.69</sub> | 93.62<sub>±16.73</sub> | 1.12.0 |
| [news-please](https://github.com/fhamborg/news-please) | 97.95<sub>±10.08</sub> | 91.89<sub>±16.15</sub> | 93.39<sub>±14.52</sub> | 1.6.13 |
| [BTE](https://github.com/dobbersc/fundus-evaluation/blob/master/src/fundus_evaluation/scrapers/bte.py) | 81.09<sub>±19.41</sub> | **98.23**<sub>±8.61</sub> | 87.14<sub>±15.48</sub> | / |
| [jusText](https://github.com/miso-belica/jusText) | 86.51<sub>±18.92</sub> | 90.23<sub>±20.61</sub> | 86.96<sub>±19.76</sub> | 3.0.1 |
| [BoilerNet](https://github.com/dobbersc/fundus-evaluation/tree/master/src/fundus_evaluation/scrapers/boilernet) | 85.96<sub>±18.55</sub> | 91.21<sub>±19.15</sub> | 86.52<sub>±18.03</sub> | / |
| [Boilerpipe](https://github.com/kohlschutter/boilerpipe) | 82.89<sub>±20.65</sub> | 82.11<sub>±29.99</sub> | 79.90<sub>±25.86</sub> | 1.3.0 |
## Cite
Please cite the following [paper](https://aclanthology.org/2024.acl-demos.29/) when using Fundus or building upon our work:
```bibtex
@inproceedings{dallabetta-etal-2024-fundus,
title = "Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions",
author = "Dallabetta, Max and
Dobberstein, Conrad and
Breiding, Adrian and
Akbik, Alan",
editor = "Cao, Yixin and
Feng, Yang and
Xiong, Deyi",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-demos.29",
pages = "305--314",
}
```
## Contact
Please email your questions or comments to [**Max Dallabetta**](mailto:max.dallabetta@googlemail.com?subject=[GitHub]%20Fundus)
## Contributing
Thanks for your interest in contributing! There are many ways to get involved;
start with our [contributor guidelines](docs/how_to_contribute.md) and then
check these [open issues](https://github.com/flairNLP/fundus/issues) for specific tasks.
## License
[MIT](LICENSE)
Raw data
{
"_id": null,
"home_page": null,
"name": "fundus",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "web scraping, web crawling",
"author": null,
"author_email": "Max Dallabetta <max.dallabetta@googlemail.com>",
"download_url": "https://files.pythonhosted.org/packages/fb/ee/e8c502176c99ea7083587645e3e2c62bb91cf9a4031aa497da251374049b/fundus-0.4.6.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <picture>\n <source media=\"(prefers-color-scheme: dark)\" srcset=\"https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_darkmode_with_font_and_clear_space.svg\">\n <source media=\"(prefers-color-scheme: light)\" srcset=\"https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_lightmode_with_font_and_clear_space.svg\">\n <img src=\"https://github.com/flairNLP/fundus/blob/master/resources/logo/svg/logo_lightmode_with_font_and_clear_space.svg\" alt=\"Logo\" width=\"50%\" height=\"50%\">\n </picture>\n</p>\n\n<p align=\"center\">A very simple <b>news crawler</b> in Python.\nDeveloped at <a href=\"https://www.informatik.hu-berlin.de/en/forschung-en/gebiete/ml-en/\">Humboldt University of Berlin</a>.\n</p>\n<p align=\"center\">\n<a href=\"https://pypi.org/project/fundus/\"><img alt=\"PyPi version\" src=\"https://badge.fury.io/py/fundus.svg\"></a>\n<img alt=\"python\" src=\"https://img.shields.io/badge/python-3.8-blue\">\n<img alt=\"Static Badge\" src=\"https://img.shields.io/badge/license-MIT-green\">\n<img alt=\"Publisher Coverage\" src=\"https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/dobbersc/ca0ae056b05cbfeaf30fa42f84ddf458/raw/fundus_publisher_coverage.json\">\n</p>\n<div align=\"center\">\n<hr>\n\n[Quick Start](#quick-start) | [Tutorials](#tutorials) | [News Sources](/docs/supported_publishers.md) | [Paper](https://aclanthology.org/2024.acl-demos.29/)\n\n</div>\n\n\n---\n\nFundus is:\n\n* **A static news crawler.** \n Fundus lets you crawl online news articles with only a few lines of Python code!\n Be it from live websites or the CC-NEWS dataset.\n\n* **An open-source Python package.**\n Fundus is built on the idea of building something together. \n We welcome your contribution to help Fundus [grow](docs/how_to_contribute.md)!\n\n<hr>\n\n## Quick Start\n\nTo install from pip, simply do:\n\n```\npip install fundus\n```\n\nFundus requires Python 3.8+.\n\n\n## Example 1: Crawl a bunch of English-language news articles\n\nLet's use Fundus to crawl 2 articles from publishers based in the US.\n\n```python\nfrom fundus import PublisherCollection, Crawler\n\n# initialize the crawler for news publishers based in the US\ncrawler = Crawler(PublisherCollection.us)\n\n# crawl 2 articles and print\nfor article in crawler.crawl(max_articles=2):\n print(article)\n```\n\nThat's already it!\n\nIf you run this code, it should print out something like this:\n\n```console\nFundus-Article:\n- Title: \"Feinstein's Return Not Enough for Confirmation of Controversial New [...]\"\n- Text: \"Democrats jammed three of President Joe Biden's controversial court nominees\n through committee votes on Thursday thanks to a last-minute [...]\"\n- URL: https://freebeacon.com/politics/feinsteins-return-not-enough-for-confirmation-of-controversial-new-hampshire-judicial-nominee/\n- From: FreeBeacon (2023-05-11 18:41)\n\nFundus-Article:\n- Title: \"Northwestern student government freezes College Republicans funding over [...]\"\n- Text: \"Student government at Northwestern University in Illinois \"indefinitely\" froze\n the funds of the university's chapter of College Republicans [...]\"\n- URL: https://www.foxnews.com/us/northwestern-student-government-freezes-college-republicans-funding-poster-critical-lgbtq-community\n- From: FoxNews (2023-05-09 14:37)\n```\n\nThis printout tells you that you successfully crawled two articles!\n\nFor each article, the printout details:\n- the \"Title\" of the article, i.e. its headline \n- the \"Text\", i.e. the main article body text\n- the \"URL\" from which it was crawled\n- the news source it is \"From\"\n\n\n## Example 2: Crawl a specific news source\n\nMaybe you want to crawl a specific news source instead. Let's crawl news articles from Washington Times only:\n\n```python\nfrom fundus import PublisherCollection, Crawler\n\n# initialize the crawler for The New Yorker\ncrawler = Crawler(PublisherCollection.us.TheNewYorker)\n\n# crawl 2 articles and print\nfor article in crawler.crawl(max_articles=2):\n print(article)\n```\n\n## Example 3: Crawl 1 Million articles\n\nTo crawl such a vast amount of data, Fundus relies on the `CommonCrawl` web archive, in particular the news crawl `CC-NEWS`.\nIf you're not familiar with [`CommonCrawl`](https://commoncrawl.org/) or [`CC-NEWS`](https://commoncrawl.org/blog/news-dataset-available) check out their websites.\nSimply import our `CCNewsCrawler` and make sure to check out our [tutorial](docs/2_crawl_from_cc_news.md) beforehand.\n\n````python\nfrom fundus import PublisherCollection, CCNewsCrawler\n\n# initialize the crawler using all publishers supported by fundus\ncrawler = CCNewsCrawler(*PublisherCollection)\n\n# crawl 1 million articles and print\nfor article in crawler.crawl(max_articles=1000000):\n print(article)\n````\n\n**_Note_**: By default, the crawler utilizes all available CPU cores on your system. \nFor optimal performance, we recommend manually setting the number of processes using the `processes` parameter. \nA good rule of thumb is to allocate `one process per 200 Mbps of bandwidth`.\nThis can vary depending on core speed.\n\n**_Note_**: The crawl above took ~7 hours using the entire `PublisherCollection` on a machine with 1000 Mbps connection, Core i9-13905H, 64GB Ram, Windows 11 and without printing the articles.\nThe estimated time can vary substantially depending on the publisher used and the available bandwidth.\nAdditionally, not all publishers are included in the `CC-NEWS` crawl (especially US based publishers).\nFor large corpus creation, one can also use the regular crawler by utilizing only sitemaps, which requires significantly less bandwidth.\n\n````python\nfrom fundus import PublisherCollection, Crawler, Sitemap\n\n# initialize a crawler for us/uk based publishers and restrict to Sitemaps only\ncrawler = Crawler(PublisherCollection.us, PublisherCollection.uk, restrict_sources_to=[Sitemap])\n\n# crawl 1 million articles and print\nfor article in crawler.crawl(max_articles=1000000):\n print(article)\n````\n\n\n## Tutorials\n\nWe provide **quick tutorials** to get you started with the library:\n\n1. [**Tutorial 1: How to crawl news with Fundus**](docs/1_getting_started.md)\n2. [**Tutorial 2: How to crawl articles from CC-NEWS**](docs/2_crawl_from_cc_news.md)\n3. [**Tutorial 3: The Article Class**](docs/3_the_article_class.md)\n4. [**Tutorial 4: How to filter articles**](docs/4_how_to_filter_articles.md)\n5. [**Tutorial 5: Advanced topics**](docs/5_advanced_topics.md)\n6. [**Tutorial 6: Logging**](docs/6_logging.md)\n\nIf you wish to contribute check out these tutorials:\n1. [**How to contribute**](docs/how_to_contribute.md)\n2. [**How to add a publisher**](docs/how_to_add_a_publisher.md)\n\n## Currently Supported News Sources\n\nYou can find the publishers currently supported [**here**](/docs/supported_publishers.md).\n\nAlso: **Adding a new publisher is easy - consider contributing to the project!**\n\n## Evaluation Benchmark\n\nCheck out our evaluation [benchmark](https://github.com/dobbersc/fundus-evaluation).\n\nThe following table summarizes the overall performance of Fundus and evaluated scrapers in terms of averaged ROUGE-LSum precision, recall and F1-score and their standard deviation. The table is sorted in descending order over the F1-score:\n\n| **Scraper** | **Precision** | **Recall** | **F1-Score** | **Version** |\n|-----------------------------------------------------------------------------------------------------------------|:--------------------------|---------------------------|---------------------------|-------------|\n| [Fundus](https://github.com/flairNLP/fundus) | **99.89**<sub>\u00b10.57</sub> | 96.75<sub>\u00b112.75</sub> | **97.69**<sub>\u00b19.75</sub> | 0.4.1 |\n| [Trafilatura](https://github.com/adbar/trafilatura) | 93.91<sub>\u00b112.89</sub> | 96.85<sub>\u00b115.69</sub> | 93.62<sub>\u00b116.73</sub> | 1.12.0 |\n| [news-please](https://github.com/fhamborg/news-please) | 97.95<sub>\u00b110.08</sub> | 91.89<sub>\u00b116.15</sub> | 93.39<sub>\u00b114.52</sub> | 1.6.13 |\n| [BTE](https://github.com/dobbersc/fundus-evaluation/blob/master/src/fundus_evaluation/scrapers/bte.py) | 81.09<sub>\u00b119.41</sub> | **98.23**<sub>\u00b18.61</sub> | 87.14<sub>\u00b115.48</sub> | / |\n| [jusText](https://github.com/miso-belica/jusText) | 86.51<sub>\u00b118.92</sub> | 90.23<sub>\u00b120.61</sub> | 86.96<sub>\u00b119.76</sub> | 3.0.1 |\n| [BoilerNet](https://github.com/dobbersc/fundus-evaluation/tree/master/src/fundus_evaluation/scrapers/boilernet) | 85.96<sub>\u00b118.55</sub> | 91.21<sub>\u00b119.15</sub> | 86.52<sub>\u00b118.03</sub> | / |\n| [Boilerpipe](https://github.com/kohlschutter/boilerpipe) | 82.89<sub>\u00b120.65</sub> | 82.11<sub>\u00b129.99</sub> | 79.90<sub>\u00b125.86</sub> | 1.3.0 |\n\n## Cite\n\nPlease cite the following [paper](https://aclanthology.org/2024.acl-demos.29/) when using Fundus or building upon our work:\n\n```bibtex\n@inproceedings{dallabetta-etal-2024-fundus,\n title = \"Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions\",\n author = \"Dallabetta, Max and\n Dobberstein, Conrad and\n Breiding, Adrian and\n Akbik, Alan\",\n editor = \"Cao, Yixin and\n Feng, Yang and\n Xiong, Deyi\",\n booktitle = \"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)\",\n month = aug,\n year = \"2024\",\n address = \"Bangkok, Thailand\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2024.acl-demos.29\",\n pages = \"305--314\",\n}\n```\n\n## Contact\n\nPlease email your questions or comments to [**Max Dallabetta**](mailto:max.dallabetta@googlemail.com?subject=[GitHub]%20Fundus)\n\n## Contributing\n\nThanks for your interest in contributing! There are many ways to get involved;\nstart with our [contributor guidelines](docs/how_to_contribute.md) and then\ncheck these [open issues](https://github.com/flairNLP/fundus/issues) for specific tasks.\n\n## License\n\n[MIT](LICENSE)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A very simple news crawler",
"version": "0.4.6",
"project_urls": {
"Repository": "https://github.com/flairNLP/fundus"
},
"split_keywords": [
"web scraping",
" web crawling"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c115d35f774df69e538e0dd3f34c724d42f912b30e9d8b9cfc83627cda5883f5",
"md5": "9f8a85f4367da50d1305bea9969207bd",
"sha256": "f99373efeab837c455e7a2dcc6306db93c2447803c19ef3cf638d9d934488531"
},
"downloads": -1,
"filename": "fundus-0.4.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9f8a85f4367da50d1305bea9969207bd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 149889,
"upload_time": "2024-11-05T18:29:52",
"upload_time_iso_8601": "2024-11-05T18:29:52.027713Z",
"url": "https://files.pythonhosted.org/packages/c1/15/d35f774df69e538e0dd3f34c724d42f912b30e9d8b9cfc83627cda5883f5/fundus-0.4.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fbeee8c502176c99ea7083587645e3e2c62bb91cf9a4031aa497da251374049b",
"md5": "61718f6b54d38d2c7d8bebba9979e411",
"sha256": "279b6fe3eafe4b16d68055f9b817745d66cf929e02ed5ee3ae2529c60d88f7d6"
},
"downloads": -1,
"filename": "fundus-0.4.6.tar.gz",
"has_sig": false,
"md5_digest": "61718f6b54d38d2c7d8bebba9979e411",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 86564,
"upload_time": "2024-11-05T18:29:54",
"upload_time_iso_8601": "2024-11-05T18:29:54.156000Z",
"url": "https://files.pythonhosted.org/packages/fb/ee/e8c502176c99ea7083587645e3e2c62bb91cf9a4031aa497da251374049b/fundus-0.4.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-05 18:29:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "flairNLP",
"github_project": "fundus",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "fundus"
}
