# Newscatcher
Programmatically collect normalized news from (almost) any website.
By [newscatcherapi.com][1].
## Demo
## Motivation
While working on newscatcherapi -- JSON API to query the news articles,
I came up with an idea to make a simple Python package that would allow
to easily grab the live news data.
When I used to be a junior data scientist working on my own side projects,
it was difficult for me to operate with external data sources. I knew Python
quite well, but in most cases it was not enough to build proper data pipelines
that required gathering data on my own.
Even though I do not recommend to use this package for any production systems, I believe that it should be enough to test your assumptions and build some MVPs.
## Installation
`pip install newscatcher`
## Tech/framework used
The package itself is nothing more than a SQLite database with
RSS feed endpoints for each website and some basic wrapper of
[feedparser][2].
## Code Example/Documentation
Let's review all possible usage of the package.
In its core, it has a class called *Newscatcher*. This class is all you need in order to get latest news.
After installing your package, import the class:
`from newscatcher import Newscatcher`
Now you just need to put a url of a desired news source as an input into our class.
**Please take the base form url of a website** (without `www.`,neither `https://`, nor `/` at the end of url).
For example: “nytimes”.com, “news.ycombinator.com” or “theverge.com”.
`news_source = Newscatcher('blackfaldslife.com')`
If you have done it right and the source that you chose is presented in our database, you will get a variable with 3 components and 1 method:
- `news_source.website` -- the same string that you entered inside the class.
- `news_source.news` -- a list of a feedparser dictionary with latest news presented on the website.
- `news_source.headlines` -- a list with latest headlines presented on the website.
- `news_source.print_headlines()` -- print headlines of all latest articles.
Each element of *news* list is a json object with all relevant and available information regarding an article. If you want to know more about the attributes that you can extract from this json, go check the official documentation of feedparser following this link: [feedparser\_attributes][3]. You can find everything that begins with *entries[i]*. But be aware that not all the attributes are provided by the news website.
If for some reason you do not like classes, you can always import 2 main methods and use them separately.
`from newscatcher import get_news`
`news = get_news('wired.co.uk')`
or
`from newscatcher import get_headlines`
`news = get_headlines('wired.co.uk')`
## Licence
MIT
[1]: %3Chttps://newscatcherapi.com/%3E
[2]: %3Chttps://pythonhosted.org/feedparser/index.html%3E
[3]: %3Chttps://pythonhosted.org/feedparser/reference.html%3E
Raw data
{
"_id": null,
"home_page": "https://www.newscatcherapi.com",
"name": "newsgather",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7,<4.0",
"maintainer_email": "",
"keywords": "News,RSS,Scraping,Data Mining",
"author": "Artem Bugara",
"author_email": "bugara.artem@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/02/95/96ddc105b1ecd6b6620ed1a1e7447e70f2e584a010d34cb56f36a2e99d08/newsgather-0.2.0.tar.gz",
"platform": null,
"description": "# Newscatcher\nProgrammatically collect normalized news from (almost) any website.\nBy [newscatcherapi.com][1].\n\n## Demo\n\n\n## Motivation\nWhile working on newscatcherapi -- JSON API to query the news articles,\nI came up with an idea to make a simple Python package that would allow\nto easily grab the live news data. \n\nWhen I used to be a junior data scientist working on my own side projects,\nit was difficult for me to operate with external data sources. I knew Python\nquite well, but in most cases it was not enough to build proper data pipelines\nthat required gathering data on my own. \n\nEven though I do not recommend to use this package for any production systems, I believe that it should be enough to test your assumptions and build some MVPs.\n\n## Installation\n`pip install newscatcher`\n\n## Tech/framework used\nThe package itself is nothing more than a SQLite database with \nRSS feed endpoints for each website and some basic wrapper of\n[feedparser][2].\n\n## Code Example/Documentation\nLet's review all possible usage of the package. \n\nIn its core, it has a class called *Newscatcher*. This class is all you need in order to get latest news.\n\nAfter installing your package, import the class:\n\n`from newscatcher import Newscatcher`\n \nNow you just need to put a url of a desired news source as an input into our class. \n**Please take the base form url of a website** (without `www.`,neither `https://`, nor `/` at the end of url).\n\nFor example: \u201cnytimes\u201d.com, \u201cnews.ycombinator.com\u201d or \u201ctheverge.com\u201d.\n\n`news_source = Newscatcher('blackfaldslife.com')`\n\nIf you have done it right and the source that you chose is presented in our database, you will get a variable with 3 components and 1 method:\n\n- `news_source.website` -- the same string that you entered inside the class.\n- `news_source.news` -- a list of a feedparser dictionary with latest news presented on the website. \n- `news_source.headlines` -- a list with latest headlines presented on the website.\n- `news_source.print_headlines()` -- print headlines of all latest articles.\n\nEach element of *news* list is a json object with all relevant and available information regarding an article. If you want to know more about the attributes that you can extract from this json, go check the official documentation of feedparser following this link: [feedparser\\_attributes][3]. You can find everything that begins with *entries[i]*. But be aware that not all the attributes are provided by the news website. \n\nIf for some reason you do not like classes, you can always import 2 main methods and use them separately.\n\n`from newscatcher import get_news`\n`news = get_news('wired.co.uk')`\n\nor\n\n`from newscatcher import get_headlines`\n`news = get_headlines('wired.co.uk')`\n\n\n## Licence\nMIT\n\n\n[1]:\t%3Chttps://newscatcherapi.com/%3E\n[2]:\t%3Chttps://pythonhosted.org/feedparser/index.html%3E\n[3]:\t%3Chttps://pythonhosted.org/feedparser/reference.html%3E\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Get the normalized latest news from (almost) any website",
"version": "0.2.0",
"split_keywords": [
"news",
"rss",
"scraping",
"data mining"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1d1c10d6ebdf15181eded878f2e664cf80a6caf176471c951ff32edf3654afc4",
"md5": "6f73f28581c32a80b8473c0625adbd28",
"sha256": "019e6a8fabcd46d6f44e2aa30099b666ff96dd3fcf25cb84c96b4b9b4d80d077"
},
"downloads": -1,
"filename": "newsgather-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6f73f28581c32a80b8473c0625adbd28",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7,<4.0",
"size": 233917,
"upload_time": "2023-01-27T04:59:29",
"upload_time_iso_8601": "2023-01-27T04:59:29.366606Z",
"url": "https://files.pythonhosted.org/packages/1d/1c/10d6ebdf15181eded878f2e664cf80a6caf176471c951ff32edf3654afc4/newsgather-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "029596ddc105b1ecd6b6620ed1a1e7447e70f2e584a010d34cb56f36a2e99d08",
"md5": "0cc00725db5e298d24332db8441926d4",
"sha256": "d7a7eb167a8a6b7cda9d8751639ff1f56d333b9842f25ac7b1bb44184ba69f1d"
},
"downloads": -1,
"filename": "newsgather-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "0cc00725db5e298d24332db8441926d4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7,<4.0",
"size": 233105,
"upload_time": "2023-01-27T04:59:31",
"upload_time_iso_8601": "2023-01-27T04:59:31.563249Z",
"url": "https://files.pythonhosted.org/packages/02/95/96ddc105b1ecd6b6620ed1a1e7447e70f2e584a010d34cb56f36a2e99d08/newsgather-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-27 04:59:31",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "newsgather"
}
JSON