| Name | morss JSON |
| Version |
20230627.2039
JSON |
| download |
| home_page | http://morss.it/ |
| Summary | Get full-text RSS feeds |
| upload_time | 2023-06-27 20:40:04 |
| maintainer | |
| docs_url | None |
| author | pictuga |
| requires_python | >=2.7 |
| license | AGPL v3 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Morss - Get full-text RSS feeds
[Homepage](https://morss.it/) •
[Upstream source code](https://git.pictuga.com/pictuga/morss) •
[Github mirror](https://github.com/pictuga/morss) (for Issues & Pull requests)
[](https://ci.pictuga.com/pictuga/morss)
[](https://github.com/pictuga/morss/stargazers)
[](https://github.com/pictuga/morss/network/members)
[](https://git.pictuga.com/pictuga/morss/src/branch/master/LICENSE)
[](https://creativecommons.org/licenses/by-nc-sa/4.0/)
This tool's goal is to get full-text RSS feeds out of striped RSS feeds,
commonly available on internet. Indeed most newspapers only make a small
description available to users in their rss feeds, which makes the RSS feed
rather useless. So this tool intends to fix that problem.
This tool opens the links from the rss feed, then downloads the full article
from the newspaper website and puts it back in the rss feed.
Morss also provides additional features, such as: .csv and json export, extended
control over output. A strength of morss is its ability to deal with broken
feeds, and to replace tracking links with direct links to the actual content.
Morss can also generate feeds from html and json files (see `feeds.py`), which
for instance makes it possible to get feeds for Facebook or Twitter, using
hand-written rules (ie. there's no automatic detection of links to build feeds).
Please mind that feeds based on html files may stop working unexpectedly, due to
html structure changes on the target website.
Additionally morss can detect rss feeds in html pages' `<meta>`.
You can use this program online for free at **[morss.it](https://morss.it/)**.
Some features of morss:
- Read RSS/Atom feeds
- Create RSS feeds from json/html pages
- Export feeds as RSS/JSON/CSV/HTML
- Fetch full-text content of feed items
- Follow 301/meta redirects
- Recover xml feeds with corrupt encoding
- Supports gzip-compressed http content
- HTTP caching with different backends (in-memory/redis/diskcache)
- Works as server/cli tool
- Deobfuscate various tracking links
## Install
### Python package
].status&url=https://ci.pictuga.com/api/repos/pictuga/morss/builds/latest)
[](https://pypi.org/project/morss/)
[](https://pypistats.org/packages/morss)
Simple install (without optional dependencies)
From pip
```shell
pip install morss
```
From git
```shell
pip install git+https://git.pictuga.com/pictuga/morss.git
```
Full installation (including optional dependencies)
From pip
```shell
pip install morss[full]
```
From git
```shell
pip install git+https://git.pictuga.com/pictuga/morss.git#egg=morss[full]
```
The full install includes all the cache backends. Otherwise, only in-memory
cache is available. The full install also includes gunicorn (for more efficient
HTTP handling).
The dependency `lxml` is fairly long to install (especially on Raspberry Pi, as
C code needs to be compiled). If possible on your distribution, try installing
it with the system package manager.
### Docker
].status&url=https://ci.pictuga.com/api/repos/pictuga/morss/builds/latest)
[](https://hub.docker.com/r/pictuga/morss)
[](https://hub.docker.com/r/pictuga/morss/tags)
From docker hub
With cli
```shell
docker pull pictuga/morss
```
With docker-compose **(recommended)**
```yml
services:
app:
image: pictuga/morss
ports:
- '8000:8000'
```
Build from source
With cli
```shell
docker build --tag morss https://git.pictuga.com/pictuga/morss.git --no-cache --pull
```
With docker-compose
```yml
services:
app:
build: https://git.pictuga.com/pictuga/morss.git
image: morss
ports:
- '8000:8000'
```
Then execute
```shell
docker-compose build --no-cache --pull
```
### Cloud providers
One-click deployment:
[](https://heroku.com/deploy?template=https://github.com/pictuga/morss)
[](https://deploy.cloud.run/?git_repo=https://github.com/pictuga/morss.git)
Providers supporting `cloud-init` (AWS, Oracle Cloud Infrastructure), based on Ubuntu:
``` yml
#cloud-config
packages:
- python3-pip
- python3-wheel
- python3-lxml
- python3-setproctitle
- ca-certificates
write_files:
- path: /etc/environment
append: true
content: |
DEBUG=1
CACHE=diskcache
CACHE_SIZE=1073741824 # 1GiB
- path: /var/lib/cloud/scripts/per-boot/morss.sh
permissions: 744
content: |
#!/bin/sh
/usr/local/bin/morss-helper daemon
runcmd:
- source /etc/environment
- update-ca-certificates
- iptables -I INPUT 6 -m state --state NEW -p tcp --dport ${PORT:-8000} -j ACCEPT
- netfilter-persistent save
- pip install morss[full]
```
## Run
morss will auto-detect what "mode" to use.
### Running on/as a server
Set up the server as indicated below, then visit:
```
http://PATH/TO/MORSS/[main.py/][:argwithoutvalue[:argwithvalue=value[...]]]/FEEDURL
```
For example: `http://morss.example/:clip/https://twitter.com/pictuga`
*(Brackets indicate optional text)*
The `main.py` part is only needed if your server doesn't support the Apache
redirect rule set in the provided `.htaccess`.
Works like a charm with [Tiny Tiny RSS](https://tt-rss.org/), and most probably
other clients.
#### Using Docker
From docker hub
```shell
docker run -p 8000:8000 pictuga/morss
```
From source
```shell
docker run -p 8000:8000 morss
```
With docker-compose **(recommended)**
```shell
docker-compose up
```
#### Using Gunicorn
```shell
gunicorn --preload morss
```
#### Using uWSGI
Running this command should do:
```shell
uwsgi --http :8000 --plugin python --wsgi-file main.py
```
#### Using morss' internal HTTP server
Morss can run its own, **very basic**, HTTP server, meant for debugging mostly.
The latter should start when you run morss without any argument, on port 8000.
I'd highly recommend you to use gunicorn or something similar for better
performance.
```shell
morss
```
You can change the port using environment variables like this `PORT=9000 morss`.
#### Via mod_cgi/FastCGI with Apache/nginx
For this, you'll want to change a bit the architecture of the files, for example
into something like this.
```
/
├── cgi
│ │
│ ├── main.py
│ ├── morss
│ │ ├── __init__.py
│ │ ├── __main__.py
│ │ ├── morss.py
│ │ └── ...
│ │
│ ├── dateutil
│ └── ...
│
├── .htaccess
├── index.html
└── ...
```
For this, you need to make sure your host allows python script execution. This
method uses HTTP calls to fetch the RSS feeds, which will be handled through
`mod_cgi` for example on Apache severs.
Please pay attention to `main.py` permissions for it to be executable. See below
some tips for the `.htaccess` file.
```htaccess
Options -Indexes
ErrorDocument 404 /cgi/main.py
# Turn debug on for all requests
SetEnv DEBUG 1
# Turn debug on for requests with :debug in the url
SetEnvIf Request_URI :debug DEBUG=1
<Files ~ "\.(py|pyc|db|log)$">
deny from all
</Files>
<Files main.py>
allow from all
AddHandler cgi-script .py
Options +ExecCGI
</Files>
```
### As a CLI application
Run:
```
morss [--argwithoutvalue] [--argwithvalue=value] [...] FEEDURL
```
For example: `morss --clip http://feeds.bbci.co.uk/news/rss.xml`
*(Brackets indicate optional text)*
If using Docker:
```shell
docker run morss --clip http://feeds.bbci.co.uk/news/rss.xml
```
### As a newsreader hook
To use it, the newsreader [Liferea](http://lzone.de/liferea/) is required
(unless other newsreaders provide the same kind of feature), since custom
scripts can be run on top of the RSS feed, using its
[output](http://lzone.de/liferea/scraping.htm) as an RSS feed.
To use this script, you have to enable "(Unix) command" in liferea feed
settings, and use the command:
```
morss [--argwithoutvalue] [--argwithvalue=value] [...] FEEDURL
```
For example: `morss http://feeds.bbci.co.uk/news/rss.xml`
*(Brackets indicate optional text)*
### As a python library
Quickly get a full-text feed:
```python
>>> import morss
>>> xml_string = morss.process('http://feeds.bbci.co.uk/news/rss.xml')
>>> xml_string[:50]
"<?xml version='1.0' encoding='UTF-8'?>\n<?xml-style"
```
Using cache and passing arguments:
```python
>>> import morss
>>> url = 'http://feeds.bbci.co.uk/news/rss.xml'
>>> cache = '/tmp/morss-cache' # diskcache cache location
>>> options = {'csv':True}
>>> xml_string = morss.process(url, cache, options)
>>> xml_string[:50]
'{"title": "BBC News - Home", "desc": "The latest s'
```
`morss.process` is actually a wrapper around simpler function. It's still
possible to call the simpler functions, to have more control on what's happening
under the hood.
Doing it step-by-step:
```python
import morss
url = 'http://newspaper.example/feed.xml'
options = morss.Options(csv=True) # arguments
url, rss = morss.FeedFetch(url, options) # this only grabs the RSS feed
rss = morss.FeedGather(rss, url, options) # this fills the feed and cleans it up
output = morss.FeedFormat(rss, options, 'unicode') # formats final feed
```
## Arguments and settings
### Arguments
morss accepts some arguments, to lightly alter the output of morss. Arguments
may need to have a value (usually a string or a number). How to pass those
arguments to morss is explained in Run above.
The list of arguments can be obtained by running `morss --help`
```
usage: morss [-h] [--post STRING] [--xpath XPATH]
[--format {rss,json,html,csv}] [--search STRING] [--clip]
[--indent] [--cache] [--force] [--proxy]
[--order {first,last,newest,oldest}] [--firstlink] [--resolve]
[--items XPATH] [--item_link XPATH] [--item_title XPATH]
[--item_content XPATH] [--item_time XPATH]
[--mode {xml,html,json}] [--nolink] [--noref] [--silent]
url
Get full-text RSS feeds
positional arguments:
url feed url
options:
-h, --help show this help message and exit
--post STRING POST request
--xpath XPATH xpath rule to manually detect the article
output:
--format {rss,json,html,csv}
output format
--search STRING does a basic case-sensitive search in the feed
--clip stick the full article content under the original feed
content (useful for twitter)
--indent returns indented XML or JSON, takes more place, but
human-readable
action:
--cache only take articles from the cache (ie. don't grab new
articles' content), so as to save time
--force force refetch the rss feed and articles
--proxy doesn't fill the articles
--order {first,last,newest,oldest}
order in which to process items (which are however NOT
sorted in the output)
--firstlink pull the first article mentioned in the description
instead of the default link
--resolve replace tracking links with direct links to articles
(not compatible with --proxy)
custom feeds:
--items XPATH (mandatory to activate the custom feeds function)
xpath rule to match all the RSS entries
--item_link XPATH xpath rule relative to items to point to the entry's
link
--item_title XPATH entry's title
--item_content XPATH entry's content
--item_time XPATH entry's date & time (accepts a wide range of time
formats)
--mode {xml,html,json}
parser to use for the custom feeds
misc:
--nolink drop links, but keeps links' inner text
--noref drop items' link
--silent don't output the final RSS (useless on its own, but
can be nice when debugging)
GNU AGPLv3 code
```
Further HTTP-only options:
- `callback=NAME`: for JSONP calls
- `cors`: allow Cross-origin resource sharing (allows XHR calls from other
servers)
- `txt`: changes the http content-type to txt (for faster "`view-source:`")
### Environment variables
To pass environment variables:
- Docker-cli: `docker run -p 8000:8000 morss --env KEY=value`
- docker-compose: add an `environment:` section in the .yml file
- Gunicorn/uWSGI/CLI: prepend `KEY=value` before the command
- Apache: via the `SetEnv` instruction (see sample `.htaccess` provided)
- cloud-init: in the `/etc/environment` file
Generic:
- `DEBUG=1`: to have some feedback from the script execution. Useful for
debugging.
- `IGNORE_SSL=1`: to ignore SSL certs when fetch feeds and articles
- `DELAY` (seconds) sets the browser cache delay, only for HTTP clients
- `TIMEOUT` (seconds) sets the HTTP timeout when fetching rss feeds and articles
- `DATA_PATH`: to set custom file location for the `www` folder
When parsing long feeds, with a lot of items (100+), morss might take a lot of
time to parse it, or might even run into a memory overflow on some shared
hosting plans (limits around 10Mb), in which case you might want to adjust the
below settings via environment variables.
Also, if the request takes too long to process, the http request might be
discarded. See relevant config for
[gunicorn](https://docs.gunicorn.org/en/stable/settings.html#timeout) or
[nginx](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout).
- `MAX_TIME` (seconds) sets the maximum amount of time spent *fetching*
articles, more time might be spent taking older articles from cache. `-1` for
unlimited.
- `MAX_ITEM` sets the maximum number of articles to fetch. `-1` for unlimited.
More articles will be taken from cache following the nexts settings.
- `LIM_TIME` (seconds) sets the maximum amount of time spent working on the feed
(whether or not it's already cached). Articles beyond that limit will be dropped
from the feed. `-1` for unlimited.
- `LIM_ITEM` sets the maximum number of article checked, limiting both the
number of articles fetched and taken from cache. Articles beyond that limit will
be dropped from the feed, even if they're cached. `-1` for unlimited.
morss uses caching to make loading faster. There are 3 possible cache backends:
- `(nothing/default)`: a simple python in-memory dict-like object.
- `CACHE=redis`: Redis cache. Connection can be defined with the following
environment variables: `REDIS_HOST`, `REDIS_PORT`, `REDIS_DB`, `REDIS_PWD`
- `CACHE=diskcache`: disk-based cache. Target directory canbe defined with
`DISKCACHE_DIR`.
To limit the size of the cache:
- `CACHE_SIZE` sets the target number of items in the cache (further items will
be deleted but the cache might be temporarily bigger than that). Defaults to 1k
entries. NB. When using `diskcache`, this is the cache max size in Bytes.
- `CACHE_LIFESPAN` (seconds) sets how often the cache must be trimmed (i.e. cut
down to the number of items set in `CACHE_SIZE`). Defaults to 1min.
Gunicorn also accepts command line arguments via the `GUNICORN_CMD_ARGS`
environment variable.
### Content matching
The content of articles is grabbed with our own readability fork. This means
that most of the time the right content is matched. However sometimes it fails,
therefore some tweaking is required. Most of the time, what has to be done is to
add some "rules" in the main script file in `readabilite.py` (not in morss).
Most of the time when hardly nothing is matched, it means that the main content
of the article is made of images, videos, pictures, etc., which readability
doesn't detect. Also, readability has some trouble to match content of very
small articles.
morss will also try to figure out whether the full content is already in place
(for those websites which understood the whole point of RSS feeds). However this
detection is very simple, and only works if the actual content is put in the
"content" section in the feed and not in the "summary" section.
Raw data
{
"_id": null,
"home_page": "http://morss.it/",
"name": "morss",
"maintainer": "",
"docs_url": null,
"requires_python": ">=2.7",
"maintainer_email": "",
"keywords": "",
"author": "pictuga",
"author_email": "contact@pictuga.com",
"download_url": "https://files.pythonhosted.org/packages/62/46/02acd5c236f6dffe33f705ea0c9ad6d4cba7df77153cf675b661b7cc544d/morss-20230627.2039.tar.gz",
"platform": null,
"description": "# Morss - Get full-text RSS feeds\n\n[Homepage](https://morss.it/) \u2022 \n[Upstream source code](https://git.pictuga.com/pictuga/morss) \u2022 \n[Github mirror](https://github.com/pictuga/morss) (for Issues & Pull requests)\n\n[](https://ci.pictuga.com/pictuga/morss)\n[](https://github.com/pictuga/morss/stargazers)\n[](https://github.com/pictuga/morss/network/members)\n[](https://git.pictuga.com/pictuga/morss/src/branch/master/LICENSE)\n[](https://creativecommons.org/licenses/by-nc-sa/4.0/)\n\nThis tool's goal is to get full-text RSS feeds out of striped RSS feeds,\ncommonly available on internet. Indeed most newspapers only make a small\ndescription available to users in their rss feeds, which makes the RSS feed\nrather useless. So this tool intends to fix that problem.\n\nThis tool opens the links from the rss feed, then downloads the full article\nfrom the newspaper website and puts it back in the rss feed.\n\nMorss also provides additional features, such as: .csv and json export, extended\ncontrol over output. A strength of morss is its ability to deal with broken\nfeeds, and to replace tracking links with direct links to the actual content.\n\nMorss can also generate feeds from html and json files (see `feeds.py`), which\nfor instance makes it possible to get feeds for Facebook or Twitter, using\nhand-written rules (ie. there's no automatic detection of links to build feeds).\nPlease mind that feeds based on html files may stop working unexpectedly, due to\nhtml structure changes on the target website.\n\nAdditionally morss can detect rss feeds in html pages' `<meta>`.\n\nYou can use this program online for free at **[morss.it](https://morss.it/)**.\n\nSome features of morss:\n\n- Read RSS/Atom feeds\n- Create RSS feeds from json/html pages\n- Export feeds as RSS/JSON/CSV/HTML\n- Fetch full-text content of feed items\n- Follow 301/meta redirects\n- Recover xml feeds with corrupt encoding\n- Supports gzip-compressed http content\n- HTTP caching with different backends (in-memory/redis/diskcache)\n- Works as server/cli tool\n- Deobfuscate various tracking links\n\n## Install\n\n### Python package\n\n].status&url=https://ci.pictuga.com/api/repos/pictuga/morss/builds/latest)\n[](https://pypi.org/project/morss/)\n[](https://pypistats.org/packages/morss)\n\nSimple install (without optional dependencies)\n\nFrom pip\n\n```shell\npip install morss\n```\n\nFrom git\n\n```shell\npip install git+https://git.pictuga.com/pictuga/morss.git\n```\n\nFull installation (including optional dependencies)\n\nFrom pip\n\n```shell\npip install morss[full]\n```\n\nFrom git\n\n```shell\npip install git+https://git.pictuga.com/pictuga/morss.git#egg=morss[full]\n```\n\nThe full install includes all the cache backends. Otherwise, only in-memory\ncache is available. The full install also includes gunicorn (for more efficient\nHTTP handling).\n\nThe dependency `lxml` is fairly long to install (especially on Raspberry Pi, as\nC code needs to be compiled). If possible on your distribution, try installing\nit with the system package manager.\n\n### Docker\n\n].status&url=https://ci.pictuga.com/api/repos/pictuga/morss/builds/latest)\n[](https://hub.docker.com/r/pictuga/morss)\n[](https://hub.docker.com/r/pictuga/morss/tags)\n\nFrom docker hub\n\nWith cli\n\n```shell\ndocker pull pictuga/morss\n```\n\nWith docker-compose **(recommended)**\n\n```yml\nservices:\n app:\n image: pictuga/morss\n ports:\n - '8000:8000'\n```\n\nBuild from source\n\nWith cli\n\n```shell\ndocker build --tag morss https://git.pictuga.com/pictuga/morss.git --no-cache --pull\n```\n\nWith docker-compose\n\n```yml\nservices:\n app:\n build: https://git.pictuga.com/pictuga/morss.git\n image: morss\n ports:\n - '8000:8000'\n```\n\nThen execute\n\n```shell\ndocker-compose build --no-cache --pull\n```\n\n### Cloud providers\n\nOne-click deployment:\n\n[](https://heroku.com/deploy?template=https://github.com/pictuga/morss)\n[](https://deploy.cloud.run/?git_repo=https://github.com/pictuga/morss.git)\n\nProviders supporting `cloud-init` (AWS, Oracle Cloud Infrastructure), based on Ubuntu:\n\n``` yml\n#cloud-config\n\npackages:\n - python3-pip\n - python3-wheel\n - python3-lxml\n - python3-setproctitle\n - ca-certificates\n\nwrite_files:\n - path: /etc/environment\n append: true\n content: |\n DEBUG=1\n CACHE=diskcache\n CACHE_SIZE=1073741824 # 1GiB\n - path: /var/lib/cloud/scripts/per-boot/morss.sh\n permissions: 744\n content: |\n #!/bin/sh\n /usr/local/bin/morss-helper daemon\n\nruncmd:\n - source /etc/environment\n - update-ca-certificates\n - iptables -I INPUT 6 -m state --state NEW -p tcp --dport ${PORT:-8000} -j ACCEPT\n - netfilter-persistent save\n - pip install morss[full]\n```\n\n## Run\n\nmorss will auto-detect what \"mode\" to use.\n\n### Running on/as a server\n\nSet up the server as indicated below, then visit:\n\n```\nhttp://PATH/TO/MORSS/[main.py/][:argwithoutvalue[:argwithvalue=value[...]]]/FEEDURL\n```\n\nFor example: `http://morss.example/:clip/https://twitter.com/pictuga`\n\n*(Brackets indicate optional text)*\n\nThe `main.py` part is only needed if your server doesn't support the Apache\nredirect rule set in the provided `.htaccess`.\n\nWorks like a charm with [Tiny Tiny RSS](https://tt-rss.org/), and most probably\nother clients.\n\n\n#### Using Docker\n\nFrom docker hub\n\n```shell\ndocker run -p 8000:8000 pictuga/morss\n```\n\nFrom source\n\n```shell\ndocker run -p 8000:8000 morss\n```\n\nWith docker-compose **(recommended)**\n\n```shell\ndocker-compose up\n```\n\n#### Using Gunicorn\n\n```shell\ngunicorn --preload morss\n```\n\n#### Using uWSGI\n\nRunning this command should do:\n\n```shell\nuwsgi --http :8000 --plugin python --wsgi-file main.py\n```\n\n#### Using morss' internal HTTP server\n\nMorss can run its own, **very basic**, HTTP server, meant for debugging mostly.\nThe latter should start when you run morss without any argument, on port 8000.\nI'd highly recommend you to use gunicorn or something similar for better\nperformance.\n\n```shell\nmorss\n```\n\nYou can change the port using environment variables like this `PORT=9000 morss`.\n\n#### Via mod_cgi/FastCGI with Apache/nginx\n\nFor this, you'll want to change a bit the architecture of the files, for example\ninto something like this.\n\n```\n/\n\u251c\u2500\u2500 cgi\n\u2502\u00a0\u00a0 \u2502\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 main.py\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 morss\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 __main__.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 morss.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 ...\n\u2502\u00a0\u00a0 \u2502\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 dateutil\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 ...\n\u2502\n\u251c\u2500\u2500 .htaccess\n\u251c\u2500\u2500 index.html\n\u2514\u2500\u2500 ...\n```\n\nFor this, you need to make sure your host allows python script execution. This\nmethod uses HTTP calls to fetch the RSS feeds, which will be handled through\n`mod_cgi` for example on Apache severs.\n\nPlease pay attention to `main.py` permissions for it to be executable. See below\nsome tips for the `.htaccess` file.\n\n```htaccess\nOptions -Indexes\n\nErrorDocument 404 /cgi/main.py\n\n# Turn debug on for all requests\nSetEnv DEBUG 1\n\n# Turn debug on for requests with :debug in the url\nSetEnvIf Request_URI :debug DEBUG=1\n\n<Files ~ \"\\.(py|pyc|db|log)$\">\n\tdeny from all\n</Files>\n\n<Files main.py>\n\tallow from all\n\tAddHandler cgi-script .py\n\tOptions +ExecCGI\n</Files>\n```\n\n### As a CLI application\n\nRun:\n\n```\nmorss [--argwithoutvalue] [--argwithvalue=value] [...] FEEDURL\n```\n\nFor example: `morss --clip http://feeds.bbci.co.uk/news/rss.xml`\n\n*(Brackets indicate optional text)*\n\nIf using Docker:\n\n```shell\ndocker run morss --clip http://feeds.bbci.co.uk/news/rss.xml\n```\n\n### As a newsreader hook\n\nTo use it, the newsreader [Liferea](http://lzone.de/liferea/) is required\n(unless other newsreaders provide the same kind of feature), since custom\nscripts can be run on top of the RSS feed, using its\n[output](http://lzone.de/liferea/scraping.htm) as an RSS feed.\n\nTo use this script, you have to enable \"(Unix) command\" in liferea feed\nsettings, and use the command:\n\n```\nmorss [--argwithoutvalue] [--argwithvalue=value] [...] FEEDURL\n```\n\nFor example: `morss http://feeds.bbci.co.uk/news/rss.xml`\n\n*(Brackets indicate optional text)*\n\n### As a python library\n\nQuickly get a full-text feed:\n\n```python\n>>> import morss\n>>> xml_string = morss.process('http://feeds.bbci.co.uk/news/rss.xml')\n>>> xml_string[:50]\n\"<?xml version='1.0' encoding='UTF-8'?>\\n<?xml-style\"\n```\n\nUsing cache and passing arguments:\n\n```python\n>>> import morss\n>>> url = 'http://feeds.bbci.co.uk/news/rss.xml'\n>>> cache = '/tmp/morss-cache' # diskcache cache location\n>>> options = {'csv':True}\n>>> xml_string = morss.process(url, cache, options)\n>>> xml_string[:50]\n'{\"title\": \"BBC News - Home\", \"desc\": \"The latest s'\n```\n\n`morss.process` is actually a wrapper around simpler function. It's still\npossible to call the simpler functions, to have more control on what's happening\nunder the hood.\n\nDoing it step-by-step:\n\n```python\nimport morss\n\nurl = 'http://newspaper.example/feed.xml'\noptions = morss.Options(csv=True) # arguments\n\nurl, rss = morss.FeedFetch(url, options) # this only grabs the RSS feed\nrss = morss.FeedGather(rss, url, options) # this fills the feed and cleans it up\n\noutput = morss.FeedFormat(rss, options, 'unicode') # formats final feed\n```\n\n## Arguments and settings\n\n### Arguments\n\nmorss accepts some arguments, to lightly alter the output of morss. Arguments\nmay need to have a value (usually a string or a number). How to pass those\narguments to morss is explained in Run above.\n\nThe list of arguments can be obtained by running `morss --help`\n\n```\nusage: morss [-h] [--post STRING] [--xpath XPATH]\n [--format {rss,json,html,csv}] [--search STRING] [--clip]\n [--indent] [--cache] [--force] [--proxy]\n [--order {first,last,newest,oldest}] [--firstlink] [--resolve]\n [--items XPATH] [--item_link XPATH] [--item_title XPATH]\n [--item_content XPATH] [--item_time XPATH]\n [--mode {xml,html,json}] [--nolink] [--noref] [--silent]\n url\n\nGet full-text RSS feeds\n\npositional arguments:\n url feed url\n\noptions:\n -h, --help show this help message and exit\n --post STRING POST request\n --xpath XPATH xpath rule to manually detect the article\n\noutput:\n --format {rss,json,html,csv}\n output format\n --search STRING does a basic case-sensitive search in the feed\n --clip stick the full article content under the original feed\n content (useful for twitter)\n --indent returns indented XML or JSON, takes more place, but\n human-readable\n\naction:\n --cache only take articles from the cache (ie. don't grab new\n articles' content), so as to save time\n --force force refetch the rss feed and articles\n --proxy doesn't fill the articles\n --order {first,last,newest,oldest}\n order in which to process items (which are however NOT\n sorted in the output)\n --firstlink pull the first article mentioned in the description\n instead of the default link\n --resolve replace tracking links with direct links to articles\n (not compatible with --proxy)\n\ncustom feeds:\n --items XPATH (mandatory to activate the custom feeds function)\n xpath rule to match all the RSS entries\n --item_link XPATH xpath rule relative to items to point to the entry's\n link\n --item_title XPATH entry's title\n --item_content XPATH entry's content\n --item_time XPATH entry's date & time (accepts a wide range of time\n formats)\n --mode {xml,html,json}\n parser to use for the custom feeds\n\nmisc:\n --nolink drop links, but keeps links' inner text\n --noref drop items' link\n --silent don't output the final RSS (useless on its own, but\n can be nice when debugging)\n\nGNU AGPLv3 code\n```\n\nFurther HTTP-only options:\n\n- `callback=NAME`: for JSONP calls\n- `cors`: allow Cross-origin resource sharing (allows XHR calls from other\nservers)\n- `txt`: changes the http content-type to txt (for faster \"`view-source:`\")\n\n### Environment variables\n\nTo pass environment variables:\n\n- Docker-cli: `docker run -p 8000:8000 morss --env KEY=value`\n- docker-compose: add an `environment:` section in the .yml file\n- Gunicorn/uWSGI/CLI: prepend `KEY=value` before the command\n- Apache: via the `SetEnv` instruction (see sample `.htaccess` provided)\n- cloud-init: in the `/etc/environment` file\n\nGeneric:\n\n- `DEBUG=1`: to have some feedback from the script execution. Useful for\ndebugging.\n- `IGNORE_SSL=1`: to ignore SSL certs when fetch feeds and articles\n- `DELAY` (seconds) sets the browser cache delay, only for HTTP clients\n- `TIMEOUT` (seconds) sets the HTTP timeout when fetching rss feeds and articles\n- `DATA_PATH`: to set custom file location for the `www` folder\n\nWhen parsing long feeds, with a lot of items (100+), morss might take a lot of\ntime to parse it, or might even run into a memory overflow on some shared\nhosting plans (limits around 10Mb), in which case you might want to adjust the\nbelow settings via environment variables.\n\nAlso, if the request takes too long to process, the http request might be\ndiscarded. See relevant config for\n[gunicorn](https://docs.gunicorn.org/en/stable/settings.html#timeout) or\n[nginx](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout).\n\n- `MAX_TIME` (seconds) sets the maximum amount of time spent *fetching*\narticles, more time might be spent taking older articles from cache. `-1` for\nunlimited.\n- `MAX_ITEM` sets the maximum number of articles to fetch. `-1` for unlimited.\nMore articles will be taken from cache following the nexts settings.\n- `LIM_TIME` (seconds) sets the maximum amount of time spent working on the feed\n(whether or not it's already cached). Articles beyond that limit will be dropped\nfrom the feed. `-1` for unlimited.\n- `LIM_ITEM` sets the maximum number of article checked, limiting both the\nnumber of articles fetched and taken from cache. Articles beyond that limit will\nbe dropped from the feed, even if they're cached. `-1` for unlimited.\n\nmorss uses caching to make loading faster. There are 3 possible cache backends:\n\n- `(nothing/default)`: a simple python in-memory dict-like object.\n- `CACHE=redis`: Redis cache. Connection can be defined with the following\nenvironment variables: `REDIS_HOST`, `REDIS_PORT`, `REDIS_DB`, `REDIS_PWD`\n- `CACHE=diskcache`: disk-based cache. Target directory canbe defined with\n`DISKCACHE_DIR`.\n\nTo limit the size of the cache:\n\n- `CACHE_SIZE` sets the target number of items in the cache (further items will\nbe deleted but the cache might be temporarily bigger than that). Defaults to 1k\nentries. NB. When using `diskcache`, this is the cache max size in Bytes.\n- `CACHE_LIFESPAN` (seconds) sets how often the cache must be trimmed (i.e. cut\ndown to the number of items set in `CACHE_SIZE`). Defaults to 1min.\n\nGunicorn also accepts command line arguments via the `GUNICORN_CMD_ARGS`\nenvironment variable.\n\n### Content matching\n\nThe content of articles is grabbed with our own readability fork. This means\nthat most of the time the right content is matched. However sometimes it fails,\ntherefore some tweaking is required. Most of the time, what has to be done is to\nadd some \"rules\" in the main script file in `readabilite.py` (not in morss).\n\nMost of the time when hardly nothing is matched, it means that the main content\nof the article is made of images, videos, pictures, etc., which readability\ndoesn't detect. Also, readability has some trouble to match content of very\nsmall articles.\n\nmorss will also try to figure out whether the full content is already in place\n(for those websites which understood the whole point of RSS feeds). However this\ndetection is very simple, and only works if the actual content is put in the\n\"content\" section in the feed and not in the \"summary\" section.\n",
"bugtrack_url": null,
"license": "AGPL v3",
"summary": "Get full-text RSS feeds",
"version": "20230627.2039",
"project_urls": {
"Bug Tracker": "https://github.com/pictuga/morss/issues",
"Homepage": "http://morss.it/",
"Source": "https://git.pictuga.com/pictuga/morss"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "66bdfd7023e0aa6a42d09dc8ec59f001a49ff40f1950246a5f0b32d588e37271",
"md5": "3ffddab6f1488023419aee9a313d6474",
"sha256": "ff5af02043628c4f2be86c15f76d515e6b2ccb989b62dfb4efbee0774a99d4e4"
},
"downloads": -1,
"filename": "morss-20230627.2039-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3ffddab6f1488023419aee9a313d6474",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=2.7",
"size": 75898,
"upload_time": "2023-06-27T20:40:02",
"upload_time_iso_8601": "2023-06-27T20:40:02.329820Z",
"url": "https://files.pythonhosted.org/packages/66/bd/fd7023e0aa6a42d09dc8ec59f001a49ff40f1950246a5f0b32d588e37271/morss-20230627.2039-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "624602acd5c236f6dffe33f705ea0c9ad6d4cba7df77153cf675b661b7cc544d",
"md5": "90afba788f88ee6d7c4206f1c9fecd7b",
"sha256": "c401ac2921d81f1d2f8ec32ef107c574effb4b89905c42d51a306a69288c7c3c"
},
"downloads": -1,
"filename": "morss-20230627.2039.tar.gz",
"has_sig": false,
"md5_digest": "90afba788f88ee6d7c4206f1c9fecd7b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.7",
"size": 56071,
"upload_time": "2023-06-27T20:40:04",
"upload_time_iso_8601": "2023-06-27T20:40:04.701543Z",
"url": "https://files.pythonhosted.org/packages/62/46/02acd5c236f6dffe33f705ea0c9ad6d4cba7df77153cf675b661b7cc544d/morss-20230627.2039.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-27 20:40:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pictuga",
"github_project": "morss",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "morss"
}
