# FPX
Standalone service for collecting content from multiple source into single
file. Typical usecase is downloading multiple files as archive using single
link. Internally FPX fetches content from the specified set of URLs and streams
zip-compressed stream to the end users.
# Installation
1. Install `fpx` package
```sh
pip install fpx
```
1. Initialize DB
```sh
fpx db up
```
1. Start FPX server
```sh
fpx server run
```
# Usage
## Authentication
Majority of FPX endpoints are available only via *client's secret*. It can be
generated via CLI command(replace `<CLIENT_NAME>` with an arbitrary combination
of letters, digits and underscores):
```sh
fpx client add <CLIENT_NAME>
```
And secret will be shown in the output of the command:
```sh
Client created: <CLIENT_NAME> - <SECRET>
```
Pass the secret via `Authorization` header with each request to identify
yourself as a client.
## Downloads
Downloading a file via FPX usually consists of two steps:
* Provide information about downloaded URLs and receive IDs of the *download ticket*
* Use the ticket's ID to download all URLs packed into a single ZIP archive
First step can be completed via cURL:
```sh
curl -X POST http://localhost:8080/ticket/generate \
-H "Authorization: <CLIENT_SECRET>" \
-d '{"items":["https://google.com", "https://google.com/search"]}'
```
Here we are making a POST request to `/ticket/generate` endpoint of FPX
service. It's requires client's secret, which is specified by `Authorization`
header. This endpoint works only with JSON requests, that's why we need
`Content-type`. Finally, body of the request must contain a JSON with an
`items` field: the list of all URLs we are going to download.
Response will be the following:
```sh
{"created":"2023-10-15T00:00:51.054523","type":"zip","id":"ca03e214-910d-419f-ad60-4b6fb8bdd10c"}
```
You need only `id` field from it. Use it to make a download URL:
`/ticket/<ID>/download`. For the example above we receive this URL:
`http://localhost:8080/ticket/ca03e214-910d-419f-ad60-4b6fb8bdd10c/download`.
Open it in web browser or use `wget`/`curl` to download file via CLI:
```sh
curl http://localhost:8080/ticket/ca03e214-910d-419f-ad60-4b6fb8bdd10c/download -o collection.zip
```
# Configuration
FPX works without explicit configuration, but default values are not suitable
for production environments. Config options can be changes via config file and
environment variables.
## Config file
FPX config file is a python script. It's read by FPX and all global variables
from it are used as config options. For example, the following file will add
`A` and `B` options to FPX application:
```python
A = 1
B = ["hello", "world"]
```
Path to this file must be specified via `FPX_CONFIG` environment variable:
```
export FPX_CONFIG=/etc/fpx/config/fpx.py
fpx server run
```
## Environment variables
In addition to config file, FPX reads all environment variables with `FPX_*`
name, strips `FPX_` prefix and use result as a config option. I.e:
* `FPX_DB_URL` envvar turns into `DB_URL` config option
* `FPX_FPX_TRANSPORT` envvar turns into `FPX_TRANSPORT` config option.
Pay attention to config options with the name starting with `FPX_`. Because
`FPX_` prefix is removed from envvars, you have to repeat it twice, like in
`FPX_FPX_TRANSPORT` above.
## Config options
FPX makes use of the following config options
| Name | Description | Default |
|-----------------|------------------------------------------------------------------------------------|-------------------------|
| `DEBUG` | Run application in debug mode. Mainly used for development | false |
| `HOST` | Bind application to the specified addres | 0.0.0.0 |
| `PORT` | Run application on the specified port | 8000 |
| `DB_URL` | DB URL used for SQLAlchemy engine | `sqlite:////tmp/fpx.db` |
| `FPX_TRANSPORT` | Underlying library for HTTP requests. `aiohttp` and `htmx` supported at the moment | `aiohttp` |
# Complete Installation Guide
1. Install FPX:
```sh
pip install fpx
```
1. Create config file. It can be created anywhere, as long as it accessible by FPX service:
```sh
echo '
PORT = 12321
DB_URL = "sqlite:////home/user/.virtualenvs/fpx/fpx.db"
' > /etc/fpx/fpx.py
```
1. Initialize database and create access token for client:
```sh
export FPX_CONFIG=/etc/fpx/fpx.py
fpx db up
fpx client add my-first-fpx-client # use any name, that match `[\w_-]`
```
Make sure, db is accessible and writable by FPX service. This
manual suggests using `www-data` user when configuring supervisor's
process, so following command required:
```sh
chown www-data:www-data /home/user/.virtualenvs/fpx/fpx.db
```
1. Test service:
```sh
FPX_CONFIG=/etc/fpx/fpx.py fpx server run
# or, if you want to explicitely use python interpreter
FPX_CONFIG=/etc/fpx/fpx.py python -m fpx
```
1. Configure system.d/supervisor/etc. unit for fpx. Make sure, that
`fpx server run` command, that spins up the service is executed using
python>=3.6 (`pyenv shell 3.8.2`). And, if SQLite is used, fpx
process has write access to db file:
```ini
[program:fpx-worker]
; Use the full paths to the virtualenv and your configuration file here.
command=/home/user/.virtualenv/fpx/bin/python -m fpx
environment=FPX_CONFIG=/etc/fpx/fpx.py
; User the worker runs as.
user=www-data
; Start just a single worker. Increase this number if you have many or
; particularly long running background jobs.
numprocs=1
process_name=%(program_name)s-%(process_num)02d
; Log files.
stdout_logfile=/var/log/fpx-worker.log
stderr_logfile=/var/log/fpx-worker.log
; Make sure that the worker is started on system start and automatically
; restarted if it crashes unexpectedly.
autostart=true
autorestart=true
; Number of seconds the process has to run before it is considered to have
; started successfully.
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
```
1. FPX service must be available via public url. As written in
[documentation](https://sanic.dev/en/guide/deployment/running.html#running-sanic),
no additional layers required. But if you decide to use it with Nginx, the
[following
link](https://sanic.dev/en/guide/deployment/nginx.html#proxied-sanic-app)
may be useful. Note, if `FPX_NO_QUEUE` config option is set to `False`, FPX
is using websockets (and it can affect configuration).
Example of Nginx section for FPX:
```conf
location /fpx/ {
proxy_pass http://127.0.0.1:12321/;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_buffering off;
# When FPX_NO_QUEUE option set to `False`
proxy_set_header connection "upgrade";
proxy_set_header upgrade $http_upgrade;
# In emergency comment out line to force caching
# proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
}
```
Example of httpd configuration:
```cond
# mod_proxy
# mod_proxy_http
ProxyPass /fpx/ http://0.0.0.0:8000/
ProxyPassReverse /fpx/ http://0.0.0.0:8000/
# When FPX_NO_QUEUE option set to `False`
# mod_proxy_wstunnel
# mod_rewrite
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule /fpx/(.*) ws://0.0.0.0:8000/$1 [P]
```
Raw data
{
"_id": null,
"home_page": "https://github.com/DataShades/fpx",
"name": "fpx",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Sergey Motornyuk",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/96/cb/2e2ced4bfab8e28d5b2449c84c0800884de03541c64377cb7241348c7014/fpx-0.6.2.tar.gz",
"platform": null,
"description": "# FPX\n\nStandalone service for collecting content from multiple source into single\nfile. Typical usecase is downloading multiple files as archive using single\nlink. Internally FPX fetches content from the specified set of URLs and streams\nzip-compressed stream to the end users.\n\n# Installation\n\n1. Install `fpx` package\n ```sh\n pip install fpx\n ```\n1. Initialize DB\n ```sh\n fpx db up\n ```\n1. Start FPX server\n ```sh\n fpx server run\n ```\n\n# Usage\n\n## Authentication\n\nMajority of FPX endpoints are available only via *client's secret*. It can be\ngenerated via CLI command(replace `<CLIENT_NAME>` with an arbitrary combination\nof letters, digits and underscores):\n\n```sh\nfpx client add <CLIENT_NAME>\n```\n\nAnd secret will be shown in the output of the command:\n\n```sh\nClient created: <CLIENT_NAME> - <SECRET>\n```\n\nPass the secret via `Authorization` header with each request to identify\nyourself as a client.\n\n## Downloads\n\nDownloading a file via FPX usually consists of two steps:\n\n* Provide information about downloaded URLs and receive IDs of the *download ticket*\n* Use the ticket's ID to download all URLs packed into a single ZIP archive\n\nFirst step can be completed via cURL:\n\n```sh\ncurl -X POST http://localhost:8080/ticket/generate \\\n -H \"Authorization: <CLIENT_SECRET>\" \\\n -d '{\"items\":[\"https://google.com\", \"https://google.com/search\"]}'\n```\n\nHere we are making a POST request to `/ticket/generate` endpoint of FPX\nservice. It's requires client's secret, which is specified by `Authorization`\nheader. This endpoint works only with JSON requests, that's why we need\n`Content-type`. Finally, body of the request must contain a JSON with an\n`items` field: the list of all URLs we are going to download.\n\nResponse will be the following:\n```sh\n{\"created\":\"2023-10-15T00:00:51.054523\",\"type\":\"zip\",\"id\":\"ca03e214-910d-419f-ad60-4b6fb8bdd10c\"}\n```\n\nYou need only `id` field from it. Use it to make a download URL:\n`/ticket/<ID>/download`. For the example above we receive this URL:\n`http://localhost:8080/ticket/ca03e214-910d-419f-ad60-4b6fb8bdd10c/download`.\n\nOpen it in web browser or use `wget`/`curl` to download file via CLI:\n\n```sh\ncurl http://localhost:8080/ticket/ca03e214-910d-419f-ad60-4b6fb8bdd10c/download -o collection.zip\n```\n\n# Configuration\n\nFPX works without explicit configuration, but default values are not suitable\nfor production environments. Config options can be changes via config file and\nenvironment variables.\n\n## Config file\n\nFPX config file is a python script. It's read by FPX and all global variables\nfrom it are used as config options. For example, the following file will add\n`A` and `B` options to FPX application:\n\n```python\nA = 1\nB = [\"hello\", \"world\"]\n```\n\nPath to this file must be specified via `FPX_CONFIG` environment variable:\n\n```\nexport FPX_CONFIG=/etc/fpx/config/fpx.py\nfpx server run\n```\n\n## Environment variables\n\nIn addition to config file, FPX reads all environment variables with `FPX_*`\nname, strips `FPX_` prefix and use result as a config option. I.e:\n* `FPX_DB_URL` envvar turns into `DB_URL` config option\n* `FPX_FPX_TRANSPORT` envvar turns into `FPX_TRANSPORT` config option.\n\nPay attention to config options with the name starting with `FPX_`. Because\n`FPX_` prefix is removed from envvars, you have to repeat it twice, like in\n`FPX_FPX_TRANSPORT` above.\n\n## Config options\n\nFPX makes use of the following config options\n\n| Name | Description | Default |\n|-----------------|------------------------------------------------------------------------------------|-------------------------|\n| `DEBUG` | Run application in debug mode. Mainly used for development | false |\n| `HOST` | Bind application to the specified addres | 0.0.0.0 |\n| `PORT` | Run application on the specified port | 8000 |\n| `DB_URL` | DB URL used for SQLAlchemy engine | `sqlite:////tmp/fpx.db` |\n| `FPX_TRANSPORT` | Underlying library for HTTP requests. `aiohttp` and `htmx` supported at the moment | `aiohttp` |\n\n# Complete Installation Guide\n\n1. Install FPX:\n ```sh\n pip install fpx\n ```\n\n1. Create config file. It can be created anywhere, as long as it accessible by FPX service:\n ```sh\n echo '\n PORT = 12321\n DB_URL = \"sqlite:////home/user/.virtualenvs/fpx/fpx.db\"\n ' > /etc/fpx/fpx.py\n ```\n\n1. Initialize database and create access token for client:\n ```sh\n export FPX_CONFIG=/etc/fpx/fpx.py\n fpx db up\n fpx client add my-first-fpx-client # use any name, that match `[\\w_-]`\n ```\n\n Make sure, db is accessible and writable by FPX service. This\n manual suggests using `www-data` user when configuring supervisor's\n process, so following command required:\n ```sh\n chown www-data:www-data /home/user/.virtualenvs/fpx/fpx.db\n ```\n\n1. Test service:\n ```sh\n FPX_CONFIG=/etc/fpx/fpx.py fpx server run\n # or, if you want to explicitely use python interpreter\n FPX_CONFIG=/etc/fpx/fpx.py python -m fpx\n ```\n\n1. Configure system.d/supervisor/etc. unit for fpx. Make sure, that\n `fpx server run` command, that spins up the service is executed using\n python>=3.6 (`pyenv shell 3.8.2`). And, if SQLite is used, fpx\n process has write access to db file:\n ```ini\n [program:fpx-worker]\n\n ; Use the full paths to the virtualenv and your configuration file here.\n command=/home/user/.virtualenv/fpx/bin/python -m fpx\n\n environment=FPX_CONFIG=/etc/fpx/fpx.py\n\n ; User the worker runs as.\n user=www-data\n\n ; Start just a single worker. Increase this number if you have many or\n ; particularly long running background jobs.\n numprocs=1\n process_name=%(program_name)s-%(process_num)02d\n\n ; Log files.\n stdout_logfile=/var/log/fpx-worker.log\n stderr_logfile=/var/log/fpx-worker.log\n\n ; Make sure that the worker is started on system start and automatically\n ; restarted if it crashes unexpectedly.\n autostart=true\n autorestart=true\n\n ; Number of seconds the process has to run before it is considered to have\n ; started successfully.\n startsecs=10\n\n ; Need to wait for currently executing tasks to finish at shutdown.\n ; Increase this if you have very long running tasks.\n stopwaitsecs = 600\n ```\n\n1. FPX service must be available via public url. As written in\n [documentation](https://sanic.dev/en/guide/deployment/running.html#running-sanic),\n no additional layers required. But if you decide to use it with Nginx, the\n [following\n link](https://sanic.dev/en/guide/deployment/nginx.html#proxied-sanic-app)\n may be useful. Note, if `FPX_NO_QUEUE` config option is set to `False`, FPX\n is using websockets (and it can affect configuration).\n\n Example of Nginx section for FPX:\n ```conf\n location /fpx/ {\n proxy_pass http://127.0.0.1:12321/;\n proxy_set_header X-Forwarded-For $remote_addr;\n proxy_set_header Host $host;\n proxy_http_version 1.1;\n proxy_request_buffering off;\n proxy_buffering off;\n\n # When FPX_NO_QUEUE option set to `False`\n proxy_set_header connection \"upgrade\";\n proxy_set_header upgrade $http_upgrade;\n\n # In emergency comment out line to force caching\n # proxy_ignore_headers X-Accel-Expires Expires Cache-Control;\n }\n ```\n\n Example of httpd configuration:\n ```cond\n # mod_proxy\n # mod_proxy_http\n ProxyPass /fpx/ http://0.0.0.0:8000/\n ProxyPassReverse /fpx/ http://0.0.0.0:8000/\n\n # When FPX_NO_QUEUE option set to `False`\n # mod_proxy_wstunnel\n # mod_rewrite\n RewriteEngine on\n RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]\n RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]\n RewriteRule /fpx/(.*) ws://0.0.0.0:8000/$1 [P]\n ```\n",
"bugtrack_url": null,
"license": "AGPL",
"summary": "Stream archiver/proxy",
"version": "0.6.2",
"project_urls": {
"Homepage": "https://github.com/DataShades/fpx"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "85576325d713b184b22ac5900399cc8001ad99977441bb20f9dc573d7b0fa0a5",
"md5": "3d6ed8f2b552f4a24162c8ccc2729548",
"sha256": "96d6145adf0ef1a467c1efa2785b21bc35b08894edc0010cfcac3cc0bb76d436"
},
"downloads": -1,
"filename": "fpx-0.6.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3d6ed8f2b552f4a24162c8ccc2729548",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 34182,
"upload_time": "2024-04-28T12:53:46",
"upload_time_iso_8601": "2024-04-28T12:53:46.770890Z",
"url": "https://files.pythonhosted.org/packages/85/57/6325d713b184b22ac5900399cc8001ad99977441bb20f9dc573d7b0fa0a5/fpx-0.6.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "96cb2e2ced4bfab8e28d5b2449c84c0800884de03541c64377cb7241348c7014",
"md5": "0dcdda1509d0c98f5f67ba03395dcd81",
"sha256": "ece93c071c859054b51862ed20e8300b520b98c1ea66288df590900b25601da1"
},
"downloads": -1,
"filename": "fpx-0.6.2.tar.gz",
"has_sig": false,
"md5_digest": "0dcdda1509d0c98f5f67ba03395dcd81",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 29218,
"upload_time": "2024-04-28T12:53:49",
"upload_time_iso_8601": "2024-04-28T12:53:49.148005Z",
"url": "https://files.pythonhosted.org/packages/96/cb/2e2ced4bfab8e28d5b2449c84c0800884de03541c64377cb7241348c7014/fpx-0.6.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-28 12:53:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DataShades",
"github_project": "fpx",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "fpx"
}
JSON
