# Fink Alert Broker Module for the TOM Toolkit
[](https://pypi.python.org/pypi/tom-fink)
[](https://www.codacy.com/gh/TOMToolkit/tom_fink/dashboard?utm_source=github.com&utm_medium=referral&utm_content=TOMToolkit/tom_fink&utm_campaign=Badge_Grade)
[](https://coveralls.io/github/TOMToolkit/tom_fink?branch=main)
This repository hosts the Fink Alert Broker Module for the TOM Toolkit. Fink is a broker currently connected to ZTF, and that will handle the massive alert stream from LSST. More information on Fink at https://fink-broker.org/. You can easily install the module using pip:
```bash
pip install tom-fink
```
## Polling data from the REST API
As of `tom-fink` version 0.4, the module simply uses the [Fink REST API](https://fink-broker.readthedocs.io/en/latest/services/science_portal/) to retrieve alerts. Note that the Fink databases are updated **once a day** with the previous night alert data (hence you do not get live alerts for the moment). Users can perform:
- Search by ZTF object ID
- Cone Search
- Search by Date
- Search by derived alert class
- Search by Solar System name
You need to declare the `FinkBroker` class in your running TOM instance. To do so just add `tom_fink.fink.FinkBroker` to the `TOM_ALERT_CLASSES` in your TOM's `settings.py`:
```python
TOM_ALERT_CLASSES = [
'tom_alerts.brokers.alerce.ALeRCEBroker',
...,
'tom_fink.fink.FinkBroker'
]
```
and finally relaunch your TOM:
```bash
./manage.py runserver
```
## Polling data from the Fink livestream service
Alternatively from `tom_fink` version 0.6.0, users can poll Fink substreams in real-time, using the [livestream service](https://fink-broker.readthedocs.io/en/latest/services/livestream/). First you need to add `tom_alertstreams` and `tom_fink` to your `INSTALLED_APPS` setting in your project `settings.py`:
```python
INSTALLED_APPS = [
...
'tom_alertstreams',
'tom_fink'
]
```
Then add the following block to your `settings.py`:
```python
ALERT_STREAMS = [
{
'ACTIVE': True,
'NAME': 'tom_fink.alertstream.FinkAlertStream',
'OPTIONS': {
'URL': os.getenv('FINK_CREDENTIAL_URL', 'set FINK_CREDENTIAL_URL value in environment'),
'USERNAME': os.getenv('FINK_CREDENTIAL_USERNAME', 'set FINK_CREDENTIAL_USERNAME value in environment'),
'GROUP_ID': os.getenv('FINK_CREDENTIAL_GROUP_ID', 'set FINK_CREDENTIAL_GROUP_ID value in environment'),
'TOPIC': os.getenv('FINK_TOPIC', 'set FINK_TOPIC value in environment'),
'MAX_POLL_NUMBER': os.getenv("FINK_MAX_POLL_NUMBER", 1e10),
'TIMEOUT': os.getenv('FINK_TIMEOUT', 10),
'TOPIC_HANDLERS': {
'fink.stream': 'tom_fink.alertstream.alert_logger',
},
},
},
]
```
and declare the following variables in your `~/.bash_profile`:
```bash
export FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream
export FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream
export FINK_CREDENTIAL_URL= # provided when registering to Fink livestream
export FINK_TOPIC= # str, topic name to poll
export FINK_MAX_POLL_NUMBER= # int, number maximum of alerts to poll before closing the connection. Default is 1e10 (never stops).
export FINK_TIMEOUT= # int, in seconds. Default is 10 seconds if not set
```
Credentials are provided when registering to Fink livestream (see https://fink-broker.readthedocs.io/en/latest/services/livestream). The available topic names and description can be found at https://fink-broker.readthedocs.io/en/latest/services/livestream. Note that we currently allow to poll only one topic at once (multi-topic is planned for later). Then launch the `readstreams` service:
```bash
./manage.py readstreams
```
You should see a log similar to:
```bash
registering new views: args: ('groups', <class 'tom_common.api_views.GroupViewSet'>, 'groups'), kwargs: {}
registering new views: args: ('targets', <class 'tom_targets.api_views.TargetViewSet'>, 'targets'), kwargs: {}
registering new views: args: ('targetextra', <class 'tom_targets.api_views.TargetExtraViewSet'>, 'targetextra'), kwargs: {}
registering new views: args: ('targetname', <class 'tom_targets.api_views.TargetNameViewSet'>, 'targetname'), kwargs: {}
registering new views: args: ('targetlist', <class 'tom_targets.api_views.TargetListViewSet'>, 'targetlist'), kwargs: {}
registering new views: args: ('observations', <class 'tom_observations.api_views.ObservationRecordViewSet'>, 'observations'), kwargs: {}
registering new views: args: ('dataproducts', <class 'tom_dataproducts.api_views.DataProductViewSet'>, 'dataproducts'), kwargs: {}
registering new views: args: ('reduceddatums', <class 'tom_dataproducts.api_views.ReducedDatumViewSet'>, 'reduceddatums'), kwargs: {}
NumExpr defaulting to 8 threads.
FinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 0)
read_streams FinkAlertStream TID=378166 ; thread identifier=125271444649536
readstreams Command.handle() returning...
fink.alert_logger topic: fink_sso_ztf_candidates_ztf
fink.alert_logger value: ZTF24aakwfsz emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 10:45:01)
FinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 1)
fink.alert_logger topic: fink_sso_ztf_candidates_ztf
fink.alert_logger value: ZTF24aakwftv emitted 2024-04-23 07:27:39.003823 (received 2024-04-29 10:45:01)
...
```
Each new alert will be saved as a `Target`. If more than one alert with the name `objectId` are emitted, you will be warned:
```bash
fink.alert_logger value: ZTF24aakwfsu emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 08:18:07)
Target ZTF24aakwfsu already in the database
```
and the program will continue. Probably we should group these in the future, instead of ignoring subsequent alerts. Then launch the app (do not close the previous process!):
```bash
./manage.py runserver
```
and you should have access to your targets, that are accessible from their topic name:
### Testing & debugging the connection
Before running in production, we advise to make tests using a test stream, and polling a few alerts:
```bash
export FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream
export FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream
export FINK_CREDENTIAL_URL= # provided when registering to Fink livestream
export FINK_TOPIC=fink_sso_ztf_candidates_ztf
export FINK_MAX_POLL_NUMBER=2
export FINK_TIMEOUT=10
```
This way, you will poll alerts two-by-two, and you will have a reservoir of thousands of alerts (solar system objects)!
## Todo list
- [ ] Add a test suite (preferably running on GitHub Actions)
- [x] Add a linter (preferably running on GitHub Actions)
- [x] Update the Query Form with all API features
- [x] Enable querying live streams using the Fink Kafka client
- [x] Update visualisation of targets from the livestream service
Raw data
{
"_id": null,
"home_page": "https://github.com/TOMToolkit/tom_fink",
"name": "tom-fink",
"maintainer": "William Lindstrom",
"docs_url": null,
"requires_python": "<3.12,>=3.8.1",
"maintainer_email": "llindstrom@lco.global",
"keywords": "tomtoolkit, astronomy, astrophysics, cosmology, science, observatory, alert, broker, fink",
"author": "Julien Peloton",
"author_email": "peloton@lal.in2p3.fr",
"download_url": "https://files.pythonhosted.org/packages/a5/66/91545add556446ae78c439462dcc70c5216e597c4c89ce572f2b8cbfb09e/tom_fink-0.6.0.tar.gz",
"platform": null,
"description": "# Fink Alert Broker Module for the TOM Toolkit\n\n[](https://pypi.python.org/pypi/tom-fink)\n[](https://www.codacy.com/gh/TOMToolkit/tom_fink/dashboard?utm_source=github.com&utm_medium=referral&utm_content=TOMToolkit/tom_fink&utm_campaign=Badge_Grade)\n[](https://coveralls.io/github/TOMToolkit/tom_fink?branch=main)\n\nThis repository hosts the Fink Alert Broker Module for the TOM Toolkit. Fink is a broker currently connected to ZTF, and that will handle the massive alert stream from LSST. More information on Fink at https://fink-broker.org/. You can easily install the module using pip:\n\n```bash\npip install tom-fink\n```\n\n## Polling data from the REST API\n\nAs of `tom-fink` version 0.4, the module simply uses the [Fink REST API](https://fink-broker.readthedocs.io/en/latest/services/science_portal/) to retrieve alerts. Note that the Fink databases are updated **once a day** with the previous night alert data (hence you do not get live alerts for the moment). Users can perform:\n- Search by ZTF object ID\n- Cone Search\n- Search by Date\n- Search by derived alert class\n- Search by Solar System name\n\nYou need to declare the `FinkBroker` class in your running TOM instance. To do so just add `tom_fink.fink.FinkBroker` to the `TOM_ALERT_CLASSES` in your TOM's `settings.py`:\n\n```python\nTOM_ALERT_CLASSES = [\n 'tom_alerts.brokers.alerce.ALeRCEBroker',\n ...,\n 'tom_fink.fink.FinkBroker'\n]\n```\n\nand finally relaunch your TOM:\n\n```bash\n./manage.py runserver\n```\n\n## Polling data from the Fink livestream service\n\nAlternatively from `tom_fink` version 0.6.0, users can poll Fink substreams in real-time, using the [livestream service](https://fink-broker.readthedocs.io/en/latest/services/livestream/). First you need to add `tom_alertstreams` and `tom_fink` to your `INSTALLED_APPS` setting in your project `settings.py`:\n\n```python\nINSTALLED_APPS = [\n ...\n 'tom_alertstreams',\n 'tom_fink'\n]\n```\n\nThen add the following block to your `settings.py`:\n\n```python\nALERT_STREAMS = [\n {\n 'ACTIVE': True,\n 'NAME': 'tom_fink.alertstream.FinkAlertStream',\n 'OPTIONS': {\n 'URL': os.getenv('FINK_CREDENTIAL_URL', 'set FINK_CREDENTIAL_URL value in environment'),\n 'USERNAME': os.getenv('FINK_CREDENTIAL_USERNAME', 'set FINK_CREDENTIAL_USERNAME value in environment'),\n 'GROUP_ID': os.getenv('FINK_CREDENTIAL_GROUP_ID', 'set FINK_CREDENTIAL_GROUP_ID value in environment'),\n 'TOPIC': os.getenv('FINK_TOPIC', 'set FINK_TOPIC value in environment'),\n 'MAX_POLL_NUMBER': os.getenv(\"FINK_MAX_POLL_NUMBER\", 1e10),\n 'TIMEOUT': os.getenv('FINK_TIMEOUT', 10),\n 'TOPIC_HANDLERS': {\n 'fink.stream': 'tom_fink.alertstream.alert_logger',\n },\n },\n },\n]\n```\n\nand declare the following variables in your `~/.bash_profile`:\n\n```bash\nexport FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream\nexport FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream\nexport FINK_CREDENTIAL_URL= # provided when registering to Fink livestream\nexport FINK_TOPIC= # str, topic name to poll\nexport FINK_MAX_POLL_NUMBER= # int, number maximum of alerts to poll before closing the connection. Default is 1e10 (never stops).\nexport FINK_TIMEOUT= # int, in seconds. Default is 10 seconds if not set\n```\n\nCredentials are provided when registering to Fink livestream (see https://fink-broker.readthedocs.io/en/latest/services/livestream). The available topic names and description can be found at https://fink-broker.readthedocs.io/en/latest/services/livestream. Note that we currently allow to poll only one topic at once (multi-topic is planned for later). Then launch the `readstreams` service:\n\n```bash\n./manage.py readstreams\n```\n\nYou should see a log similar to:\n\n```bash\nregistering new views: args: ('groups', <class 'tom_common.api_views.GroupViewSet'>, 'groups'), kwargs: {}\nregistering new views: args: ('targets', <class 'tom_targets.api_views.TargetViewSet'>, 'targets'), kwargs: {}\nregistering new views: args: ('targetextra', <class 'tom_targets.api_views.TargetExtraViewSet'>, 'targetextra'), kwargs: {}\nregistering new views: args: ('targetname', <class 'tom_targets.api_views.TargetNameViewSet'>, 'targetname'), kwargs: {}\nregistering new views: args: ('targetlist', <class 'tom_targets.api_views.TargetListViewSet'>, 'targetlist'), kwargs: {}\nregistering new views: args: ('observations', <class 'tom_observations.api_views.ObservationRecordViewSet'>, 'observations'), kwargs: {}\nregistering new views: args: ('dataproducts', <class 'tom_dataproducts.api_views.DataProductViewSet'>, 'dataproducts'), kwargs: {}\nregistering new views: args: ('reduceddatums', <class 'tom_dataproducts.api_views.ReducedDatumViewSet'>, 'reduceddatums'), kwargs: {}\nNumExpr defaulting to 8 threads.\nFinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 0)\nread_streams FinkAlertStream TID=378166 ; thread identifier=125271444649536\nreadstreams Command.handle() returning...\nfink.alert_logger topic: fink_sso_ztf_candidates_ztf\nfink.alert_logger value: ZTF24aakwfsz emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 10:45:01)\nFinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 1)\nfink.alert_logger topic: fink_sso_ztf_candidates_ztf\nfink.alert_logger value: ZTF24aakwftv emitted 2024-04-23 07:27:39.003823 (received 2024-04-29 10:45:01)\n...\n```\n\nEach new alert will be saved as a `Target`. If more than one alert with the name `objectId` are emitted, you will be warned:\n\n```bash\nfink.alert_logger value: ZTF24aakwfsu emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 08:18:07)\nTarget ZTF24aakwfsu already in the database\n```\n\nand the program will continue. Probably we should group these in the future, instead of ignoring subsequent alerts. Then launch the app (do not close the previous process!):\n\n```bash\n./manage.py runserver\n```\n\nand you should have access to your targets, that are accessible from their topic name:\n\n\n\n### Testing & debugging the connection\n\nBefore running in production, we advise to make tests using a test stream, and polling a few alerts:\n\n```bash\nexport FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream\nexport FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream\nexport FINK_CREDENTIAL_URL= # provided when registering to Fink livestream\nexport FINK_TOPIC=fink_sso_ztf_candidates_ztf\nexport FINK_MAX_POLL_NUMBER=2\nexport FINK_TIMEOUT=10\n```\n\nThis way, you will poll alerts two-by-two, and you will have a reservoir of thousands of alerts (solar system objects)!\n\n## Todo list\n\n- [ ] Add a test suite (preferably running on GitHub Actions)\n- [x] Add a linter (preferably running on GitHub Actions)\n- [x] Update the Query Form with all API features\n- [x] Enable querying live streams using the Fink Kafka client\n- [x] Update visualisation of targets from the livestream service\n\n",
"bugtrack_url": null,
"license": "GPL-3.0 license",
"summary": "The Fink Alert Broker Module for the TOM Toolkit. Fink is a broker currently connected to ZTF. More information on Fink at https://fink-portal.org",
"version": "0.6.0",
"project_urls": {
"Homepage": "https://github.com/TOMToolkit/tom_fink",
"Repository": "https://github.com/TOMToolkit/tom_fink"
},
"split_keywords": [
"tomtoolkit",
" astronomy",
" astrophysics",
" cosmology",
" science",
" observatory",
" alert",
" broker",
" fink"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9cafbcd6ceac11fe005b4e77bda77c410d9c9b1dc24d4bb65a2cb60fbc633888",
"md5": "a72791521fb462e4829d48b62e9abe3b",
"sha256": "4b14e7a123c77e06e6130f969c5eacbf36d6a6ab4b2af12992672265c07a53c2"
},
"downloads": -1,
"filename": "tom_fink-0.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a72791521fb462e4829d48b62e9abe3b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.8.1",
"size": 25798,
"upload_time": "2024-04-29T11:33:37",
"upload_time_iso_8601": "2024-04-29T11:33:37.507722Z",
"url": "https://files.pythonhosted.org/packages/9c/af/bcd6ceac11fe005b4e77bda77c410d9c9b1dc24d4bb65a2cb60fbc633888/tom_fink-0.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a56691545add556446ae78c439462dcc70c5216e597c4c89ce572f2b8cbfb09e",
"md5": "11b3562afed3dfcc33c7886a562d6725",
"sha256": "68a9fc0d889acfc6454118d4628eab0e76768ac73e0f4e2676efcf89dad4d3c5"
},
"downloads": -1,
"filename": "tom_fink-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "11b3562afed3dfcc33c7886a562d6725",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.8.1",
"size": 24705,
"upload_time": "2024-04-29T11:33:38",
"upload_time_iso_8601": "2024-04-29T11:33:38.620933Z",
"url": "https://files.pythonhosted.org/packages/a5/66/91545add556446ae78c439462dcc70c5216e597c4c89ce572f2b8cbfb09e/tom_fink-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-29 11:33:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TOMToolkit",
"github_project": "tom_fink",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "tom-fink"
}
JSON
