apf-base


Nameapf-base JSON
Version 2.5.2 PyPI version JSON
download
home_page
SummaryALeRCE Alert Processing Framework.
upload_time2023-07-21 16:03:10
maintainer
docs_urlNone
authorALeRCE Team
requires_python
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            Alert Processing Framework (*apf*)
================================================

[![Documentation Status](https://readthedocs.org/projects/apf/badge/?version=latest)](https://apf.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/alercebroker/APF/branch/master/graph/badge.svg)](https://codecov.io/gh/alercebroker/APF)


*apf* is a framework developed to create a dockerized pipeline to
process an alert stream, that can be easily be deployed in a local
machine or distributed using [Kubernetes](https://kubernetes.io).

First developed to process [ZTF data](https://www.ztf.caltech.edu/) it
is capable to be used for any stream/static data processing pipeline.

Installing *apf*
================

*apf* installation can be done with *pip*

``` {.sourceCode .bash}
pip install apf_base
```

This will install the *apf* python package and *apf* command line
script.

*apf* design
============

*apf* is based on *steps* conected through [Apache
Kafka](https://kafka.apache.org/) topics.

Each *step* is composed by a `consumer` and is isolated from other
steps inside a docker container.

When running, the step calls the **execute()** method for each *message* or *message batch*
consumed. A step can have multiple producers and databases back-ends
plugins that can be accessed inside the *execute* method to have a more
complex logic.


<p align="center">
  <img src="docs/source/_static/images/apf-flow.png">
</p>


This generic step greatly reduce the development of each component of
the pipeline and make it easier to test each component separately.


*apf* features
==============

- Automatic Metric Sender (KafkaMetrics)
- Automatic Code Generation (`apf new-step <step_name>`)
- Multiple Consumer Plugins:
  - Kafka
  - AVRO
  - CSV
  - JSON
- Producers:
  - Kafka
  - CSV
- Metrics:
  - Kafka

*apf* Quickstart
===================

A **quick-start** guide to create a new step can be found [here](https://apf.readthedocs.io/en/latest/new_step.html).

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "apf-base",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "ALeRCE Team",
    "author_email": "contact@alerce.online",
    "download_url": "https://files.pythonhosted.org/packages/a1/0c/505c0bd9651fffd695649978f6ef5efe026e982c268662aadda5ba8dc299/apf_base-2.5.2.tar.gz",
    "platform": null,
    "description": "Alert Processing Framework (*apf*)\n================================================\n\n[![Documentation Status](https://readthedocs.org/projects/apf/badge/?version=latest)](https://apf.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/alercebroker/APF/branch/master/graph/badge.svg)](https://codecov.io/gh/alercebroker/APF)\n\n\n*apf* is a framework developed to create a dockerized pipeline to\nprocess an alert stream, that can be easily be deployed in a local\nmachine or distributed using [Kubernetes](https://kubernetes.io).\n\nFirst developed to process [ZTF data](https://www.ztf.caltech.edu/) it\nis capable to be used for any stream/static data processing pipeline.\n\nInstalling *apf*\n================\n\n*apf* installation can be done with *pip*\n\n``` {.sourceCode .bash}\npip install apf_base\n```\n\nThis will install the *apf* python package and *apf* command line\nscript.\n\n*apf* design\n============\n\n*apf* is based on *steps* conected through [Apache\nKafka](https://kafka.apache.org/) topics.\n\nEach *step* is composed by a `consumer` and is isolated from other\nsteps inside a docker container.\n\nWhen running, the step calls the **execute()** method for each *message* or *message batch*\nconsumed. A step can have multiple producers and databases back-ends\nplugins that can be accessed inside the *execute* method to have a more\ncomplex logic.\n\n\n<p align=\"center\">\n  <img src=\"docs/source/_static/images/apf-flow.png\">\n</p>\n\n\nThis generic step greatly reduce the development of each component of\nthe pipeline and make it easier to test each component separately.\n\n\n*apf* features\n==============\n\n- Automatic Metric Sender (KafkaMetrics)\n- Automatic Code Generation (`apf new-step <step_name>`)\n- Multiple Consumer Plugins:\n  - Kafka\n  - AVRO\n  - CSV\n  - JSON\n- Producers:\n  - Kafka\n  - CSV\n- Metrics:\n  - Kafka\n\n*apf* Quickstart\n===================\n\nA **quick-start** guide to create a new step can be found [here](https://apf.readthedocs.io/en/latest/new_step.html).\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "ALeRCE Alert Processing Framework.",
    "version": "2.5.2",
    "project_urls": {
        "Documentation": "https://apf.readthedocs.io/en/latest/index.html",
        "Github": "https://github.com/alercebroker/APF"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "c138feaf618aad398d8d1ca5d5eb8985bcfc35bc7448fb8c5daab4a3fda00a02",
                "md5": "e739c3d4f49c472b421b90e8fb912c9c",
                "sha256": "eb495838a5159fae030a514f901e4fe028e58c058dcf95bf0b9fc2c1a208343f"
            },
            "downloads": -1,
            "filename": "apf_base-2.5.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "e739c3d4f49c472b421b90e8fb912c9c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 27205,
            "upload_time": "2023-07-21T16:03:09",
            "upload_time_iso_8601": "2023-07-21T16:03:09.828463Z",
            "url": "https://files.pythonhosted.org/packages/c1/38/feaf618aad398d8d1ca5d5eb8985bcfc35bc7448fb8c5daab4a3fda00a02/apf_base-2.5.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a10c505c0bd9651fffd695649978f6ef5efe026e982c268662aadda5ba8dc299",
                "md5": "e32e616399c64b5236e38cf886f783ca",
                "sha256": "c60b80db5793c6ee5499ac5286aaa734221e187813f6c37eb184ebd762685fde"
            },
            "downloads": -1,
            "filename": "apf_base-2.5.2.tar.gz",
            "has_sig": false,
            "md5_digest": "e32e616399c64b5236e38cf886f783ca",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 18789,
            "upload_time": "2023-07-21T16:03:10",
            "upload_time_iso_8601": "2023-07-21T16:03:10.847300Z",
            "url": "https://files.pythonhosted.org/packages/a1/0c/505c0bd9651fffd695649978f6ef5efe026e982c268662aadda5ba8dc299/apf_base-2.5.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-07-21 16:03:10",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "alercebroker",
    "github_project": "APF",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "apf-base"
}
        
Elapsed time: 0.13810s