.. image:: https://readthedocs.org/projects/abstract-producer/badge/?version=latest
:target: https://abstract-producer.readthedocs.io/en/latest/
:alt: Documentation Status
.. image:: https://github.com/MacHu-GWU/abstract_producer-project/workflows/CI/badge.svg
:target: https://github.com/MacHu-GWU/abstract_producer-project/actions?query=workflow:CI
.. image:: https://codecov.io/gh/MacHu-GWU/abstract_producer-project/branch/main/graph/badge.svg
:target: https://codecov.io/gh/MacHu-GWU/abstract_producer-project
.. image:: https://img.shields.io/pypi/v/abstract-producer.svg
:target: https://pypi.python.org/pypi/abstract-producer
.. image:: https://img.shields.io/pypi/l/abstract-producer.svg
:target: https://pypi.python.org/pypi/abstract-producer
.. image:: https://img.shields.io/pypi/pyversions/abstract-producer.svg
:target: https://pypi.python.org/pypi/abstract-producer
.. image:: https://img.shields.io/badge/Release_History!--None.svg?style=social
:target: https://github.com/MacHu-GWU/abstract_producer-project/blob/main/release-history.rst
.. image:: https://img.shields.io/badge/STAR_Me_on_GitHub!--None.svg?style=social
:target: https://github.com/MacHu-GWU/abstract_producer-project
------
.. image:: https://img.shields.io/badge/Link-Document-blue.svg
:target: https://abstract-producer.readthedocs.io/en/latest/
.. image:: https://img.shields.io/badge/Link-API-blue.svg
:target: https://abstract-producer.readthedocs.io/en/latest/py-modindex.html
.. image:: https://img.shields.io/badge/Link-Install-blue.svg
:target: `install`_
.. image:: https://img.shields.io/badge/Link-GitHub-blue.svg
:target: https://github.com/MacHu-GWU/abstract_producer-project
.. image:: https://img.shields.io/badge/Link-Submit_Issue-blue.svg
:target: https://github.com/MacHu-GWU/abstract_producer-project/issues
.. image:: https://img.shields.io/badge/Link-Request_Feature-blue.svg
:target: https://github.com/MacHu-GWU/abstract_producer-project/issues
.. image:: https://img.shields.io/badge/Link-Download-blue.svg
:target: https://pypi.org/pypi/abstract-producer#files
Welcome to ``abstract_producer`` Documentation
==============================================================================
📔 See `Full Documentation HERE <https://abstract-producer.readthedocs.io/>`_.
.. image:: https://abstract-producer.readthedocs.io/en/latest/_static/abstract_producer-logo.png
:target: https://abstract-producer.readthedocs.io/
This library provides the abstraction of data producer, which is a common client sending data to stream processing system, such as Apache Kafka, Apache Pulsar, AWS Kinesis, AWS SQS, AWS CloudWatch logs, etc.
It has the following business critical features out-of-box:
1. group records into micro batch to maximize the utilization of network bandwidth.
2. use local write-ahead-log to prevent unexpected error and data loss.
3. auto-retry using exponential backoff strategy to handle transient error.
With this library, it is easy to create data producer client library for any stream processing system.
Usage Examples
------------------------------------------------------------------------------
- `A POC producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/simple_producer.py>`_: This is a very simple producer that write data to a local, append-only file.
- `A POC consumer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/simple_consumer.py>`_: This is a very simple consumer that work closely with the POC producer for demonstration purpose. You can follow these two example to implement your own producer and consumer library for any stream processing system.
- `AWS CloudWatch Logs producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_cloudwatch_logs_producer.py>`_.
- `AWS Kinesis Stream producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_kinesis_producer.py>`_.
- `AWS Kinesis Stream consumer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_kinesis_consumer.py>`_.
.. _install:
Install
------------------------------------------------------------------------------
``abstract_producer`` is released on PyPI, so all you need is to:
.. code-block:: console
$ pip install abstract-producer
To upgrade to latest version:
.. code-block:: console
$ pip install --upgrade abstract-producer
Raw data
{
"_id": null,
"home_page": "https://github.com/MacHu-GWU/abstract_producer-project",
"name": "abstract-producer",
"maintainer": "Sanhe Hu",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "husanhe@gmail.com",
"keywords": "",
"author": "Sanhe Hu",
"author_email": "husanhe@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/6b/29/35d3f0808c918b4d0208ae69b249f1c711c12cb7fb0144cb5db81504f97f/abstract_producer-0.3.1.tar.gz",
"platform": "Windows",
"description": "\n.. image:: https://readthedocs.org/projects/abstract-producer/badge/?version=latest\n :target: https://abstract-producer.readthedocs.io/en/latest/\n :alt: Documentation Status\n\n.. image:: https://github.com/MacHu-GWU/abstract_producer-project/workflows/CI/badge.svg\n :target: https://github.com/MacHu-GWU/abstract_producer-project/actions?query=workflow:CI\n\n.. image:: https://codecov.io/gh/MacHu-GWU/abstract_producer-project/branch/main/graph/badge.svg\n :target: https://codecov.io/gh/MacHu-GWU/abstract_producer-project\n\n.. image:: https://img.shields.io/pypi/v/abstract-producer.svg\n :target: https://pypi.python.org/pypi/abstract-producer\n\n.. image:: https://img.shields.io/pypi/l/abstract-producer.svg\n :target: https://pypi.python.org/pypi/abstract-producer\n\n.. image:: https://img.shields.io/pypi/pyversions/abstract-producer.svg\n :target: https://pypi.python.org/pypi/abstract-producer\n\n.. image:: https://img.shields.io/badge/Release_History!--None.svg?style=social\n :target: https://github.com/MacHu-GWU/abstract_producer-project/blob/main/release-history.rst\n\n.. image:: https://img.shields.io/badge/STAR_Me_on_GitHub!--None.svg?style=social\n :target: https://github.com/MacHu-GWU/abstract_producer-project\n\n------\n\n.. image:: https://img.shields.io/badge/Link-Document-blue.svg\n :target: https://abstract-producer.readthedocs.io/en/latest/\n\n.. image:: https://img.shields.io/badge/Link-API-blue.svg\n :target: https://abstract-producer.readthedocs.io/en/latest/py-modindex.html\n\n.. image:: https://img.shields.io/badge/Link-Install-blue.svg\n :target: `install`_\n\n.. image:: https://img.shields.io/badge/Link-GitHub-blue.svg\n :target: https://github.com/MacHu-GWU/abstract_producer-project\n\n.. image:: https://img.shields.io/badge/Link-Submit_Issue-blue.svg\n :target: https://github.com/MacHu-GWU/abstract_producer-project/issues\n\n.. image:: https://img.shields.io/badge/Link-Request_Feature-blue.svg\n :target: https://github.com/MacHu-GWU/abstract_producer-project/issues\n\n.. image:: https://img.shields.io/badge/Link-Download-blue.svg\n :target: https://pypi.org/pypi/abstract-producer#files\n\n\nWelcome to ``abstract_producer`` Documentation\n==============================================================================\n\ud83d\udcd4 See `Full Documentation HERE <https://abstract-producer.readthedocs.io/>`_.\n\n.. image:: https://abstract-producer.readthedocs.io/en/latest/_static/abstract_producer-logo.png\n :target: https://abstract-producer.readthedocs.io/\n\nThis library provides the abstraction of data producer, which is a common client sending data to stream processing system, such as Apache Kafka, Apache Pulsar, AWS Kinesis, AWS SQS, AWS CloudWatch logs, etc.\n\nIt has the following business critical features out-of-box:\n\n1. group records into micro batch to maximize the utilization of network bandwidth.\n2. use local write-ahead-log to prevent unexpected error and data loss.\n3. auto-retry using exponential backoff strategy to handle transient error.\n\nWith this library, it is easy to create data producer client library for any stream processing system.\n\n\nUsage Examples\n------------------------------------------------------------------------------\n- `A POC producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/simple_producer.py>`_: This is a very simple producer that write data to a local, append-only file.\n- `A POC consumer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/simple_consumer.py>`_: This is a very simple consumer that work closely with the POC producer for demonstration purpose. You can follow these two example to implement your own producer and consumer library for any stream processing system.\n- `AWS CloudWatch Logs producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_cloudwatch_logs_producer.py>`_.\n- `AWS Kinesis Stream producer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_kinesis_producer.py>`_.\n- `AWS Kinesis Stream consumer <https://github.com/MacHu-GWU/abstract_producer-project/blob/main/examples/aws_kinesis_consumer.py>`_.\n\n\n.. _install:\n\nInstall\n------------------------------------------------------------------------------\n\n``abstract_producer`` is released on PyPI, so all you need is to:\n\n.. code-block:: console\n\n $ pip install abstract-producer\n\nTo upgrade to latest version:\n\n.. code-block:: console\n\n $ pip install --upgrade abstract-producer\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Library for creating stream data producer client library.",
"version": "0.3.1",
"project_urls": {
"Download": "https://pypi.python.org/pypi/abstract_producer/0.3.1#downloads",
"Homepage": "https://github.com/MacHu-GWU/abstract_producer-project"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8fbe94c85b8ef4884f3d4fd757b33dfbc98d0948f6f80a462fe02c26146c23fe",
"md5": "99239ac6fd2ff9e11b8858433256205c",
"sha256": "52191f3df1325b5949cdbb8418891ad605e4be1907b6c15959e830ef5c05001c"
},
"downloads": -1,
"filename": "abstract_producer-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "99239ac6fd2ff9e11b8858433256205c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 42501,
"upload_time": "2024-01-09T06:46:22",
"upload_time_iso_8601": "2024-01-09T06:46:22.575953Z",
"url": "https://files.pythonhosted.org/packages/8f/be/94c85b8ef4884f3d4fd757b33dfbc98d0948f6f80a462fe02c26146c23fe/abstract_producer-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6b2935d3f0808c918b4d0208ae69b249f1c711c12cb7fb0144cb5db81504f97f",
"md5": "d6911b24feb17fa724b283023d2e2a05",
"sha256": "23eeff6b85660814430b4e403ab3fa158890e258174ee6fe4515c0753023f923"
},
"downloads": -1,
"filename": "abstract_producer-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "d6911b24feb17fa724b283023d2e2a05",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 39441,
"upload_time": "2024-01-09T06:46:23",
"upload_time_iso_8601": "2024-01-09T06:46:23.776096Z",
"url": "https://files.pythonhosted.org/packages/6b/29/35d3f0808c918b4d0208ae69b249f1c711c12cb7fb0144cb5db81504f97f/abstract_producer-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-09 06:46:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MacHu-GWU",
"github_project": "abstract_producer-project",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "abstract-producer"
}
JSON
