pysoa


Namepysoa JSON
Version 0.36.1 PyPI version JSON
download
home_pagehttp://github.com/eventbrite/pysoa
SummaryA Python library for writing (micro)services and their clients
upload_time2018-04-14 19:22:23
maintainer
docs_urlNone
authorEventbrite, Inc.
requires_python
licenseApache 2.0
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI
coveralls test coverage No coveralls.
            PySOA
=====

.. image:: https://api.travis-ci.org/eventbrite/pysoa.svg
    :target: https://travis-ci.org/eventbrite/pysoa

.. image:: https://img.shields.io/pypi/v/pysoa.svg
    :target: https://pypi.python.org/pypi/pysoa

.. image:: https://img.shields.io/pypi/l/pysoa.svg
    :target: https://pypi.python.org/pypi/pysoa


A general-purpose library for writing Python (micro)services and their clients, based on an RPC (remote procedure call)
calling style. Provides both a client and a server, which can be used directly by themselves or, as we do, extended with
extra functionality (our authentication, metrics, and other code is all written as middleware and run on top of this
library).

PySOA uses the concept of "transports" to define a layer for sending requests and responses (messages) between clients
and servers. The intended transport is a `Redis <https://redis.io/>`_ pub-sub layer, which we use in combination with
Redis Sentinel in clusters. There is also a local transport implementation for testing and other uses.

The basic tenets of the framework are:

* Services and actions both have simple names, and are called from the client by name. You can call actions
  individually, or bundle multiple action calls into a Job to be run serially (either aborting or continuing on error).

* Requests and responses are simply Python ``dicts``, and PySOA uses our open source validation framework
  `conformity <https://github.com/eventbrite/conformity>`_ in order to verify their schema on the way in and out.

* Message bodies are encoded using `MessagePack <http://msgpack.org/>`_ by default (however, you can define your own
  serializer), with a few non-standard types encoded using msgpack's ``ext``, such as dates, times, date-times, and
  amounts of currency (using our open source `currint <https://github.com/eventbrite/currint>`_ library)

* Requests have a ``context``, which is sourced from the original client context (web request, API request, etc.) and
  automatically chained down into subsequent client calls made inside the service. This is used for things like
  correlation IDs, locales, etc.

* We include "SOA Switches" as a first-party implementation of feature flags/toggles. Like the context, they are
  bundled along with every request and automatically chained, and are packed to try and ensure they have minimal
  overhead.

This intro summarizes some of the key concepts of using PySOA. For more thorough documentation, see the
`PySOA documentation <docs/index.rst>`_.


Servers
-------

SOA servers run as standalone processes and connect out to their transport to service requests and send responses, with
no listening ports. This means they can easily be scaled by simply launching or killing instances with whatever
orchestration software you want to use.

You can run all of the servers under a single channel layer (Redis instance/Sentinel cluster), have a separate layer
per service, or have separate layers for different quality of service levels for your site based on the access point
and type of accessing user.

Servers declare one or more Actions, which are registered on the class. Actions are callable objects of some type (such
as a function or method, or a class with a ``__call__`` method that will get instantiated before being called) that get
called with a request and return a response. We provide a base ``Action`` class that extends this contract to also
implement validation on requests and responses, but there is no requirement to use this if your needs are more complex.
Actions that are classes will be passed a reference to the server's settings object when instantiated.

.. code:: python

    from pysoa import server

    from example_service.actions.call_service import CallServiceAction
    from example_service.actions.square import SquareAction
    from example_service.actions.status import StatusAction


    class Server(server.BaseServer):

        service_name = 'example'

        action_class_map = {
            'call_service': CallServiceAction,
            'square': SquareAction,
            'status': StatusAction,
        }


A fully-functional `Example Service <https://github.com/eventbrite/example_service>`_ is available for your analysis
and experimentation. We encourage you to browse its source code, and even start it up, to see how it works and get a
better idea how to build services using PySOA.


Clients
-------

Clients are instantiated with a dictionary of service names and the transports by which they can be reached. There are
several approaches for calling service actions with a ``Client`` object:

* Calling a single action and getting the action response back directly using ``call_action``:

  .. code:: python

      action_response = client.call_action('example', 'square', {'number': 42})

* Creating a single job of multiple action requests, and sending it off to all be processed by the same server
  instance, serially:

  .. code:: python

      job_response = client.call_actions('example', [
          {'action': 'square', 'body': {'number': 42}},
          {'action': 'status', 'body': {'verbose': True}},
      ])

* Creating multiple jobs, one for each action belonging to the same service, and send them off to be processed by
  multiple server instances in parallel:

  .. code:: python

      action_responses = client.call_actions_parallel('example', [
          {'action': 'square', 'body': {'number': 1035}},
          {'action': 'status', 'body': {'verbose': True}},
      ])

* Creating multiple jobs, each with its own service name and one or more actions, and send them off to be processed by
  multiple server instances in parallel:

  .. code:: python

      job_responses = client.call_jobs_parallel([
          {'service_name': 'example', 'actions': [
              {'action': 'square', 'body': {'number': 4}},
              {'action': 'square', 'body': {'number': 8}},
              {'action': 'square', 'body': {'number': 17}},
          ]},
          {'service_name': 'example', 'actions': [{'action': 'status', 'body': {'verbose': True}}]},
          {'service_name': 'flight_booking', 'actions': [
              {'action': 'get_available_flights', 'body': {
                  'departure_airport': 'BNA',
                  'arrival_airport': 'SFO',
                  'departure_date': '2018-07-15',
                  'return_date': '2018-07-20',
              }},
          ]},
      ])


Middleware
----------

Both clients and servers can be extended using middleware, which, in the Django style, is code that wraps around a
request-response call, either on the client or server side, to add or mutate things in the request or response.

For example, some of our internal server middleware:

* Reads authentication tokens from the request and validates them to make sure the request is valid and not too old

* Logs metrics at the start and end of an action being processed so we can track how long our code is taking to run

* Catches errors in server code and logs it into Sentry so we can track and fix problems in production


Settings
--------

Both client and server use a dict-based settings system, with a
`conformity <https://github.com/eventbrite/conformity>`_-defined schema to ensure that whatever settings are provided
are valid (this schema is extensible by service implementations if they have special settings they need set).

The server also has an integration mode with Django where it will read its settings from
``django.conf.settings.SOA_SERVER_SETTINGS`` for both running and for tests, which allows easy integration of Django
models and application logic into services (we make heavy use of the Django ORM in our services).


Testing
-------

Services can be tested using standard unit tests and either by calling the actions directly (after all, they are just
callable objects), or, if a run through the server machinery is desired, using the ``ServerTestCase`` base class, which
takes care of setting up local transports for you.

For entire-system integration tests, you will need to spin up a copy of each desired service individually and point
them at an integration-test-specific channel layer to ensure isolation from the rest of the system.

There is also a ``StubClient`` available for testing code that calls services, but where you do not actually want to
have the service code in place, and a ``stub_action`` decorator / context manager that makes easy work of using it.

For more information about using these test utilities in your services or service-calling applications, see the testing
documentation in the `PySOA documentation <docs/index.rst>`_.

For testing the PySOA library directly, you must first install Lua on your system (on Mac OS X this is done with
``brew install lua``), ensure Lua is on your ``$PKG_CONFIG_PATH`` environment variable (in Mac OS X), and then install
dependencies (``pip install -e .[testing]``). After this, you can simply run ``pytest`` or ``setup.py test``.



            

Raw data

            {
    "maintainer": "", 
    "docs_url": null, 
    "requires_python": "", 
    "maintainer_email": "", 
    "cheesecake_code_kwalitee_id": null, 
    "keywords": "", 
    "upload_time": "2018-04-14 19:22:23", 
    "author": "Eventbrite, Inc.", 
    "home_page": "http://github.com/eventbrite/pysoa", 
    "github_user": "eventbrite", 
    "download_url": "https://pypi.python.org/packages/22/65/93942ab56bb61552486ac2c6923dca540a0af64387cb17ed86a7a877986d/pysoa-0.36.1.tar.gz", 
    "platform": "", 
    "version": "0.36.1", 
    "cheesecake_documentation_id": null, 
    "description": "PySOA\n=====\n\n.. image:: https://api.travis-ci.org/eventbrite/pysoa.svg\n    :target: https://travis-ci.org/eventbrite/pysoa\n\n.. image:: https://img.shields.io/pypi/v/pysoa.svg\n    :target: https://pypi.python.org/pypi/pysoa\n\n.. image:: https://img.shields.io/pypi/l/pysoa.svg\n    :target: https://pypi.python.org/pypi/pysoa\n\n\nA general-purpose library for writing Python (micro)services and their clients, based on an RPC (remote procedure call)\ncalling style. Provides both a client and a server, which can be used directly by themselves or, as we do, extended with\nextra functionality (our authentication, metrics, and other code is all written as middleware and run on top of this\nlibrary).\n\nPySOA uses the concept of \"transports\" to define a layer for sending requests and responses (messages) between clients\nand servers. The intended transport is a `Redis <https://redis.io/>`_ pub-sub layer, which we use in combination with\nRedis Sentinel in clusters. There is also a local transport implementation for testing and other uses.\n\nThe basic tenets of the framework are:\n\n* Services and actions both have simple names, and are called from the client by name. You can call actions\n  individually, or bundle multiple action calls into a Job to be run serially (either aborting or continuing on error).\n\n* Requests and responses are simply Python ``dicts``, and PySOA uses our open source validation framework\n  `conformity <https://github.com/eventbrite/conformity>`_ in order to verify their schema on the way in and out.\n\n* Message bodies are encoded using `MessagePack <http://msgpack.org/>`_ by default (however, you can define your own\n  serializer), with a few non-standard types encoded using msgpack's ``ext``, such as dates, times, date-times, and\n  amounts of currency (using our open source `currint <https://github.com/eventbrite/currint>`_ library)\n\n* Requests have a ``context``, which is sourced from the original client context (web request, API request, etc.) and\n  automatically chained down into subsequent client calls made inside the service. This is used for things like\n  correlation IDs, locales, etc.\n\n* We include \"SOA Switches\" as a first-party implementation of feature flags/toggles. Like the context, they are\n  bundled along with every request and automatically chained, and are packed to try and ensure they have minimal\n  overhead.\n\nThis intro summarizes some of the key concepts of using PySOA. For more thorough documentation, see the\n`PySOA documentation <docs/index.rst>`_.\n\n\nServers\n-------\n\nSOA servers run as standalone processes and connect out to their transport to service requests and send responses, with\nno listening ports. This means they can easily be scaled by simply launching or killing instances with whatever\norchestration software you want to use.\n\nYou can run all of the servers under a single channel layer (Redis instance/Sentinel cluster), have a separate layer\nper service, or have separate layers for different quality of service levels for your site based on the access point\nand type of accessing user.\n\nServers declare one or more Actions, which are registered on the class. Actions are callable objects of some type (such\nas a function or method, or a class with a ``__call__`` method that will get instantiated before being called) that get\ncalled with a request and return a response. We provide a base ``Action`` class that extends this contract to also\nimplement validation on requests and responses, but there is no requirement to use this if your needs are more complex.\nActions that are classes will be passed a reference to the server's settings object when instantiated.\n\n.. code:: python\n\n    from pysoa import server\n\n    from example_service.actions.call_service import CallServiceAction\n    from example_service.actions.square import SquareAction\n    from example_service.actions.status import StatusAction\n\n\n    class Server(server.BaseServer):\n\n        service_name = 'example'\n\n        action_class_map = {\n            'call_service': CallServiceAction,\n            'square': SquareAction,\n            'status': StatusAction,\n        }\n\n\nA fully-functional `Example Service <https://github.com/eventbrite/example_service>`_ is available for your analysis\nand experimentation. We encourage you to browse its source code, and even start it up, to see how it works and get a\nbetter idea how to build services using PySOA.\n\n\nClients\n-------\n\nClients are instantiated with a dictionary of service names and the transports by which they can be reached. There are\nseveral approaches for calling service actions with a ``Client`` object:\n\n* Calling a single action and getting the action response back directly using ``call_action``:\n\n  .. code:: python\n\n      action_response = client.call_action('example', 'square', {'number': 42})\n\n* Creating a single job of multiple action requests, and sending it off to all be processed by the same server\n  instance, serially:\n\n  .. code:: python\n\n      job_response = client.call_actions('example', [\n          {'action': 'square', 'body': {'number': 42}},\n          {'action': 'status', 'body': {'verbose': True}},\n      ])\n\n* Creating multiple jobs, one for each action belonging to the same service, and send them off to be processed by\n  multiple server instances in parallel:\n\n  .. code:: python\n\n      action_responses = client.call_actions_parallel('example', [\n          {'action': 'square', 'body': {'number': 1035}},\n          {'action': 'status', 'body': {'verbose': True}},\n      ])\n\n* Creating multiple jobs, each with its own service name and one or more actions, and send them off to be processed by\n  multiple server instances in parallel:\n\n  .. code:: python\n\n      job_responses = client.call_jobs_parallel([\n          {'service_name': 'example', 'actions': [\n              {'action': 'square', 'body': {'number': 4}},\n              {'action': 'square', 'body': {'number': 8}},\n              {'action': 'square', 'body': {'number': 17}},\n          ]},\n          {'service_name': 'example', 'actions': [{'action': 'status', 'body': {'verbose': True}}]},\n          {'service_name': 'flight_booking', 'actions': [\n              {'action': 'get_available_flights', 'body': {\n                  'departure_airport': 'BNA',\n                  'arrival_airport': 'SFO',\n                  'departure_date': '2018-07-15',\n                  'return_date': '2018-07-20',\n              }},\n          ]},\n      ])\n\n\nMiddleware\n----------\n\nBoth clients and servers can be extended using middleware, which, in the Django style, is code that wraps around a\nrequest-response call, either on the client or server side, to add or mutate things in the request or response.\n\nFor example, some of our internal server middleware:\n\n* Reads authentication tokens from the request and validates them to make sure the request is valid and not too old\n\n* Logs metrics at the start and end of an action being processed so we can track how long our code is taking to run\n\n* Catches errors in server code and logs it into Sentry so we can track and fix problems in production\n\n\nSettings\n--------\n\nBoth client and server use a dict-based settings system, with a\n`conformity <https://github.com/eventbrite/conformity>`_-defined schema to ensure that whatever settings are provided\nare valid (this schema is extensible by service implementations if they have special settings they need set).\n\nThe server also has an integration mode with Django where it will read its settings from\n``django.conf.settings.SOA_SERVER_SETTINGS`` for both running and for tests, which allows easy integration of Django\nmodels and application logic into services (we make heavy use of the Django ORM in our services).\n\n\nTesting\n-------\n\nServices can be tested using standard unit tests and either by calling the actions directly (after all, they are just\ncallable objects), or, if a run through the server machinery is desired, using the ``ServerTestCase`` base class, which\ntakes care of setting up local transports for you.\n\nFor entire-system integration tests, you will need to spin up a copy of each desired service individually and point\nthem at an integration-test-specific channel layer to ensure isolation from the rest of the system.\n\nThere is also a ``StubClient`` available for testing code that calls services, but where you do not actually want to\nhave the service code in place, and a ``stub_action`` decorator / context manager that makes easy work of using it.\n\nFor more information about using these test utilities in your services or service-calling applications, see the testing\ndocumentation in the `PySOA documentation <docs/index.rst>`_.\n\nFor testing the PySOA library directly, you must first install Lua on your system (on Mac OS X this is done with\n``brew install lua``), ensure Lua is on your ``$PKG_CONFIG_PATH`` environment variable (in Mac OS X), and then install\ndependencies (``pip install -e .[testing]``). After this, you can simply run ``pytest`` or ``setup.py test``.\n\n\n", 
    "lcname": "pysoa", 
    "bugtrack_url": "", 
    "github": true, 
    "coveralls": false, 
    "name": "pysoa", 
    "license": "Apache 2.0", 
    "travis_ci": true, 
    "github_project": "pysoa", 
    "summary": "A Python library for writing (micro)services and their clients", 
    "split_keywords": [], 
    "author_email": "opensource@eventbrite.com", 
    "urls": [
        {
            "has_sig": false, 
            "upload_time": "2018-04-14T19:22:21", 
            "comment_text": "", 
            "python_version": "py27.py35.py36", 
            "url": "https://pypi.python.org/packages/d9/97/cc9c8ee7c2a3ce58e657d030763eb01b2a5cc6826b84ded6b2e219839f7a/pysoa-0.36.1-py27.py35.py36-none-any.whl", 
            "md5_digest": "5ec140e97c33e8a46e470b594ce2a00d", 
            "downloads": 0, 
            "filename": "pysoa-0.36.1-py27.py35.py36-none-any.whl", 
            "packagetype": "bdist_wheel", 
            "path": "d9/97/cc9c8ee7c2a3ce58e657d030763eb01b2a5cc6826b84ded6b2e219839f7a/pysoa-0.36.1-py27.py35.py36-none-any.whl", 
            "digests": {
                "sha256": "4591fd8b012d323ce4c178fa8a042187f42dbc0269f2650a8c84aa3dfef1ab31", 
                "md5": "5ec140e97c33e8a46e470b594ce2a00d"
            }, 
            "sha256_digest": "4591fd8b012d323ce4c178fa8a042187f42dbc0269f2650a8c84aa3dfef1ab31", 
            "size": 85064
        }, 
        {
            "has_sig": false, 
            "upload_time": "2018-04-14T19:22:23", 
            "comment_text": "", 
            "python_version": "source", 
            "url": "https://pypi.python.org/packages/22/65/93942ab56bb61552486ac2c6923dca540a0af64387cb17ed86a7a877986d/pysoa-0.36.1.tar.gz", 
            "md5_digest": "760f204fb41f02551243322547de31f7", 
            "downloads": 0, 
            "filename": "pysoa-0.36.1.tar.gz", 
            "packagetype": "sdist", 
            "path": "22/65/93942ab56bb61552486ac2c6923dca540a0af64387cb17ed86a7a877986d/pysoa-0.36.1.tar.gz", 
            "digests": {
                "sha256": "41e9d54368b83164392589f079c050aa0a973f37c5fa465f63b7b04eecd95704", 
                "md5": "760f204fb41f02551243322547de31f7"
            }, 
            "sha256_digest": "41e9d54368b83164392589f079c050aa0a973f37c5fa465f63b7b04eecd95704", 
            "size": 62891
        }
    ], 
    "_id": null, 
    "cheesecake_installability_id": null
}