Project Hadron Foundation Package
#################################
**Project Hadron** Foundation package is an open-source application framework, written in
pure Python using PyArrow as its canonical and depends on Python and pyArrow packages
only. It provides a set of abstractions that allow a quick to market solution of
component services (microservices) relevant to a use case. Component services are built
for tasks called capabilities with each capability performing a single function. Because
they are independently run, each capability can be updated, deployed, and scaled to meet
demand for specific functions of an application.
What are capabilities?
----------------------
In project Hadron capabilities are components that adhere to the fundamental concepts
of `capability patterns` and `separation of concern` (SoC). They are design principle
that advocates breaking a software system into distinct, independent modules or
components with, low coupling and high cohesion each addressing a specific concern or
aspect of the system's functionality.
Capabilities are reusable and encapsulated tasks which can be applied at any stage of the
life cycle and prescribes a work breakdown structure of functionalities and features a
software solution possesses.
Together, capability patterns help in understanding what a reusable component task should
achieve, while separation of concerns ensures that the component task is designed in a
modular and maintainable way, with each part addressing a specific aspect of its
functionality. Both principles contribute to building modular, robust and scalable
software solutions.
The build overview
------------------
At the heart of Project Hadron is a multi-tenant, NoSQL, singleton, in memory data store that has
minimal code and functionality and has been custom built specifically for Hadron tasks in mind.
Abstracted from this is the component store which allows us to build a reusable set of methods
that define each tenanted component that sits separately from the store itself. In addition, a
dynamic key value class provides labeling so that each tenant is not tied to a fixed set of
reference values unless by specificity. Each of the classes, the data store, the component
property manager, and the key value pairs that make up the component are all independent,
giving complete flexibility and minimum code footprint to the build process of new components.
Installation
============
package install
---------------
The best way to install this package is directly from the Python Package Index repository using pip
.. code-block:: bash
$ pip install discovery-core
if you want to upgrade your current version then using pip
.. code-block:: bash
$ pip install --upgrade discovery-core
Package Overview
================
AbstractComponent
-----------------
The ``AbstractComponent`` class is a foundation class for the component build. It provides an encapsulated view of
the Property Management and Parameterised Intent
Abstract AI Single Task Application Component (AI-STAC) component class provides all the basic building blocks
of a components build including property management, augmented knowledge notes and parameterised intent pipeline.
For convenience there are three Factory Initialisation methods available``from_env(...)``, ``from_memory(...)`` and
``from_uri(...)`` the first two being abstract methods. The thrid factory method initialises the concrete
PropertyManager and IntentModel classes and use the parent ``_init_properties(...)`` methods to set the properties
connector. When creating the concrete class the ``from_uri(...)`` should be implemented. The following method can be
used as a template replacing ``ExamplePropertyManager`` and ``ExampleIntentModel`` with your oen concrete
implementations
.. code-block:: python
@classmethod
def from_uri(cls, task_name: str, uri_pm_path: str, username: str, uri_pm_repo: str=None,
pm_file_type: str=None, pm_module: str=None, pm_handler: str=None, pm_kwargs: dict=None,
default_save=None, reset_templates: bool=None, template_path: str=None, template_module: str=None,
template_source_handler: str=None, template_persist_handler: str=None, align_connectors: bool=None,
default_save_intent: bool=None, default_intent_level: bool=None, order_next_available: bool=None,
default_replace_intent: bool=None, has_contract: bool=None):
pm_file_type = pm_file_type if isinstance(pm_file_type, str) else 'parquet'
pm_module = pm_module if isinstance(pm_module, str) else cls.DEFAULT_MODULE
pm_handler = pm_handler if isinstance(pm_handler, str) else cls.DEFAULT_PERSIST_HANDLER
_pm = ExamplePropertyManager(task_name=task_name, username=username)
_intent_model = ExampleIntentModel(property_manager=_pm, default_save_intent=default_save_intent,
default_intent_level=default_intent_level,
order_next_available=order_next_available,
default_replace_intent=default_replace_intent)
super()._init_properties(property_manager=_pm, uri_pm_path=uri_pm_path, default_save=default_save,
uri_pm_repo=uri_pm_repo, pm_file_type=pm_file_type, pm_module=pm_module,
pm_handler=pm_handler, pm_kwargs=pm_kwargs, has_contract=has_contract)
return cls(property_manager=_pm, intent_model=_intent_model, default_save=default_save,
reset_templates=reset_templates, template_path=template_path, template_module=template_module,
template_source_handler=template_source_handler, template_persist_handler=template_persist_handler,
align_connectors=align_connectors)
AbstractPropertyManager
-----------------------
The ``AbstractPropertiesManager`` facilitates the management of all the contract properties including that of the
connector handlers, parameterised intent and Augmented Knowledge
Abstract AI Single Task Application Component (AI-STAC) class that creates a super class for all properties
managers
The Class initialisation is abstracted and is the only abstracted method. A concrete implementation of the
overloaded ``__init__`` manages the ``root_key`` and ``knowledge_key`` for this construct. The ``root_key`` adds a key
property reference to the root of the properties and can be referenced directly with ``<name>_key``. Likewise
the ``knowledge_key`` adds a catalog key to the restricted catalog keys.
More complex ``root_key`` constructs, where a grouping of keys might be desirable, passing a dictionary of name
value pairs as part of the list allows a root base to group related next level keys. For example
.. code-block:: python
root_key = [{base: [primary, secondary}]
would add ``base.primary_key`` and ``base.secondary_key`` to the list of keys.
Here is a default example of an initialisation method:
.. code-block:: python
def __init__(self, task_name: str):
# set additional keys
root_keys = []
knowledge_keys = []
super().__init__(task_name=task_name, root_keys=root_keys, knowledge_keys=knowledge_keys)
The property manager is not responsible for persisting the properties but provides the methods to load and persist
its in memory structure. To initialise the load and persist a ConnectorContract must be set up.
The following is a code snippet of setting a ConnectorContract and loading its content
.. code-block:: python
self.set_property_connector(connector_contract=connector_contract)
if self.get_connector_handler(self.CONNECTOR_PM_CONTRACT).exists():
self.load_properties(replace=replace)
When using the property manager it will not automatically persist its properties and must be explicitely managed in
the component class. This removes the persist decision making away from the property manager. To persist the
properties use the method call ``persist_properties()``
AbstractIntentModel
-------------------
The ``AbstractIntentModel`` facilitates the Parameterised Intent, giving the base methods to record and replay intent.
Abstract AI Single Task Application Component (AI-STAC) Class for Parameterised Intent containing parameterised
intent registration methods ``_intent_builder(...)`` and ``_set_intend_signature(...)``.
it is creating a construct initialisation to allow for the control and definition of an ``intent_param_exclude``
list, ``default_save_intent`` boolean and a ``default_intent_level`` value.
As an example of an initialisation method
.. code-block:: python
def __init__(self, property_manager: AbstractPropertyManager, default_save_intent: bool=None,
default_intent_level: bool=None, order_next_available: bool=None, default_replace_intent: bool=None):
# set all the defaults
default_save_intent = default_save_intent if isinstance(default_save_intent, bool) else True
default_replace_intent = default_replace_intent if isinstance(default_replace_intent, bool) else True
default_intent_level = default_intent_level if isinstance(default_intent_level, (str, int, float)) else 0
default_intent_order = -1 if isinstance(order_next_available, bool) and order_next_available else 0
intent_param_exclude = ['data', 'inplace']
intent_type_additions = []
super().__init__(property_manager=property_manager, default_save_intent=default_save_intent,
intent_param_exclude=intent_param_exclude, default_intent_level=default_intent_level,
default_intent_order=default_intent_order, default_replace_intent=default_replace_intent,
intent_type_additions=intent_type_additions)
in order to define the run pattern for the component task ``run_intent_pipeline(...)`` is an abstracted method
that defines the run pipeline of the intent.
As an example of a run_pipeline that iteratively updates a canonical with each intent
.. code-block:: python
def run_intent_pipeline(self, canonical, intent_levels: [int, str, list]=None, **kwargs):
# test if there is any intent to run
if self._pm.has_intent():
# get the list of levels to run
if isinstance(intent_levels, (int, str, list)):
intent_levels = Commons.list_formatter(intent_levels)
else:
intent_levels = sorted(self._pm.get_intent().keys())
for level in intent_levels:
level_key = self._pm.join(self._pm.KEY.intent_key, level)
for order in sorted(self._pm.get(level_key, {})):
for method, params in self._pm.get(self._pm.join(level_key, order), {}).items():
if method in self.__dir__():
# add method kwargs to the params
if isinstance(kwargs, dict):
params.update(kwargs)
# add excluded parameters to the params
params.update({'inplace': False, 'save_intent': False})
canonical = eval(f"self.{method}(canonical, **{params})", globals(), locals())
return canonical
The code signature for an intent method would have the following construct
.. code-block:: python
def <method>(self, <params>..., save_intent: bool=None, intent_level: [int, str]=None, intent_order: int=None,
replace_intent: bool=None, remove_duplicates: bool=None):
# resolve intent persist options
self._set_intend_signature(self._intent_builder(method=inspect.currentframe().f_code.co_name, params=locals()),
intent_level=intent_level, intent_order=intent_order, replace_intent=replace_intent,
remove_duplicates=remove_duplicates, save_intent=save_intent)
# intend code block on the canonical
...
Reference
=========
Python version
--------------
Python 3.7 or less is not supported. Although Python 3.8 is supported, it is recommended to
install ``discovery-core`` against the latest Python release.
Licence
-------
MIT License: `<https://opensource.org/license/mit/>`_.
Authors
-------
`Gigas64`_ (`@gigas64`_) created discover-core.
.. _pip: https://pip.pypa.io/en/stable/installing/
.. _Github API: http://developer.github.com/v3/issues/comments/#create-a-comment
.. _Gigas64: http://opengrass.io
.. _@gigas64: https://twitter.com/gigas64
Raw data
{
"_id": null,
"home_page": "http://github.com/gigas64/discovery-core",
"name": "discovery-core",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Foundation SDK Machine learning AI component intent data science",
"author": "Gigas64",
"author_email": "gigas64@opengrass.net",
"download_url": "https://files.pythonhosted.org/packages/61/ae/288e7cdded8d27127ce17038224572fb3d469ba7ea23c94dde3dcb694f97/discovery_core-3.0.1.tar.gz",
"platform": null,
"description": "Project Hadron Foundation Package\n#################################\n\n**Project Hadron** Foundation package is an open-source application framework, written in\npure Python using PyArrow as its canonical and depends on Python and pyArrow packages\nonly. It provides a set of abstractions that allow a quick to market solution of\ncomponent services (microservices) relevant to a use case. Component services are built\nfor tasks called capabilities with each capability performing a single function. Because\nthey are independently run, each capability can be updated, deployed, and scaled to meet\ndemand for specific functions of an application.\n\nWhat are capabilities?\n----------------------\n\nIn project Hadron capabilities are components that adhere to the fundamental concepts\nof `capability patterns` and `separation of concern` (SoC). They are design principle\nthat advocates breaking a software system into distinct, independent modules or\ncomponents with, low coupling and high cohesion each addressing a specific concern or\naspect of the system's functionality.\n\nCapabilities are reusable and encapsulated tasks which can be applied at any stage of the\nlife cycle and prescribes a work breakdown structure of functionalities and features a\nsoftware solution possesses.\n\nTogether, capability patterns help in understanding what a reusable component task should\nachieve, while separation of concerns ensures that the component task is designed in a\nmodular and maintainable way, with each part addressing a specific aspect of its\nfunctionality. Both principles contribute to building modular, robust and scalable\nsoftware solutions.\n\nThe build overview\n------------------\n\nAt the heart of Project Hadron is a multi-tenant, NoSQL, singleton, in memory data store that has\nminimal code and functionality and has been custom built specifically for Hadron tasks in mind.\nAbstracted from this is the component store which allows us to build a reusable set of methods\nthat define each tenanted component that sits separately from the store itself. In addition, a\ndynamic key value class provides labeling so that each tenant is not tied to a fixed set of\nreference values unless by specificity. Each of the classes, the data store, the component\nproperty manager, and the key value pairs that make up the component are all independent,\ngiving complete flexibility and minimum code footprint to the build process of new components.\n\n\nInstallation\n============\n\npackage install\n---------------\n\nThe best way to install this package is directly from the Python Package Index repository using pip\n\n.. code-block:: bash\n\n $ pip install discovery-core\n\nif you want to upgrade your current version then using pip\n\n.. code-block:: bash\n\n $ pip install --upgrade discovery-core\n\nPackage Overview\n================\n\nAbstractComponent\n-----------------\n\nThe ``AbstractComponent`` class is a foundation class for the component build. It provides an encapsulated view of\nthe Property Management and Parameterised Intent\n\nAbstract AI Single Task Application Component (AI-STAC) component class provides all the basic building blocks\nof a components build including property management, augmented knowledge notes and parameterised intent pipeline.\n\nFor convenience there are three Factory Initialisation methods available``from_env(...)``, ``from_memory(...)`` and\n``from_uri(...)`` the first two being abstract methods. The thrid factory method initialises the concrete\nPropertyManager and IntentModel classes and use the parent ``_init_properties(...)`` methods to set the properties\nconnector. When creating the concrete class the ``from_uri(...)`` should be implemented. The following method can be\nused as a template replacing ``ExamplePropertyManager`` and ``ExampleIntentModel`` with your oen concrete\nimplementations\n\n.. code-block:: python\n\n @classmethod\n def from_uri(cls, task_name: str, uri_pm_path: str, username: str, uri_pm_repo: str=None,\n pm_file_type: str=None, pm_module: str=None, pm_handler: str=None, pm_kwargs: dict=None,\n default_save=None, reset_templates: bool=None, template_path: str=None, template_module: str=None,\n template_source_handler: str=None, template_persist_handler: str=None, align_connectors: bool=None,\n default_save_intent: bool=None, default_intent_level: bool=None, order_next_available: bool=None,\n default_replace_intent: bool=None, has_contract: bool=None):\n pm_file_type = pm_file_type if isinstance(pm_file_type, str) else 'parquet'\n pm_module = pm_module if isinstance(pm_module, str) else cls.DEFAULT_MODULE\n pm_handler = pm_handler if isinstance(pm_handler, str) else cls.DEFAULT_PERSIST_HANDLER\n _pm = ExamplePropertyManager(task_name=task_name, username=username)\n _intent_model = ExampleIntentModel(property_manager=_pm, default_save_intent=default_save_intent,\n default_intent_level=default_intent_level,\n order_next_available=order_next_available,\n default_replace_intent=default_replace_intent)\n super()._init_properties(property_manager=_pm, uri_pm_path=uri_pm_path, default_save=default_save,\n uri_pm_repo=uri_pm_repo, pm_file_type=pm_file_type, pm_module=pm_module,\n pm_handler=pm_handler, pm_kwargs=pm_kwargs, has_contract=has_contract)\n return cls(property_manager=_pm, intent_model=_intent_model, default_save=default_save,\n reset_templates=reset_templates, template_path=template_path, template_module=template_module,\n template_source_handler=template_source_handler, template_persist_handler=template_persist_handler,\n align_connectors=align_connectors)\n\n\nAbstractPropertyManager\n-----------------------\nThe ``AbstractPropertiesManager`` facilitates the management of all the contract properties including that of the\nconnector handlers, parameterised intent and Augmented Knowledge\n\nAbstract AI Single Task Application Component (AI-STAC) class that creates a super class for all properties\nmanagers\n\nThe Class initialisation is abstracted and is the only abstracted method. A concrete implementation of the\noverloaded ``__init__`` manages the ``root_key`` and ``knowledge_key`` for this construct. The ``root_key`` adds a key\nproperty reference to the root of the properties and can be referenced directly with ``<name>_key``. Likewise\nthe ``knowledge_key`` adds a catalog key to the restricted catalog keys.\n\nMore complex ``root_key`` constructs, where a grouping of keys might be desirable, passing a dictionary of name\nvalue pairs as part of the list allows a root base to group related next level keys. For example\n\n.. code-block:: python\n\n root_key = [{base: [primary, secondary}]\n\nwould add ``base.primary_key`` and ``base.secondary_key`` to the list of keys.\n\nHere is a default example of an initialisation method:\n\n.. code-block:: python\n\n def __init__(self, task_name: str):\n # set additional keys\n root_keys = []\n knowledge_keys = []\n super().__init__(task_name=task_name, root_keys=root_keys, knowledge_keys=knowledge_keys)\n\n\nThe property manager is not responsible for persisting the properties but provides the methods to load and persist\nits in memory structure. To initialise the load and persist a ConnectorContract must be set up.\n\nThe following is a code snippet of setting a ConnectorContract and loading its content\n\n.. code-block:: python\n\n self.set_property_connector(connector_contract=connector_contract)\n if self.get_connector_handler(self.CONNECTOR_PM_CONTRACT).exists():\n self.load_properties(replace=replace)\n\nWhen using the property manager it will not automatically persist its properties and must be explicitely managed in\nthe component class. This removes the persist decision making away from the property manager. To persist the\nproperties use the method call ``persist_properties()``\n\n\nAbstractIntentModel\n-------------------\nThe ``AbstractIntentModel`` facilitates the Parameterised Intent, giving the base methods to record and replay intent.\n\nAbstract AI Single Task Application Component (AI-STAC) Class for Parameterised Intent containing parameterised\nintent registration methods ``_intent_builder(...)`` and ``_set_intend_signature(...)``.\n\nit is creating a construct initialisation to allow for the control and definition of an ``intent_param_exclude``\nlist, ``default_save_intent`` boolean and a ``default_intent_level`` value.\n\nAs an example of an initialisation method\n\n.. code-block:: python\n\n def __init__(self, property_manager: AbstractPropertyManager, default_save_intent: bool=None,\n default_intent_level: bool=None, order_next_available: bool=None, default_replace_intent: bool=None):\n # set all the defaults\n default_save_intent = default_save_intent if isinstance(default_save_intent, bool) else True\n default_replace_intent = default_replace_intent if isinstance(default_replace_intent, bool) else True\n default_intent_level = default_intent_level if isinstance(default_intent_level, (str, int, float)) else 0\n default_intent_order = -1 if isinstance(order_next_available, bool) and order_next_available else 0\n intent_param_exclude = ['data', 'inplace']\n intent_type_additions = []\n super().__init__(property_manager=property_manager, default_save_intent=default_save_intent,\n intent_param_exclude=intent_param_exclude, default_intent_level=default_intent_level,\n default_intent_order=default_intent_order, default_replace_intent=default_replace_intent,\n intent_type_additions=intent_type_additions)\n\nin order to define the run pattern for the component task ``run_intent_pipeline(...)`` is an abstracted method\nthat defines the run pipeline of the intent.\n\nAs an example of a run_pipeline that iteratively updates a canonical with each intent\n\n.. code-block:: python\n\n def run_intent_pipeline(self, canonical, intent_levels: [int, str, list]=None, **kwargs):\n # test if there is any intent to run\n if self._pm.has_intent():\n # get the list of levels to run\n if isinstance(intent_levels, (int, str, list)):\n intent_levels = Commons.list_formatter(intent_levels)\n else:\n intent_levels = sorted(self._pm.get_intent().keys())\n for level in intent_levels:\n level_key = self._pm.join(self._pm.KEY.intent_key, level)\n for order in sorted(self._pm.get(level_key, {})):\n for method, params in self._pm.get(self._pm.join(level_key, order), {}).items():\n if method in self.__dir__():\n # add method kwargs to the params\n if isinstance(kwargs, dict):\n params.update(kwargs)\n # add excluded parameters to the params\n params.update({'inplace': False, 'save_intent': False})\n canonical = eval(f\"self.{method}(canonical, **{params})\", globals(), locals())\n return canonical\n\nThe code signature for an intent method would have the following construct\n\n.. code-block:: python\n\n def <method>(self, <params>..., save_intent: bool=None, intent_level: [int, str]=None, intent_order: int=None,\n replace_intent: bool=None, remove_duplicates: bool=None):\n # resolve intent persist options\n self._set_intend_signature(self._intent_builder(method=inspect.currentframe().f_code.co_name, params=locals()),\n intent_level=intent_level, intent_order=intent_order, replace_intent=replace_intent,\n remove_duplicates=remove_duplicates, save_intent=save_intent)\n # intend code block on the canonical\n ...\n\n\nReference\n=========\n\n\nPython version\n--------------\n\nPython 3.7 or less is not supported. Although Python 3.8 is supported, it is recommended to\ninstall ``discovery-core`` against the latest Python release.\n\nLicence\n-------\n\nMIT License: `<https://opensource.org/license/mit/>`_.\n\n\nAuthors\n-------\n\n`Gigas64`_ (`@gigas64`_) created discover-core.\n\n\n.. _pip: https://pip.pypa.io/en/stable/installing/\n.. _Github API: http://developer.github.com/v3/issues/comments/#create-a-comment\n.. _Gigas64: http://opengrass.io\n.. _@gigas64: https://twitter.com/gigas64\n\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "Augmented Intent Single Task Adaptive Components",
"version": "3.0.1",
"project_urls": {
"Homepage": "http://github.com/gigas64/discovery-core"
},
"split_keywords": [
"foundation",
"sdk",
"machine",
"learning",
"ai",
"component",
"intent",
"data",
"science"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "293972d14ca9178a9f7e2d4188fbb082ef26f19b8b4d280882033019ab7c4fb3",
"md5": "8fa23823e10aa2b21c6de04474d080a5",
"sha256": "7e36f99259ad7b87f6f72ab83fdbfff1b20bb32309ec436f7d470cad3b63fdd6"
},
"downloads": -1,
"filename": "discovery_core-3.0.1-py38-none-any.whl",
"has_sig": false,
"md5_digest": "8fa23823e10aa2b21c6de04474d080a5",
"packagetype": "bdist_wheel",
"python_version": "py38",
"requires_python": ">=3.8",
"size": 106300,
"upload_time": "2024-07-10T18:01:44",
"upload_time_iso_8601": "2024-07-10T18:01:44.986665Z",
"url": "https://files.pythonhosted.org/packages/29/39/72d14ca9178a9f7e2d4188fbb082ef26f19b8b4d280882033019ab7c4fb3/discovery_core-3.0.1-py38-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "61ae288e7cdded8d27127ce17038224572fb3d469ba7ea23c94dde3dcb694f97",
"md5": "1ec7628d798e14bf1c4a94a46cf83349",
"sha256": "db0f5a773cc427a6bb396d21438d657e011e4d69098d349e6dccad07dc80f575"
},
"downloads": -1,
"filename": "discovery_core-3.0.1.tar.gz",
"has_sig": false,
"md5_digest": "1ec7628d798e14bf1c4a94a46cf83349",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 86424,
"upload_time": "2024-07-10T18:01:47",
"upload_time_iso_8601": "2024-07-10T18:01:47.869348Z",
"url": "https://files.pythonhosted.org/packages/61/ae/288e7cdded8d27127ce17038224572fb3d469ba7ea23c94dde3dcb694f97/discovery_core-3.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-10 18:01:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gigas64",
"github_project": "discovery-core",
"github_not_found": true,
"lcname": "discovery-core"
}
JSON
