qoa4ml


Nameqoa4ml JSON
Version 0.0.82 PyPI version JSON
download
home_pagehttps://rdsea.github.io/
SummaryQuality of Analysis for Machine Learning
upload_time2023-05-26 11:03:14
maintainer
docs_urlNone
authorAaltosea
requires_python
licenseApache License 2.0
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # QoA4ML - Quality of Analytics for ML

## Source code
https://github.com/rdsea/QoA4ML

## Probes
* [QoA4ML Probes](https://github.com/rdsea/QoA4ML/tree/main/qoa4ml_lib/qoa4ml/probes.py): libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML code
* Probe properties:
  - Can be written in different languages (Python, GoLang)
  - Can have different communications to monitoring systems (depending on probes and its ML support)
  - Capture metrics with a clear definition/scope
    - e.g., Response time for an ML stage (training) or a service call (of ML APIs)
    - Thus output of probes must be correlated to objects to be monitored and the tenant
  - Support high or low-level metrics/attributes
    - depending on probes implementation
  - Can be instrumented into source code or standlone

Provide some metric classes for collecting different types of metric: Counter, Gauge, Summary, Histogram

- `Metric`: an original class providing some common functions on an metric object.
    - Attribute:
        - `metric_name`
        - `description`
        - `value`
    - Function:
        - `__init__`: let user define the metric name, description and default value.
        - `set`: set its `value` to a specific value
        - `get_val`: get current value
        - `get_name`: return metric name 
        - `get_des`: return metric description 
        - `__str__`: return information about the metric in form of string
        - `to_dict`: return information about the metric in form of dictionary
- `Counter`
    - Attribute: same as `Metric` & on further developing
    - Function:
        - `inc`: increase the value of the metric by the given number/by 1 by default.
        - `reset`: set the value back to zero.
- `Gauge`
    - Attribute: same as `Metric` & on further developing
    - Function:
        - `inc`: increase the value of the metric by a given number/by 1 by default.
        - `dec`: decrease the value of the metric by a given number/by 1 by default.
        - `set`: set the value to a given number.
- `Summary`
    - Attribute: same as `Metric` & on further developing
    - Function:
        - `inc`: increase the value of the metric by a given number/by 1 by default.
        - `dec`: decrease the value of the metric by a given number/by 1 by default.
        - `set`: set the value to a given number.
- `Histogram`
    - Attribute: same as `Metric` & on further developing
    - Function:
        - `inc`: increase the value of the metric by a given number/by 1 by default.
        - `dec`: decrease the value of the metric by a given number/by 1 by default.
        - `set`: set the value to a given number.

## [QoA4ML Reports](https://github.com/rdsea/QoA4ML/blob/main/qoa4ml_lib/qoa4ml/reports.py)

This module defines ``QoA_Report``, an object that provide functions to export monitored metric to the following schema:
```json
{
    "execution_graph":{
        "instances":{
            "@instance_id":{
                "instance_name": "@name_of_instance",
                "method": "@method/task/function",
                "previous_instance":["@list_of_previous_instance"]
            },
            ...
        },
        "last_instance": "@name_of_last_instance_in_the_graph"
    },
    "quality":{
        "data":{
            "@stage_id":{
                "@metric_name":{
                    "@instance_id": "@value"
                }
            }
        },
        "service":{
            "@stage_id":{
                "@metric_name":{
                    "@instance_id": "@value"
                }
            }
        },
        "inference":{
            "@inference_id":{
                "value": "@value",
                "confident": "@confidence",
                "accuracy": "@accuracy",
                "instance_id": "@instance_id",
                "source": ["@list_of_inferences_to_infer_this_inference"]
            }
        }
    }
}
```

The example is shown in `example/reports/qoa_report/example.txt`

- Attribute:
    - `previous_report_instance` = list previous services
    - `report_list`: list of reports from previous services
    - `previous_inference`: list previous inferences
    - `quality_report`: report all quality (data, service, inference qualtiy) of the service
    - `execution_graph`: report the execution graph
    - `report`: the final report to be sent

- Function:
    - `__init__`: init as empty report.
    - `import_report_from_file`: init QoA Report from `json` file.
    - `import_pReport`: import reports from previous service to build the execution and inference graph
    - `build_execution_graph`: build execution graph from list of previous reports
    - `build_quality_report`: build the quality report from metrics collected in runtime
    - `generate_report`: return the final report.
    - `observe_metric`: observe metrics in runtime with 3 categories: service quality, data quality, inference qualtiy. This can be extended to observe resource metrics.


## [Examples](https://github.com/rdsea/QoA4ML/tree/main/example)
https://github.com/rdsea/QoA4ML/tree/main/example




## Overview
![Class](../img/class.png)

Probes will be integrated to client program or system service to collect metrics at the edge
Probes will generate reports and sent to message broker using different connector. Coresponding collector should be used to acquire the metrics.

## Collector
The manager/orchestrator have to integrate collector to collect metric using different protocols for further analysis.
- Attribute:
- Function:
    - `__init__`: take a configuration as a `dict` containing information about the data source, eg. broker, channel, queue, etc. It can take an `object` as an attribute `host` to return the message for further processing.


    - If the collector is initiated by an object inherited class, this class must implement `message_processing` function to process the message returned by the collector. Otherwise, the collector will print the message to the console.

    - `on_request`: handle message from data source (message broker,...)

    - `start` & `stop`: start and stop consuming message

    - `get_queue`: return the queue name.

## Connector
Connectors are implement with different protocols for sending report. Example: sending report to message broker - AMQP/MQTT
- Attribute:
- Function:
    - `__init__`: take a configuration as a `dict` containing information about the data sink, eg. broker, channel, queue, etc. It can take a `bool` parameter `log` for logging messages for further processing.

    - `send_data`: a function to send data to specified `routing_key`/`queue` with a corresponding key `corr_id` to trace back message.



## Utilities
A module provide some frequently used functions and some function to directly collect system metrics.



Change Log
======================

0.0.13 (18/04/2022)
---------------------
First Release 

0.0.18 (10/05/2022)
---------------------
Update system metric

0.0.19 (31/05/2022)
---------------------
Update process metric

0.0.54 (20/09/2022)
---------------------
Update monitoring system/process/docker

0.0.62 (20/09/2022)
---------------------
Add metric and modify format of system/process reports

0.0.64 (15/03/2022)
---------------------
Add metric and modify format of system/process reports

0.0.72 (22/05/2023)
---------------------
Refactor source code
            

Raw data

            {
    "_id": null,
    "home_page": "https://rdsea.github.io/",
    "name": "qoa4ml",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "Aaltosea",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/60/71/a5f01df3127b00d66a68fa0d481044e769cc99caba9b061968cc2c49455b/qoa4ml-0.0.82.tar.gz",
    "platform": null,
    "description": "# QoA4ML - Quality of Analytics for ML\n\n## Source code\nhttps://github.com/rdsea/QoA4ML\n\n## Probes\n* [QoA4ML Probes](https://github.com/rdsea/QoA4ML/tree/main/qoa4ml_lib/qoa4ml/probes.py): libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML code\n* Probe properties:\n  - Can be written in different languages (Python, GoLang)\n  - Can have different communications to monitoring systems (depending on probes and its ML support)\n  - Capture metrics with a clear definition/scope\n    - e.g., Response time for an ML stage (training) or a service call (of ML APIs)\n    - Thus output of probes must be correlated to objects to be monitored and the tenant\n  - Support high or low-level metrics/attributes\n    - depending on probes implementation\n  - Can be instrumented into source code or standlone\n\nProvide some metric classes for collecting different types of metric: Counter, Gauge, Summary, Histogram\n\n- `Metric`: an original class providing some common functions on an metric object.\n    - Attribute:\n        - `metric_name`\n        - `description`\n        - `value`\n    - Function:\n        - `__init__`: let user define the metric name, description and default value.\n        - `set`: set its `value` to a specific value\n        - `get_val`: get current value\n        - `get_name`: return metric name \n        - `get_des`: return metric description \n        - `__str__`: return information about the metric in form of string\n        - `to_dict`: return information about the metric in form of dictionary\n- `Counter`\n    - Attribute: same as `Metric` & on further developing\n    - Function:\n        - `inc`: increase the value of the metric by the given number/by 1 by default.\n        - `reset`: set the value back to zero.\n- `Gauge`\n    - Attribute: same as `Metric` & on further developing\n    - Function:\n        - `inc`: increase the value of the metric by a given number/by 1 by default.\n        - `dec`: decrease the value of the metric by a given number/by 1 by default.\n        - `set`: set the value to a given number.\n- `Summary`\n    - Attribute: same as `Metric` & on further developing\n    - Function:\n        - `inc`: increase the value of the metric by a given number/by 1 by default.\n        - `dec`: decrease the value of the metric by a given number/by 1 by default.\n        - `set`: set the value to a given number.\n- `Histogram`\n    - Attribute: same as `Metric` & on further developing\n    - Function:\n        - `inc`: increase the value of the metric by a given number/by 1 by default.\n        - `dec`: decrease the value of the metric by a given number/by 1 by default.\n        - `set`: set the value to a given number.\n\n## [QoA4ML Reports](https://github.com/rdsea/QoA4ML/blob/main/qoa4ml_lib/qoa4ml/reports.py)\n\nThis module defines ``QoA_Report``, an object that provide functions to export monitored metric to the following schema:\n```json\n{\n    \"execution_graph\":{\n        \"instances\":{\n            \"@instance_id\":{\n                \"instance_name\": \"@name_of_instance\",\n                \"method\": \"@method/task/function\",\n                \"previous_instance\":[\"@list_of_previous_instance\"]\n            },\n            ...\n        },\n        \"last_instance\": \"@name_of_last_instance_in_the_graph\"\n    },\n    \"quality\":{\n        \"data\":{\n            \"@stage_id\":{\n                \"@metric_name\":{\n                    \"@instance_id\": \"@value\"\n                }\n            }\n        },\n        \"service\":{\n            \"@stage_id\":{\n                \"@metric_name\":{\n                    \"@instance_id\": \"@value\"\n                }\n            }\n        },\n        \"inference\":{\n            \"@inference_id\":{\n                \"value\": \"@value\",\n                \"confident\": \"@confidence\",\n                \"accuracy\": \"@accuracy\",\n                \"instance_id\": \"@instance_id\",\n                \"source\": [\"@list_of_inferences_to_infer_this_inference\"]\n            }\n        }\n    }\n}\n```\n\nThe example is shown in `example/reports/qoa_report/example.txt`\n\n- Attribute:\n    - `previous_report_instance` = list previous services\n    - `report_list`: list of reports from previous services\n    - `previous_inference`: list previous inferences\n    - `quality_report`: report all quality (data, service, inference qualtiy) of the service\n    - `execution_graph`: report the execution graph\n    - `report`: the final report to be sent\n\n- Function:\n    - `__init__`: init as empty report.\n    - `import_report_from_file`: init QoA Report from `json` file.\n    - `import_pReport`: import reports from previous service to build the execution and inference graph\n    - `build_execution_graph`: build execution graph from list of previous reports\n    - `build_quality_report`: build the quality report from metrics collected in runtime\n    - `generate_report`: return the final report.\n    - `observe_metric`: observe metrics in runtime with 3 categories: service quality, data quality, inference qualtiy. This can be extended to observe resource metrics.\n\n\n## [Examples](https://github.com/rdsea/QoA4ML/tree/main/example)\nhttps://github.com/rdsea/QoA4ML/tree/main/example\n\n\n\n\n## Overview\n![Class](../img/class.png)\n\nProbes will be integrated to client program or system service to collect metrics at the edge\nProbes will generate reports and sent to message broker using different connector. Coresponding collector should be used to acquire the metrics.\n\n## Collector\nThe manager/orchestrator have to integrate collector to collect metric using different protocols for further analysis.\n- Attribute:\n- Function:\n    - `__init__`: take a configuration as a `dict` containing information about the data source, eg. broker, channel, queue, etc. It can take an `object` as an attribute `host` to return the message for further processing.\n\n\n    - If the collector is initiated by an object inherited class, this class must implement `message_processing` function to process the message returned by the collector. Otherwise, the collector will print the message to the console.\n\n    - `on_request`: handle message from data source (message broker,...)\n\n    - `start` & `stop`: start and stop consuming message\n\n    - `get_queue`: return the queue name.\n\n## Connector\nConnectors are implement with different protocols for sending report. Example: sending report to message broker - AMQP/MQTT\n- Attribute:\n- Function:\n    - `__init__`: take a configuration as a `dict` containing information about the data sink, eg. broker, channel, queue, etc. It can take a `bool` parameter `log` for logging messages for further processing.\n\n    - `send_data`: a function to send data to specified `routing_key`/`queue` with a corresponding key `corr_id` to trace back message.\n\n\n\n## Utilities\nA module provide some frequently used functions and some function to directly collect system metrics.\n\n\n\nChange Log\n======================\n\n0.0.13 (18/04/2022)\n---------------------\nFirst Release \n\n0.0.18 (10/05/2022)\n---------------------\nUpdate system metric\n\n0.0.19 (31/05/2022)\n---------------------\nUpdate process metric\n\n0.0.54 (20/09/2022)\n---------------------\nUpdate monitoring system/process/docker\n\n0.0.62 (20/09/2022)\n---------------------\nAdd metric and modify format of system/process reports\n\n0.0.64 (15/03/2022)\n---------------------\nAdd metric and modify format of system/process reports\n\n0.0.72 (22/05/2023)\n---------------------\nRefactor source code",
    "bugtrack_url": null,
    "license": "Apache License 2.0",
    "summary": "Quality of Analysis for Machine Learning",
    "version": "0.0.82",
    "project_urls": {
        "Homepage": "https://rdsea.github.io/"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "6071a5f01df3127b00d66a68fa0d481044e769cc99caba9b061968cc2c49455b",
                "md5": "bb50125037e7ad2c0c39afa51080e8d1",
                "sha256": "f9a0231f067080b5f4cc6be390e9c6eac863e58330f7ca218c4f76eb0677f54b"
            },
            "downloads": -1,
            "filename": "qoa4ml-0.0.82.tar.gz",
            "has_sig": false,
            "md5_digest": "bb50125037e7ad2c0c39afa51080e8d1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 16533,
            "upload_time": "2023-05-26T11:03:14",
            "upload_time_iso_8601": "2023-05-26T11:03:14.125206Z",
            "url": "https://files.pythonhosted.org/packages/60/71/a5f01df3127b00d66a68fa0d481044e769cc99caba9b061968cc2c49455b/qoa4ml-0.0.82.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-05-26 11:03:14",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "qoa4ml"
}
        
Elapsed time: 0.14198s