# Loadero-Python
Python client for managing [Loadero](https://loadero.com) tests.
Loadero-Python provides easy-to-use programmatic access to Loadero API. Allows
to manage tests, participants, asserts, runs and other Loadero resources, start
and stop tests and extract test run results. Example usage might be running
Loadero tests as a part of CI/CD.
## Table of Contents
<!-- generate with https://luciopaiva.com/markdown-toc/ -->
- [Installation](#installation)
- [Usage](#usage)
- [API Access](#api-access)
- [Initialization](#initialization)
- [Working with Existing Resources](#working-with-existing-resources)
- [Creating a Test](#creating-a-test)
- [Running a Test](#running-a-test)
- [Polling](#polling)
- [Stopping Test Execution](#stopping-test-execution)
- [Getting Results](#getting-results)
- [Participant Results](#participant-results)
- [Log Retrieval](#log-retrieval)
- [Extracting Failed Asserts](#extracting-failed-asserts)
- [Checking metrics](#checking-metrics)
- [Filtering and Pagination](#filtering-and-pagination)
- [Filtering](#filtering)
- [Pagination](#pagination)
- [Structure](#structure)
- [API client](#api-client)
- [Resources](#resources)
- [Resource Params](#resource-params)
- [Constants](#constants)
- [Resource API](#resource-api)
- [Contributing](#contributing)
## Installation
Loadero-Python is available on [PyPI](https://pypi.org/project/loadero-python/)
and can be installed by running
```bash
pip install loadero-python
```
## Usage
### API Access
Before using the Loadero-Python client an API access token needs to be acquired.
This can be done in the projects settings page in Loadero web-app.
More information about the API can be found in the
[Loadero wiki](https://wiki.loadero.com/docs/loadero-usage/api/direct-api-access)
**Note** Access tokens are project specific and cannot be used across multiple
projects. Make sure to specify the project ID in the request for an access
token.
### Initialization
After acquiring the access token Loadero-Python needs to be initialized with it.
Loadero-Python uses a singleton object `APIClient` from
`loadero_python.api_client` module to perform all requests to the API. Since
`APIClient` is a singleton it needs to be initialized once like so
```py
from loadero_python.api_client import APIClient
APIClient(
project_id=1234,
access_token="your_access_token",
)
```
Further examples will not include `APIClient` initialization. It is assumed that
the client has been initialized at an earlier step.
### Working with Existing Resources
Loadero resources have a tree-like structure hence there can be child and parent
resources.
```
project
|
|----tests----groups-----participants
| |
| |------asserts----assert_preconditions
|
|----files
|
|----runs-----results
```
Every parent resource can read all of its child resources.
`Project` class from `loadero_python.resources.project` module provides an entry
point to access all resources.
`Project` class can be imported with
```py
from loadero_python.resources.project import Project
```
All tests in a project can be read with
```py
tests, pagination, filters = Project().tests()
```
Notice, that project ID was not specified in this example, this is because the
`APIClient` has already been initialized with the project ID and corresponding
access token.
- `tests` is a list of `Test` objects from `loadero_python.resources.test`
module
- `pagination` is a `PaginationParams` object from
`loadero_python.resources.pagination` module.
- `filters` is a python dictionary of applied filters.
A more detailed explanation of `pagination` and `filters` return values is
available in the [Filtering and Pagination](#filtering-and-pagination) section.
### Creating a Test
With an initialized `APIClient` Loadero-Python can now manage resources in the
project. Test is a resource one of many in Loadero-Python. More information
about all the resources that Loadero-Python provides can be found in the
[Structure](#structure) section. This usage guide cannot demonstrate all of
Loadero-Python's functionality hence will cover only common use case scenarios
starting with creating a test.
Test resource is contained within the `loadero_python.resources.test` module.
From it `Test`, `TestParams`, and `Script` classes need to be imported.
```py
from loadero_python.resources.test import Test, TestParams, Script
```
Additionally, `TestMode` and `IncrementStrategy` classificator constant
enumerations need to be imported from `loadero_python.resources.classificator`.
They will be used for test attribute definitions.
```py
from loadero_python.resources.classificator import TestMode, IncrementStrategy
```
Test attributes can be specified in two ways. Directly as arguments in params
initialization.
```py
test = Test(
params=TestParams(
name="my first loadero python test",
start_interval=1,
participant_timeout=10 * 60, # ten minutes
mode=TestMode.TM_LOAD,
increment_strategy=IncrementStrategy.IS_LINEAR,
script=Script(content='print("hello test script")'),
)
).create()
```
or with builder methods.
```py
test = Test(
params=TestParams()
.with_name("my second loadero python test")
.with_start_interval(1)
.with_participant_timeout(10 * 60) # ten minutes
.with_mode(TestMode.TM_PERFORMANCE)
.with_increment_strategy(IncrementStrategy.IS_RANDOM)
.with_script(Script(content='print("hello test script")'))
).create()
```
Resource create and update operations have required and optional attributes. If
a required attribute is missing the API call will fail. Loadero-Python checks if
all of the required attributes have been populated before making the API call
and raises an exception if one or more required attributes are missing.
For test resources, the required attributes are:
- `name`
- `start_interval`
- `participant_timeout`
- `mode`
- `increment_strategy`
After the create operation completes the `test` object will have a few more of
its attributes populated. Any resource attributes can be simply printed.
```py
print(test)
```
This will output a JSON object representation of the resource.
```json
{
"id": 1234,
"name": "my first loadero python test",
"start_interval": 1,
"participant_timeout": 600,
"mode": "load",
"increment_strategy": "linear",
"script": "print(\"hello test script\")",
"created": "2022-08-25 15:33:04+00:00",
"updated": "2022-08-25 15:33:04+00:00",
"script_file_id": 12345
}
```
Different output formats can be achieved by using `to_dict` and `to_dict_full`
resource params methods. Both methods return a python dictionary representation
of the resource. `to_dict` will return only the required attributes and optional
attributes if present. `to_dict_full` will return all attributes present.
**Note** `to_dict` will raise an exception if one or more required attribute is
missing.
```py
import yaml
print(yaml.dump(test.params.to_dict_full()))
```
```yaml
created: "2022-08-25 15:33:04+00:00"
id: 1234
increment_strategy: linear
mode: load
name: my first loadero python test
participant_timeout: 600
script: print("hello test script")
script_file_id: 12345
start_interval: 1
updated: "2022-08-25 15:33:04+00:00"
```
### Running a Test
To run a test the only required attribute is test ID.
For the `test` object from previous examples, the `test_id` attribute has been
populated by create operation, so it can simply be run by calling the `launch`
method.
```py
run = test.launch()
```
If a test ID is known it can be run directly.
```py
run = Test(test_id=1234).launch()
```
All tests in a project can be run with
```py
for test in Project().tests()[0]:
test.launch()
```
#### Polling
After a test has been launched, waiting for the test to finish can be achieved
with the `poll` method.
```py
run.poll()
```
By default, the `poll` method will make an API call to check if the test
execution has finished every 15 seconds and will wait up to 12 hours. This
functionality can be customized with the `interval` and `timeout` arguments.
```py
# will poll every 5 seconds and will wait up to 10 minutes.
run.poll(interval=5.0, timeout=10 * 60.0)
```
If test execution does not finish within the specified timeout, the `poll`
method will raise an exception.
#### Stopping Test Execution
If test execution needs to be prematurely stopped, it can be done with the
`stop` method.
```py
run.stop()
```
**Note** `stop` only sends an API request that starts a Loadero procedure of
stopping the test. This process is NOT immediate. Even though the `stop` API
request completes relatively quickly, the test can remain running for a while
longer.
**Note** If another process is polling the test execution, it will automatically
stop if the test is stopped.
### Getting Results
After the test run finishes execution, the `run` object already contains many
useful attributes that may be used in result analysis. The attributes are stored
on the `run.params` field. `run.params` is an `RunParams` object from
`loadero_python.resources.run` module.
```py
print(run.params.success_rate)
```
#### Participant Results
`run` object describes a result overview of the whole test. To get a more
detailed result information about each test participant's results needs to be
read.
```py
results, _, _ = run.Results()
result = results[0]
```
The ignored return values are pagination and filters. They are not relevant for
result retrieval, hence they are omitted. A more detailed explanation of these
values is available in the [Filtering and Pagination](#filtering-and-pagination)
section.
`results` is a list of `Result` objects from `loadero_python.resources.result`
module. A single result corresponds to a single participant in the test.
`Result` just like a regular resource object has a `params` field of type
`ResultParams` that contains its attributes. The result resource has the largest
amount of attributes, so this showcase will cover only common use cases.
##### Log Retrieval
```py
import requests
resp = requests.get(result.params.log_paths.selenium)
if not resp:
print("failed to download selenium log")
exit(1)
with open(f"selenium_log_of_result_{result.params.result_id}", "w") as f:
f.write(resp.text)
```
`result.params.log_paths.selenium` is an URL to an Selenium log. It first needs
to be downloaded using the HTTP library `requests`. Then it can be written to a
file.
##### Extracting Failed Asserts
Before extracting failed asserts. `AssertStatus` classificator constant
enumeration needs to be imported.
```py
from loadero_python.resources.classificator import AssertStatus
```
```py
failed_asserts = []
for result_assert in result.params.asserts:
if result_assert.status == AssertStatus.AS_FAIL
failed_asserts.append(result_assert)
```
##### Checking metrics
Loadero tests collect various different metrics from CPU, RAM, and network usage
to video and audio quality indicators. Loadero organizes these metrics with
metric base paths - a path-like string that uniquely identifies metric data. For
example, CPU usage metric data is described by the metric base path
`machine/cpu/used`.
After test execution finishes Loadero processes the collected metric data by
applying aggregator functions.
- total
- minimum
- maximum
- average
- standard deviation
- relative standard deviation
- 1st percentile
- 5th percentile
- 25th percentile
- 50th percentile
- 75th percentile
- 95th percentile
- 99th percentile
The result is a single float value identified by a metric path. For example, the
maximal CPU usage is described by the metric path - `machine/cpu/used/max`
In Loadero-Python metric base paths - `MetricBasePath` and metric paths -
`MetricPath` are constant enumerations of all the available metric and metric
base paths. Contained within the `loadero_python.resources.metric_path` module.
To access a specific metric `MetricBasePath` enumeration needs to be imported.
```py
from loadero_python.resources.metric_path import MetricBasePath
```
Then a specific metric can be checked like this
```py
if result.params.metrics is None or result.metrics.machine is None:
print("result has no machine metrics")
exit(1)
if MetricBasePath.MACHINE_CPU_AVAILABLE not in result.params.metrics.machine:
print("result has no machine cpu available metric")
exit(1)
if (
result.params.metrics.machine[MetricBasePath.MACHINE_CPU_AVAILABLE].average
< 10.0
):
print("test is well configured. efficient usage of CPU resources")
```
The `not None` checks are required because some or all metrics for a result can
be missing. For example, non-WebRTC tests will not have any `webrtc` metrics.
### Filtering and Pagination
Read-all operations have the option to limit the number of resources returned,
offset a limited read-all operation by some amount of resources and filter out
undesired resources.
This is done by passing a query params argument when performing a read-all
operation.
`QueryParams` class is contained in `loadero_python.resources.resource` module
and can be imported with
```py
from loadero_python.resources.resource import QueryParams
```
#### Filtering
Filters are resource-specific and are defined in each resource module. For
example, test resource filters are defined in the `TestFilterKey` constant
enumeration in the `loadero_python.resources.test` module.
`TestFilterKey` can be imported with
```py
from loadero_python.resources.test import TestFilterKey
```
Now test read all operations can be filtered like this
```py
tests, _, filters = Project().tests(
query_params=QueryParams().filter(
TestFilterKey.PARTICIPANT_TIMEOUT_TO, 10 * 60 # ten minutes
)
)
```
The ignored value is pagination. It can be ignored because pagination will
contain useful page information when limit and offset query_params have been
applied.
This will return tests whose participant timeout attribute is smaller than ten
minutes.
`filters` is a python dictionary with the applied filters.
#### Pagination
When performing a read-all operation that will return many resources it is good
practice to limit the number of resources returned and perform multiple smaller
reads. This can be achieved by limiting and offsetting the number of resources
returned.
```py
tests, pagination, _ = Project().tests(
query_params=QueryParams().limit(20).offset(10)
)
```
This time the ignored value is filters. It can be ignored because no filters
were applied.
Let's assume that the project has 28 tests numbered from 1 to 28, then this
read-all operation would return tests with numbers from 11 to 28. This happens
because the returned resources were offset by 10 and the next resource after the
10th is the 11th. Only 18 resources were returned because the remaining
resources after offset were smaller than the defined limit - 20.
`pagination` is an instance `PaginationParams` class from
`loadero_python.resources.resource` module. It contains information about the
applied limit and offset, plus additional information describing how many
resources remain to be read.
## Structure
The Loadero-Python library structure is similar to the Loadero API structure.
The main structural components are:
- API client
- resources and operations with them
### API client
Contained within `loadero_python.api_client` module is the `APIClient` singleton
object. It needs to be initialized once with the project ID and access token.
All requests to Loadero API are done with the `APIClient` object. It adds the
required headers to make valid API requests. Additionally, the `APIClient` rate
limits all requests to be compliant with Loadero API's access limits. Rate
limiting can be opted out on initialization.
```py
APIClient(
project_id=1234,
access_token="your_access_token",
rate_limit=False,
)
```
### Resources
Each resource has a separate module.
| Resource class | Module |
| -------------------- | ---------------------------------------------- |
| `AssertPrecondition` | `loadero_python.resources.assert_precondition` |
| `Assert` | `loadero_python.resources.assert_resource` |
| `File` | `loadero_python.resources.file` |
| `Group` | `loadero_python.resources.group` |
| `Participant` | `loadero_python.resources.participant` |
| `Project` | `loadero_python.resources.project` |
| `Result` | `loadero_python.resources.result` |
| `RunParticipant` | `loadero_python.resources.run_participant` |
| `Run` | `loadero_python.resources.run` |
| `Test` | `loadero_python.resources.test` |
All resource classes have a similar structure:
- Resource classes have an attribute `params` that is used to store the data of
a single instance of the resource. Read more about resource params
[here](#resource-params).
- most resources implement common CRUD manipulation methods - `create`, `read`,
`update`, `delete`, `duplicate`. Some resources do not have these methods
because they are impossible or not available via API access. For example,
`Project` resource has only `read` method because API access prohibits updates
to this resource.
- resources that can have child resources have methods for reading all the child
resources.
```py
# reads all groups in test
groups, _, _ = Test(test_id=123).groups()
```
### Resource Params
Every resource has a resource params class that stores the data of a single
resource instance.
Resource params class for each resource is available in the resources module.
| Resource params class | Module |
| -------------------------- | ---------------------------------------------- |
| `AssertPreconditionParams` | `loadero_python.resources.assert_precondition` |
| `AssertParams` | `loadero_python.resources.assert_resource` |
| `FileParams` | `loadero_python.resources.file` |
| `GroupParams` | `loadero_python.resources.group` |
| `ParticipantParams` | `loadero_python.resources.participant` |
| `ProjectParams` | `loadero_python.resources.project` |
| `ResultParams` | `loadero_python.resources.result` |
| `RunParticipantParams` | `loadero_python.resources.run_participant` |
| `RunParams` | `loadero_python.resources.run` |
| `TestParams` | `loadero_python.resources.test` |
Resource params classes provide access to the resource attributes.
```py
# read a test and print its name
print(Test(test_id=123).read().params.name)
```
### Constants
Loadero-Python has two modules for constants.
- `loadero_python.resources.classificator` for classificator constant
enumerations.
- `loadero_python.resources.metric_path` for metric path and metric base path
constant enumerations.
### Resource API
Every resource has its own API class also available in the resources module.
| Resource class | Module |
| ----------------------- | ---------------------------------------------- |
| `AssertPreconditionAPI` | `loadero_python.resources.assert_precondition` |
| `AssertAPI` | `loadero_python.resources.assert_resource` |
| `FileAPI` | `loadero_python.resources.file` |
| `GroupAPI` | `loadero_python.resources.group` |
| `ParticipantAPI` | `loadero_python.resources.participant` |
| `ProjectAPI` | `loadero_python.resources.project` |
| `ResultAPI` | `loadero_python.resources.result` |
| `RunParticipantAPI` | `loadero_python.resources.run_participant` |
| `RunAPI` | `loadero_python.resources.run` |
| `TestAPI` | `loadero_python.resources.test` |
Resource API class implements all the available API operations of that resource.
Resource API classes are internally used by all resources, but the class on its
own is not very useful.
## Contributing
Found a bug? - Feel free to open an
[issue](https://github.com/loadero/loadero-python/issues/new/choose).
Would like to request a feature? - Open an issue describing the request and the
reason for it or contact Loadero [support](mailto:support@loadero.com).
Want to contribute? - Open
[issues](https://github.com/loadero/loadero-python/issues) is a good place where
to find stuff that needs to be worked on.
Raw data
{
"_id": null,
"home_page": "https://github.com/loadero/loadero-python",
"name": "loadero-python",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7, <4",
"maintainer_email": "",
"keywords": "loadero",
"author": "Loadero Team",
"author_email": "support@loadero.com",
"download_url": "https://files.pythonhosted.org/packages/51/c4/7e880b48aef39bc3bef49f99cf20272ff8fe93c62201e51bc16cd7e7842a/loadero_python-1.2.2.tar.gz",
"platform": null,
"description": "# Loadero-Python\n\nPython client for managing [Loadero](https://loadero.com) tests.\n\nLoadero-Python provides easy-to-use programmatic access to Loadero API. Allows\nto manage tests, participants, asserts, runs and other Loadero resources, start\nand stop tests and extract test run results. Example usage might be running\nLoadero tests as a part of CI/CD.\n\n## Table of Contents\n\n<!-- generate with https://luciopaiva.com/markdown-toc/ -->\n\n- [Installation](#installation)\n- [Usage](#usage)\n - [API Access](#api-access)\n - [Initialization](#initialization)\n - [Working with Existing Resources](#working-with-existing-resources)\n - [Creating a Test](#creating-a-test)\n - [Running a Test](#running-a-test)\n - [Polling](#polling)\n - [Stopping Test Execution](#stopping-test-execution)\n - [Getting Results](#getting-results)\n - [Participant Results](#participant-results)\n - [Log Retrieval](#log-retrieval)\n - [Extracting Failed Asserts](#extracting-failed-asserts)\n - [Checking metrics](#checking-metrics)\n - [Filtering and Pagination](#filtering-and-pagination)\n - [Filtering](#filtering)\n - [Pagination](#pagination)\n- [Structure](#structure)\n - [API client](#api-client)\n - [Resources](#resources)\n - [Resource Params](#resource-params)\n - [Constants](#constants)\n - [Resource API](#resource-api)\n- [Contributing](#contributing)\n\n## Installation\n\nLoadero-Python is available on [PyPI](https://pypi.org/project/loadero-python/)\nand can be installed by running\n\n```bash\npip install loadero-python\n```\n\n## Usage\n\n### API Access\n\nBefore using the Loadero-Python client an API access token needs to be acquired.\nThis can be done in the projects settings page in Loadero web-app.\nMore information about the API can be found in the\n[Loadero wiki](https://wiki.loadero.com/docs/loadero-usage/api/direct-api-access)\n\n**Note** Access tokens are project specific and cannot be used across multiple\nprojects. Make sure to specify the project ID in the request for an access\ntoken.\n\n### Initialization\n\nAfter acquiring the access token Loadero-Python needs to be initialized with it.\nLoadero-Python uses a singleton object `APIClient` from\n`loadero_python.api_client` module to perform all requests to the API. Since\n`APIClient` is a singleton it needs to be initialized once like so\n\n```py\nfrom loadero_python.api_client import APIClient\n\nAPIClient(\n project_id=1234,\n access_token=\"your_access_token\",\n)\n```\n\nFurther examples will not include `APIClient` initialization. It is assumed that\nthe client has been initialized at an earlier step.\n\n### Working with Existing Resources\n\nLoadero resources have a tree-like structure hence there can be child and parent\nresources.\n\n```\nproject\n |\n |----tests----groups-----participants\n | |\n | |------asserts----assert_preconditions\n |\n |----files\n |\n |----runs-----results\n```\n\nEvery parent resource can read all of its child resources.\n\n`Project` class from `loadero_python.resources.project` module provides an entry\npoint to access all resources.\n\n`Project` class can be imported with\n\n```py\nfrom loadero_python.resources.project import Project\n```\n\nAll tests in a project can be read with\n\n```py\ntests, pagination, filters = Project().tests()\n```\n\nNotice, that project ID was not specified in this example, this is because the\n`APIClient` has already been initialized with the project ID and corresponding\naccess token.\n\n- `tests` is a list of `Test` objects from `loadero_python.resources.test`\n module\n- `pagination` is a `PaginationParams` object from\n `loadero_python.resources.pagination` module.\n- `filters` is a python dictionary of applied filters.\n\nA more detailed explanation of `pagination` and `filters` return values is\navailable in the [Filtering and Pagination](#filtering-and-pagination) section.\n\n### Creating a Test\n\nWith an initialized `APIClient` Loadero-Python can now manage resources in the\nproject. Test is a resource one of many in Loadero-Python. More information\nabout all the resources that Loadero-Python provides can be found in the\n[Structure](#structure) section. This usage guide cannot demonstrate all of\nLoadero-Python's functionality hence will cover only common use case scenarios\nstarting with creating a test.\n\nTest resource is contained within the `loadero_python.resources.test` module.\nFrom it `Test`, `TestParams`, and `Script` classes need to be imported.\n\n```py\nfrom loadero_python.resources.test import Test, TestParams, Script\n```\n\nAdditionally, `TestMode` and `IncrementStrategy` classificator constant\nenumerations need to be imported from `loadero_python.resources.classificator`.\nThey will be used for test attribute definitions.\n\n```py\nfrom loadero_python.resources.classificator import TestMode, IncrementStrategy\n```\n\nTest attributes can be specified in two ways. Directly as arguments in params\ninitialization.\n\n```py\ntest = Test(\n params=TestParams(\n name=\"my first loadero python test\",\n start_interval=1,\n participant_timeout=10 * 60, # ten minutes\n mode=TestMode.TM_LOAD,\n increment_strategy=IncrementStrategy.IS_LINEAR,\n script=Script(content='print(\"hello test script\")'),\n )\n).create()\n```\n\nor with builder methods.\n\n```py\ntest = Test(\n params=TestParams()\n .with_name(\"my second loadero python test\")\n .with_start_interval(1)\n .with_participant_timeout(10 * 60) # ten minutes\n .with_mode(TestMode.TM_PERFORMANCE)\n .with_increment_strategy(IncrementStrategy.IS_RANDOM)\n .with_script(Script(content='print(\"hello test script\")'))\n).create()\n```\n\nResource create and update operations have required and optional attributes. If\na required attribute is missing the API call will fail. Loadero-Python checks if\nall of the required attributes have been populated before making the API call\nand raises an exception if one or more required attributes are missing.\n\nFor test resources, the required attributes are:\n\n- `name`\n- `start_interval`\n- `participant_timeout`\n- `mode`\n- `increment_strategy`\n\nAfter the create operation completes the `test` object will have a few more of\nits attributes populated. Any resource attributes can be simply printed.\n\n```py\nprint(test)\n```\n\nThis will output a JSON object representation of the resource.\n\n```json\n{\n \"id\": 1234,\n \"name\": \"my first loadero python test\",\n \"start_interval\": 1,\n \"participant_timeout\": 600,\n \"mode\": \"load\",\n \"increment_strategy\": \"linear\",\n \"script\": \"print(\\\"hello test script\\\")\",\n \"created\": \"2022-08-25 15:33:04+00:00\",\n \"updated\": \"2022-08-25 15:33:04+00:00\",\n \"script_file_id\": 12345\n}\n```\n\nDifferent output formats can be achieved by using `to_dict` and `to_dict_full`\nresource params methods. Both methods return a python dictionary representation\nof the resource. `to_dict` will return only the required attributes and optional\nattributes if present. `to_dict_full` will return all attributes present.\n**Note** `to_dict` will raise an exception if one or more required attribute is\nmissing.\n\n```py\nimport yaml\n\nprint(yaml.dump(test.params.to_dict_full()))\n```\n\n```yaml\ncreated: \"2022-08-25 15:33:04+00:00\"\nid: 1234\nincrement_strategy: linear\nmode: load\nname: my first loadero python test\nparticipant_timeout: 600\nscript: print(\"hello test script\")\nscript_file_id: 12345\nstart_interval: 1\nupdated: \"2022-08-25 15:33:04+00:00\"\n```\n\n### Running a Test\n\nTo run a test the only required attribute is test ID.\n\nFor the `test` object from previous examples, the `test_id` attribute has been\npopulated by create operation, so it can simply be run by calling the `launch`\nmethod.\n\n```py\nrun = test.launch()\n```\n\nIf a test ID is known it can be run directly.\n\n```py\nrun = Test(test_id=1234).launch()\n```\n\nAll tests in a project can be run with\n\n```py\nfor test in Project().tests()[0]:\n test.launch()\n```\n\n#### Polling\n\nAfter a test has been launched, waiting for the test to finish can be achieved\nwith the `poll` method.\n\n```py\nrun.poll()\n```\n\nBy default, the `poll` method will make an API call to check if the test\nexecution has finished every 15 seconds and will wait up to 12 hours. This\nfunctionality can be customized with the `interval` and `timeout` arguments.\n\n```py\n# will poll every 5 seconds and will wait up to 10 minutes.\nrun.poll(interval=5.0, timeout=10 * 60.0)\n```\n\nIf test execution does not finish within the specified timeout, the `poll`\nmethod will raise an exception.\n\n#### Stopping Test Execution\n\nIf test execution needs to be prematurely stopped, it can be done with the\n`stop` method.\n\n```py\nrun.stop()\n```\n\n**Note** `stop` only sends an API request that starts a Loadero procedure of\nstopping the test. This process is NOT immediate. Even though the `stop` API\nrequest completes relatively quickly, the test can remain running for a while\nlonger.\n\n**Note** If another process is polling the test execution, it will automatically\nstop if the test is stopped.\n\n### Getting Results\n\nAfter the test run finishes execution, the `run` object already contains many\nuseful attributes that may be used in result analysis. The attributes are stored\non the `run.params` field. `run.params` is an `RunParams` object from\n`loadero_python.resources.run` module.\n\n```py\nprint(run.params.success_rate)\n```\n\n#### Participant Results\n\n`run` object describes a result overview of the whole test. To get a more\ndetailed result information about each test participant's results needs to be\nread.\n\n```py\nresults, _, _ = run.Results()\nresult = results[0]\n```\n\nThe ignored return values are pagination and filters. They are not relevant for\nresult retrieval, hence they are omitted. A more detailed explanation of these\nvalues is available in the [Filtering and Pagination](#filtering-and-pagination)\nsection.\n\n`results` is a list of `Result` objects from `loadero_python.resources.result`\nmodule. A single result corresponds to a single participant in the test.\n\n`Result` just like a regular resource object has a `params` field of type\n`ResultParams` that contains its attributes. The result resource has the largest\namount of attributes, so this showcase will cover only common use cases.\n\n##### Log Retrieval\n\n```py\nimport requests\n\nresp = requests.get(result.params.log_paths.selenium)\nif not resp:\n print(\"failed to download selenium log\")\n exit(1)\n\nwith open(f\"selenium_log_of_result_{result.params.result_id}\", \"w\") as f:\n f.write(resp.text)\n```\n\n`result.params.log_paths.selenium` is an URL to an Selenium log. It first needs\nto be downloaded using the HTTP library `requests`. Then it can be written to a\nfile.\n\n##### Extracting Failed Asserts\n\nBefore extracting failed asserts. `AssertStatus` classificator constant\nenumeration needs to be imported.\n\n```py\nfrom loadero_python.resources.classificator import AssertStatus\n```\n\n```py\nfailed_asserts = []\n\nfor result_assert in result.params.asserts:\n if result_assert.status == AssertStatus.AS_FAIL\n failed_asserts.append(result_assert)\n```\n\n##### Checking metrics\n\nLoadero tests collect various different metrics from CPU, RAM, and network usage\nto video and audio quality indicators. Loadero organizes these metrics with\nmetric base paths - a path-like string that uniquely identifies metric data. For\nexample, CPU usage metric data is described by the metric base path\n`machine/cpu/used`.\n\nAfter test execution finishes Loadero processes the collected metric data by\napplying aggregator functions.\n\n- total\n- minimum\n- maximum\n- average\n- standard deviation\n- relative standard deviation\n- 1st percentile\n- 5th percentile\n- 25th percentile\n- 50th percentile\n- 75th percentile\n- 95th percentile\n- 99th percentile\n\nThe result is a single float value identified by a metric path. For example, the\nmaximal CPU usage is described by the metric path - `machine/cpu/used/max`\n\nIn Loadero-Python metric base paths - `MetricBasePath` and metric paths -\n`MetricPath` are constant enumerations of all the available metric and metric\nbase paths. Contained within the `loadero_python.resources.metric_path` module.\n\nTo access a specific metric `MetricBasePath` enumeration needs to be imported.\n\n```py\nfrom loadero_python.resources.metric_path import MetricBasePath\n```\n\nThen a specific metric can be checked like this\n\n```py\nif result.params.metrics is None or result.metrics.machine is None:\n print(\"result has no machine metrics\")\n exit(1)\n\nif MetricBasePath.MACHINE_CPU_AVAILABLE not in result.params.metrics.machine:\n print(\"result has no machine cpu available metric\")\n exit(1)\n\nif (\n result.params.metrics.machine[MetricBasePath.MACHINE_CPU_AVAILABLE].average\n < 10.0\n):\n print(\"test is well configured. efficient usage of CPU resources\")\n```\n\nThe `not None` checks are required because some or all metrics for a result can\nbe missing. For example, non-WebRTC tests will not have any `webrtc` metrics.\n\n### Filtering and Pagination\n\nRead-all operations have the option to limit the number of resources returned,\noffset a limited read-all operation by some amount of resources and filter out\nundesired resources.\n\nThis is done by passing a query params argument when performing a read-all\noperation.\n\n`QueryParams` class is contained in `loadero_python.resources.resource` module\nand can be imported with\n\n```py\nfrom loadero_python.resources.resource import QueryParams\n```\n\n#### Filtering\n\nFilters are resource-specific and are defined in each resource module. For\nexample, test resource filters are defined in the `TestFilterKey` constant\nenumeration in the `loadero_python.resources.test` module.\n\n`TestFilterKey` can be imported with\n\n```py\nfrom loadero_python.resources.test import TestFilterKey\n```\n\nNow test read all operations can be filtered like this\n\n```py\ntests, _, filters = Project().tests(\n query_params=QueryParams().filter(\n TestFilterKey.PARTICIPANT_TIMEOUT_TO, 10 * 60 # ten minutes\n )\n)\n```\n\nThe ignored value is pagination. It can be ignored because pagination will\ncontain useful page information when limit and offset query_params have been\napplied.\n\nThis will return tests whose participant timeout attribute is smaller than ten\nminutes.\n\n`filters` is a python dictionary with the applied filters.\n\n#### Pagination\n\nWhen performing a read-all operation that will return many resources it is good\npractice to limit the number of resources returned and perform multiple smaller\nreads. This can be achieved by limiting and offsetting the number of resources\nreturned.\n\n```py\ntests, pagination, _ = Project().tests(\n query_params=QueryParams().limit(20).offset(10)\n)\n```\n\nThis time the ignored value is filters. It can be ignored because no filters\nwere applied.\n\nLet's assume that the project has 28 tests numbered from 1 to 28, then this\nread-all operation would return tests with numbers from 11 to 28. This happens\nbecause the returned resources were offset by 10 and the next resource after the\n10th is the 11th. Only 18 resources were returned because the remaining\nresources after offset were smaller than the defined limit - 20.\n\n`pagination` is an instance `PaginationParams` class from\n`loadero_python.resources.resource` module. It contains information about the\napplied limit and offset, plus additional information describing how many\nresources remain to be read.\n\n## Structure\n\nThe Loadero-Python library structure is similar to the Loadero API structure.\nThe main structural components are:\n\n- API client\n- resources and operations with them\n\n### API client\n\nContained within `loadero_python.api_client` module is the `APIClient` singleton\nobject. It needs to be initialized once with the project ID and access token.\nAll requests to Loadero API are done with the `APIClient` object. It adds the\nrequired headers to make valid API requests. Additionally, the `APIClient` rate\nlimits all requests to be compliant with Loadero API's access limits. Rate\nlimiting can be opted out on initialization.\n\n```py\nAPIClient(\n project_id=1234,\n access_token=\"your_access_token\",\n rate_limit=False,\n)\n```\n\n### Resources\n\nEach resource has a separate module.\n\n| Resource class | Module |\n| -------------------- | ---------------------------------------------- |\n| `AssertPrecondition` | `loadero_python.resources.assert_precondition` |\n| `Assert` | `loadero_python.resources.assert_resource` |\n| `File` | `loadero_python.resources.file` |\n| `Group` | `loadero_python.resources.group` |\n| `Participant` | `loadero_python.resources.participant` |\n| `Project` | `loadero_python.resources.project` |\n| `Result` | `loadero_python.resources.result` |\n| `RunParticipant` | `loadero_python.resources.run_participant` |\n| `Run` | `loadero_python.resources.run` |\n| `Test` | `loadero_python.resources.test` |\n\nAll resource classes have a similar structure:\n\n- Resource classes have an attribute `params` that is used to store the data of\n a single instance of the resource. Read more about resource params\n [here](#resource-params).\n- most resources implement common CRUD manipulation methods - `create`, `read`,\n `update`, `delete`, `duplicate`. Some resources do not have these methods\n because they are impossible or not available via API access. For example,\n `Project` resource has only `read` method because API access prohibits updates\n to this resource.\n- resources that can have child resources have methods for reading all the child\n resources.\n ```py\n # reads all groups in test\n groups, _, _ = Test(test_id=123).groups()\n ```\n\n### Resource Params\n\nEvery resource has a resource params class that stores the data of a single\nresource instance.\n\nResource params class for each resource is available in the resources module.\n\n| Resource params class | Module |\n| -------------------------- | ---------------------------------------------- |\n| `AssertPreconditionParams` | `loadero_python.resources.assert_precondition` |\n| `AssertParams` | `loadero_python.resources.assert_resource` |\n| `FileParams` | `loadero_python.resources.file` |\n| `GroupParams` | `loadero_python.resources.group` |\n| `ParticipantParams` | `loadero_python.resources.participant` |\n| `ProjectParams` | `loadero_python.resources.project` |\n| `ResultParams` | `loadero_python.resources.result` |\n| `RunParticipantParams` | `loadero_python.resources.run_participant` |\n| `RunParams` | `loadero_python.resources.run` |\n| `TestParams` | `loadero_python.resources.test` |\n\nResource params classes provide access to the resource attributes.\n\n```py\n# read a test and print its name\nprint(Test(test_id=123).read().params.name)\n```\n\n### Constants\n\nLoadero-Python has two modules for constants.\n\n- `loadero_python.resources.classificator` for classificator constant\n enumerations.\n- `loadero_python.resources.metric_path` for metric path and metric base path\n constant enumerations.\n\n### Resource API\n\nEvery resource has its own API class also available in the resources module.\n\n| Resource class | Module |\n| ----------------------- | ---------------------------------------------- |\n| `AssertPreconditionAPI` | `loadero_python.resources.assert_precondition` |\n| `AssertAPI` | `loadero_python.resources.assert_resource` |\n| `FileAPI` | `loadero_python.resources.file` |\n| `GroupAPI` | `loadero_python.resources.group` |\n| `ParticipantAPI` | `loadero_python.resources.participant` |\n| `ProjectAPI` | `loadero_python.resources.project` |\n| `ResultAPI` | `loadero_python.resources.result` |\n| `RunParticipantAPI` | `loadero_python.resources.run_participant` |\n| `RunAPI` | `loadero_python.resources.run` |\n| `TestAPI` | `loadero_python.resources.test` |\n\nResource API class implements all the available API operations of that resource.\nResource API classes are internally used by all resources, but the class on its\nown is not very useful.\n\n## Contributing\n\nFound a bug? - Feel free to open an\n[issue](https://github.com/loadero/loadero-python/issues/new/choose).\n\nWould like to request a feature? - Open an issue describing the request and the\nreason for it or contact Loadero [support](mailto:support@loadero.com).\n\nWant to contribute? - Open\n[issues](https://github.com/loadero/loadero-python/issues) is a good place where\nto find stuff that needs to be worked on.\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Python client for Loadero API",
"version": "1.2.2",
"project_urls": {
"Documentation": "https://loadero.github.io/loadero-python/",
"Homepage": "https://github.com/loadero/loadero-python",
"Loadero": "https://loadero.com/",
"Source": "https://github.com/loadero/loadero-python",
"Tracker": "https://github.com/loadero/loadero-python/issues"
},
"split_keywords": [
"loadero"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "12ed9d64f3018350a58b81ed7daa7c791f2e6fd8afc0a33ab1340002f2e0eb2d",
"md5": "43a5aec0663315b1a251004c699a2153",
"sha256": "d40a28f15da4d868676c6069f9eb19dd9480200970b800693981216bffbf1dfe"
},
"downloads": -1,
"filename": "loadero_python-1.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "43a5aec0663315b1a251004c699a2153",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7, <4",
"size": 70327,
"upload_time": "2024-02-01T09:44:48",
"upload_time_iso_8601": "2024-02-01T09:44:48.948305Z",
"url": "https://files.pythonhosted.org/packages/12/ed/9d64f3018350a58b81ed7daa7c791f2e6fd8afc0a33ab1340002f2e0eb2d/loadero_python-1.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "51c47e880b48aef39bc3bef49f99cf20272ff8fe93c62201e51bc16cd7e7842a",
"md5": "6680581b19d4199a647e8db91972930f",
"sha256": "d3edf28350691de7cd5ee80b06abfb811c034fd9bc6f3a53c9c95ff8bcb8ca1a"
},
"downloads": -1,
"filename": "loadero_python-1.2.2.tar.gz",
"has_sig": false,
"md5_digest": "6680581b19d4199a647e8db91972930f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7, <4",
"size": 66297,
"upload_time": "2024-02-01T09:44:50",
"upload_time_iso_8601": "2024-02-01T09:44:50.745370Z",
"url": "https://files.pythonhosted.org/packages/51/c4/7e880b48aef39bc3bef49f99cf20272ff8fe93c62201e51bc16cd7e7842a/loadero_python-1.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-01 09:44:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "loadero",
"github_project": "loadero-python",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "loadero-python"
}
JSON
