# dials-py
The Python api client interface to DIALS service.
## Installation
To install dials-py, simply
```bash
$ pip install cmsdials
```
It is also possible to specify the following extras, to enable optional features:
```
pandas, tqdm
```
## Usage
Before interfacing with any route you need to generate valid credentials, it is possible to authenticate trough the `device authorization flow` or using the `client secret key` of any application registered in DIALS. Note that, the device flow is an interactively authentication procedure that is possible to distinguish users in DIALS backend and the client secret flow is not interactive and is not possible to distinguish users so it should only be used for automation scripts.
### Generating credentials with client secret
```python
from cmsdials.auth.secret_key import Credentials
creds = Credentials(token=".....")
```
### Generating credentials with device
#### Loading from AuthClient
```python
from cmsdials.auth.client import AuthClient
from cmsdials.auth.bearer import Credentials
auth = AuthClient()
token = auth.device_auth_flow()
creds = Credentials.from_authclient_token(token)
```
#### Loading from cached credentials file
Credentials are always cached once you authenticate at least one time, calling this method without having a cached credential file will automatically trigger the AuthClient device flow.
```python
from cmsdials.auth.bearer import Credentials
creds = Credentials.from_creds_file()
```
### Basic Example
```python
from cmsdials.auth.bearer import Credentials
from cmsdials import Dials
from cmsdials.filters import LumisectionHistogram1DFilters
creds = Credentials.from_creds_file()
dials = Dials(creds)
# Getting h1d data
data = dials.h1d.list_all(LumisectionHistogram1DFilters(me="PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2"), max_pages=5)
```
### Workspace
Users are automatically routed to a workspace based on e-groups, but it is possible to overwrite this configuration and inspect data from others workspaces:
```python
dials = Dials(creds, workspace="jetmet")
```
## Available endpoints
This package interacts with DIALS api endpoints using underlying classes in `Dials` object.
### Retrieving a specific object using `get`
```python
dials.file_index.get(dataset_id=14677060, file_id=3393809397)
dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=1)
dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=96)
dials.lumi.get(dataset_id=14677060, run_number=367112, ls_number=10)
dials.run.get(dataset_id=14677060, run_number=367112)
```
### Retrieving a list of objects per page using `list`
It is possible to get a list of entries from those endpoint using the `list` and `list_all` methods, the `list` method will fetch only one page and the `list_all` will fetch all available pages:
```python
dials.file_index.list()
dials.h1d.list()
dials.h2d.list()
dials.lumi.list()
dials.run.list()
```
### Retrieving all available pages of a list of objects using `list_all`
Note: Keep in mind that running `list_all` without any filter can take too much time, since you will be retrieving all rows in the database.
```python
dials.file_index.list_all()
dials.h1d.list_all()
dials.h2d.list_all()
dials.lumi.list_all()
dials.run.list_all()
```
If you don't need all available pages but just a subset of then, it is possible to specify a `max_pages` integer parameter:
```python
dials.run.list_all(..., max_pages=5)
```
### Using filters
Keep in mind that calling those methods without any filter can take a lot of time, because the underlying query will try to load the entire database table through multiple requests, then it is recommended to apply filters according to DIALS [live documentation](https://cmsdials-api.web.cern.ch/api/v1/swagger#/) using filter classes for each table:
```python
from cmsdials.filters import (
FileIndexFilters,
LumisectionHistogram1DFilters,
LumisectionHistogram2DFilters,
LumisectionFilters,
RunFilters
)
dials.file_index.list(FileIndexFilters(dataset__regex="2024B"))
dials.h1d.list(LumisectionHistogram1DFilters(me="PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2"))
dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="PXBarrel", ls_number=78, entries__gte=100), max_pages=5)
dials.lumi.list_all(LumisectionFilters(run_number=360392), max_pages=5)
dials.run.list_all(RunFilters(run_number__gte=360392, run_number__lte=365000), max_pages=5)
```
### Dials MEs
It is possible to inspect the list of ingested MEs in DIALS listing the endpoint `mes` trough the method:
```python
dials.mes.list()
```
### Automatically convert paginated results to pandas DataFrame
You can enable this optional feature by installing the package with the `pandas` extra.
All `Paginated` metaclasses contain the method `to_pandas` that will automatically transform the `results` attribute of the metaclass into a pandas dataframe, for example:
```python
data = dials.h1d.list_all(LumisectionHistogram1DFilters(me="PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2"), max_pages=5)
data.to_pandas()
```
### Indefinite progress bar when fetch multi-page result
You can enable this optional feature by installing the package with the `tqdm` extra.
Whenever you call a `list_all` method that fetches multiple pages a dynamic progress will be rendered to indicate duration and number of pages, for example:
```python
>>> dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="PXBarrel", ls_number=78, entries__gte=100), max_pages=5)
Progress: 100%|█████████████████████████████████████| 5/5 [00:02<00:00, 1.70it/s]
```
The total attribute of the bar is dynamically updated while fetching the pages.
## Usage with local DIALS
All classes that interface the DIALS service inherits the class `BaseAPIClient` which propagate the `base_url`, `route` and `version` attributes with production values. In order to use dials-py with a local version of DIALS it is possible to overwrite those attributes when instantiating the `AuthClient` and the `Dials` client, for example:
```python
from cmsdials.auth.client import AuthClient
from cmsdials.auth.bearer import Credentials
from cmsdials import Dials
from cmsdials.filters import LumisectionHistogram2DFilters
DEV_URL = "http://localhost:8000/"
DEV_CACHE_DIR = ".cache-dev"
auth = AuthClient(base_url=DEV_URL)
creds = Credentials.from_creds_file(cache_dir=DEV_CACHE_DIR, client=auth) # Make sure to specify the auth client with overwritten values, using another cache_dir is recommended
dials = Dials(creds, base_url=DEV_URL)
dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="EEOT digi occupancy EE +", entries__gte=100, run_number__gte=360392, run_number__lte=365000), max_pages=5)
```
## Running tests
The repository has some tests written to make sure DIALS responses are compatible with pydantic metaclasses, you can use pytest to run all tests but you need to specify a secret key to authenticate non-interactively against DIALS api:
```bash
SECRET_KEY=... pytest tests
```
The secret key is an api client enabled secret key and can be obtained from the applications portal, any api client secret key whitelisted in DIALS can be used. The interactive authentication flow should be tested manually, an example for this can be found in [this line](/tests/integration/test_auth_client.py#L33).
If testing against a local version of DIALS you need to specify the BASE_URL:
```bash
SECRET_KEY=... BASE_URl=http://localhost:8000 pytest tests
```
Raw data
{
"_id": null,
"home_page": "https://github.com/cms-DQM/dials-py",
"name": "cmsdials",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Gabriel Moreira",
"author_email": "gabrielmscampos@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/ce/10/1208f2a60a99c805a8d3f7c3c6d2175e48ba5056996800fbcab2f1e3ed5f/cmsdials-1.3.0.tar.gz",
"platform": null,
"description": "# dials-py\n\nThe Python api client interface to DIALS service.\n\n## Installation\n\nTo install dials-py, simply\n\n```bash\n$ pip install cmsdials\n```\n\nIt is also possible to specify the following extras, to enable optional features:\n\n```\npandas, tqdm\n```\n\n## Usage\n\nBefore interfacing with any route you need to generate valid credentials, it is possible to authenticate trough the `device authorization flow` or using the `client secret key` of any application registered in DIALS. Note that, the device flow is an interactively authentication procedure that is possible to distinguish users in DIALS backend and the client secret flow is not interactive and is not possible to distinguish users so it should only be used for automation scripts.\n\n### Generating credentials with client secret\n\n```python\nfrom cmsdials.auth.secret_key import Credentials\n\ncreds = Credentials(token=\".....\")\n```\n\n### Generating credentials with device\n\n#### Loading from AuthClient\n\n```python\nfrom cmsdials.auth.client import AuthClient\nfrom cmsdials.auth.bearer import Credentials\n\nauth = AuthClient()\ntoken = auth.device_auth_flow()\ncreds = Credentials.from_authclient_token(token)\n```\n\n#### Loading from cached credentials file\n\nCredentials are always cached once you authenticate at least one time, calling this method without having a cached credential file will automatically trigger the AuthClient device flow.\n\n```python\nfrom cmsdials.auth.bearer import Credentials\n\ncreds = Credentials.from_creds_file()\n```\n\n### Basic Example\n\n```python\nfrom cmsdials.auth.bearer import Credentials\nfrom cmsdials import Dials\nfrom cmsdials.filters import LumisectionHistogram1DFilters\n\ncreds = Credentials.from_creds_file()\ndials = Dials(creds)\n\n# Getting h1d data\ndata = dials.h1d.list_all(LumisectionHistogram1DFilters(me=\"PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2\"), max_pages=5)\n```\n\n### Workspace\n\nUsers are automatically routed to a workspace based on e-groups, but it is possible to overwrite this configuration and inspect data from others workspaces:\n\n```python\ndials = Dials(creds, workspace=\"jetmet\")\n```\n\n## Available endpoints\n\nThis package interacts with DIALS api endpoints using underlying classes in `Dials` object.\n\n### Retrieving a specific object using `get`\n\n```python\ndials.file_index.get(dataset_id=14677060, file_id=3393809397)\ndials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=1)\ndials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=96)\ndials.lumi.get(dataset_id=14677060, run_number=367112, ls_number=10)\ndials.run.get(dataset_id=14677060, run_number=367112)\n```\n\n### Retrieving a list of objects per page using `list`\n\nIt is possible to get a list of entries from those endpoint using the `list` and `list_all` methods, the `list` method will fetch only one page and the `list_all` will fetch all available pages:\n\n```python\ndials.file_index.list()\ndials.h1d.list()\ndials.h2d.list()\ndials.lumi.list()\ndials.run.list()\n```\n\n### Retrieving all available pages of a list of objects using `list_all`\n\nNote: Keep in mind that running `list_all` without any filter can take too much time, since you will be retrieving all rows in the database.\n\n```python\ndials.file_index.list_all()\ndials.h1d.list_all()\ndials.h2d.list_all()\ndials.lumi.list_all()\ndials.run.list_all()\n```\n\nIf you don't need all available pages but just a subset of then, it is possible to specify a `max_pages` integer parameter:\n\n```python\ndials.run.list_all(..., max_pages=5)\n```\n\n### Using filters\n\nKeep in mind that calling those methods without any filter can take a lot of time, because the underlying query will try to load the entire database table through multiple requests, then it is recommended to apply filters according to DIALS [live documentation](https://cmsdials-api.web.cern.ch/api/v1/swagger#/) using filter classes for each table:\n\n```python\nfrom cmsdials.filters import (\n FileIndexFilters,\n LumisectionHistogram1DFilters,\n LumisectionHistogram2DFilters,\n LumisectionFilters,\n RunFilters\n)\n\ndials.file_index.list(FileIndexFilters(dataset__regex=\"2024B\"))\n\ndials.h1d.list(LumisectionHistogram1DFilters(me=\"PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2\"))\n\ndials.h2d.list_all(LumisectionHistogram2DFilters(me__regex=\"PXBarrel\", ls_number=78, entries__gte=100), max_pages=5)\n\ndials.lumi.list_all(LumisectionFilters(run_number=360392), max_pages=5)\n\ndials.run.list_all(RunFilters(run_number__gte=360392, run_number__lte=365000), max_pages=5)\n```\n\n### Dials MEs\n\nIt is possible to inspect the list of ingested MEs in DIALS listing the endpoint `mes` trough the method:\n\n```python\ndials.mes.list()\n```\n\n### Automatically convert paginated results to pandas DataFrame\n\nYou can enable this optional feature by installing the package with the `pandas` extra.\n\nAll `Paginated` metaclasses contain the method `to_pandas` that will automatically transform the `results` attribute of the metaclass into a pandas dataframe, for example:\n\n```python\ndata = dials.h1d.list_all(LumisectionHistogram1DFilters(me=\"PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2\"), max_pages=5)\ndata.to_pandas()\n```\n\n### Indefinite progress bar when fetch multi-page result\n\nYou can enable this optional feature by installing the package with the `tqdm` extra.\n\nWhenever you call a `list_all` method that fetches multiple pages a dynamic progress will be rendered to indicate duration and number of pages, for example:\n\n```python\n>>> dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex=\"PXBarrel\", ls_number=78, entries__gte=100), max_pages=5)\nProgress: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [00:02<00:00, 1.70it/s]\n```\n\nThe total attribute of the bar is dynamically updated while fetching the pages.\n\n## Usage with local DIALS\n\nAll classes that interface the DIALS service inherits the class `BaseAPIClient` which propagate the `base_url`, `route` and `version` attributes with production values. In order to use dials-py with a local version of DIALS it is possible to overwrite those attributes when instantiating the `AuthClient` and the `Dials` client, for example:\n\n```python\nfrom cmsdials.auth.client import AuthClient\nfrom cmsdials.auth.bearer import Credentials\nfrom cmsdials import Dials\nfrom cmsdials.filters import LumisectionHistogram2DFilters\n\nDEV_URL = \"http://localhost:8000/\"\nDEV_CACHE_DIR = \".cache-dev\"\n\nauth = AuthClient(base_url=DEV_URL)\ncreds = Credentials.from_creds_file(cache_dir=DEV_CACHE_DIR, client=auth) # Make sure to specify the auth client with overwritten values, using another cache_dir is recommended\ndials = Dials(creds, base_url=DEV_URL)\n\ndials.h2d.list_all(LumisectionHistogram2DFilters(me__regex=\"EEOT digi occupancy EE +\", entries__gte=100, run_number__gte=360392, run_number__lte=365000), max_pages=5)\n```\n\n## Running tests\n\nThe repository has some tests written to make sure DIALS responses are compatible with pydantic metaclasses, you can use pytest to run all tests but you need to specify a secret key to authenticate non-interactively against DIALS api:\n\n```bash\nSECRET_KEY=... pytest tests\n```\n\nThe secret key is an api client enabled secret key and can be obtained from the applications portal, any api client secret key whitelisted in DIALS can be used. The interactive authentication flow should be tested manually, an example for this can be found in [this line](/tests/integration/test_auth_client.py#L33).\n\nIf testing against a local version of DIALS you need to specify the BASE_URL:\n\n```bash\nSECRET_KEY=... BASE_URl=http://localhost:8000 pytest tests\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "The Python api client interface to DIALS service",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/cms-DQM/dials-py",
"Repository": "https://github.com/cms-DQM/dials-py"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4ad0f3b013bacd6d29621f60d17950b02bf473865229d2f42f8fed3025b9e1e1",
"md5": "406bf2bb79557db5294f29b9f2d6e750",
"sha256": "52459b85b328e1570b162630346db2ed392f43a51529168b7ba5b1fde70f98e4"
},
"downloads": -1,
"filename": "cmsdials-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "406bf2bb79557db5294f29b9f2d6e750",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 46112,
"upload_time": "2024-07-31T13:36:45",
"upload_time_iso_8601": "2024-07-31T13:36:45.084749Z",
"url": "https://files.pythonhosted.org/packages/4a/d0/f3b013bacd6d29621f60d17950b02bf473865229d2f42f8fed3025b9e1e1/cmsdials-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ce101208f2a60a99c805a8d3f7c3c6d2175e48ba5056996800fbcab2f1e3ed5f",
"md5": "0b742d65fdcac7a63cea8d8cd3f36ea2",
"sha256": "1291d01d64929a63063e164336da9d1219820aeb1226ded0e1f5538edcffa8f7"
},
"downloads": -1,
"filename": "cmsdials-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "0b742d65fdcac7a63cea8d8cd3f36ea2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 26626,
"upload_time": "2024-07-31T13:36:46",
"upload_time_iso_8601": "2024-07-31T13:36:46.800370Z",
"url": "https://files.pythonhosted.org/packages/ce/10/1208f2a60a99c805a8d3f7c3c6d2175e48ba5056996800fbcab2f1e3ed5f/cmsdials-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-31 13:36:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cms-DQM",
"github_project": "dials-py",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "cmsdials"
}
JSON
