| Name | pyftdc JSON |
| Version |
0.3.11
JSON |
| download |
| home_page | |
| Summary | A parser for MongoDB FTDC files, with Python bindings. |
| upload_time | 2023-12-27 21:35:41 |
| maintainer | |
| docs_url | None |
| author | Jorge Imperial |
| requires_python | >=3.8 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
pyftdc
==============
| | Link | status |
|--------------|------|------|
| Conda builds | https://anaconda.org/jimper/pyftdc | |
| Pypi builds | https://pypi.org/project/pyftdc/ | |
A MongoDB FTDC files parser written in C++ that provides Python bindings using [pybind11](https://github.com/pybind/pybind11) and scikit-build.
[gitter-badge]: https://badges.gitter.im/pybind/Lobby.svg
[gitter-link]: https://gitter.im/pybind/Lobby
[actions-badge]: https://github.com/pybind/pyftdc/workflows/Tests/badge.svg
[actions-conda-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3AConda
[actions-conda-badge]: https://github.com/pybind/pyftdc/workflows/Conda/badge.svg
[actions-pip-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3APip
[actions-pip-badge]: https://github.com/pybind/pyftdc/workflows/Pip/badge.svg
[actions-wheels-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3AWheels
[actions-wheels-badge]: https://github.com/pybind/pyftdc/workflows/Wheels/badge.svg
Requisites
------------
To build the source distribution, you will need Python 3.8 or newer, git, python3-dev/python3-devel installed.
Please read the [build document](docs/build.md) for more details.
Installation
------------
You can install the source distributions via pip or conda:
- pip3 install pyftdc
- conda install -c jimper pyftdc
Building
------------
**Building on Unix (Ubuntu, Centos, macOS)**
1. clone this repository and change to the top level directory.
```
git clone git@gitlab.com:jimper/mongo_ftdc.git
cd mongo_ftdc
```
2. Install Python libraries to build binaries. Create a virtual environment to make your life easier.
```
python3 -m venv venv
source venv/bin/activate
pip3 install --user-pep517 .
```
You will now have built and installed in your virtual environment.
Alternatively, you can use setup.py directly, but for that you will need to manually install the required libraries into your virtual environment by running
cd mongo_ftdc
pip install -r requirements.txt
After which you can create a source distribution or a binary wheel:
python3 setup.py sdist
python3 setup.py bdist_wheel
These will reside in the _dist_ directory.
**Building on Windows**
Not tested yet, but 'It should work'(TM)
License
-------
Apache V2
Usage
---------
The module provides two classes, FTDCParser and Dataset.
The proper usage is to obtain a Dataset object from a file by calling FTDCParse.parse_file(), or a list of Dataset objects from the FTDCParse.parse_dir(), and then call the methods on these objects.
FTDCParser
- set_verbose(bool).
Set verbosity flag.
- parse_dir(dir, only_metadata, only_metrics_names, lazy)
Returns a list of Dataset objects.
- parse_file(file_path, only_metadata, only_metrics_names, lazy)
Returns a single Dataset object.
- get_parsed_file_info():
Returns information on the last parsed file (absolute path, samples, start timestamp, end timestamp).
- dump_file_as_json(input_file, output_file)
Dumps the contents of an FTDC file to a JSON file. (WIP)
- dump_file_as_csv(input_file, output_file)
Dumps file contents to an FTDC file as CSV file. (WIP)
- get_metric(metric_name, start, end, rated_metric).
Returns a list of the values of a metric for the last file parsed. If rated_metric is true, will convert values to metrics per second. start and end are ignored.
- get_metrics_list(metric_name_list)
Returns a list of lists of values for the metrics specified in the list metric_name_list for the last file parsed. If rated_metric is true, will convert values to metrics per second. start and end are ignored.
- get_timestamps(start, end)
Returns the timestamps of the last file parsed.
- metadata.
Contains a string with the metadata of the last file parsed.
- get_metric_numpy(metric_name, start, end, rated_metric)
Returns a list of the values of a metric for the last file parsed as a numpy array. If rated_metric is true, will convert values to metrics per second. start and end are ignored.
- get_metrics_list_numpy(metric_names_list, start, end, rated_metric)
Returns a list of numpy arrays of the values of a metric for the last file parsed . If rated_metric is true, will convert values to metrics per second. start and end are ignored.
Dataset
- interinm
Contains a boolean that tells if this is an interim FTDC file.
- metadata
Contains the metadata associated with this datased.
- metrics_names
Contains a list of the metrics names contained in the dataset.
- file
Contains the file name from which this dataset was parsed.
- path
Contains the full path to the file from which this dataset was parsed.
- get_metric(metric_name, start, end, rated_metric)
Returns a list of the values of a metric for this dataset. If rated_metric is true, will convert values to metrics per second. start and end are ignored.
- get_metric_numpy(metric_name, start, end, rated_metric)
Returns a list of the values of a metric for the last file parsed as a numpy array. If rated_metric is true, will convert values to metrics per second. start and end are ignored.
- get_metrics_list_numpy(metric_names_list, start, end, rated_metric)
Returns a list of numpy arrays of the values of a metric for the last file parsed . If rated_metric is true, will convert values to metrics per second. start and end are ignored.
Example
---------
```python
import pyftdc
import numpy
def get_prefixed_metrics_names(param, ds):
ops_counters = []
for name in ds.metrics_names:
if name.startswith(param):
ops = ds.get_metric(name)
ops_counters.append((name, ops))
return ops_counters
def open_files_in_dir(dir_path, prefix):
from os import listdir
files_read = []
try:
dir_list = listdir(dir_path)
for file_name in dir_list:
if file_name.startswith(prefix):
parser = pyftdc.FTDCParser()
ds = parser.parse_file(dir_path + '/' + file_name)
if ds:
print(f'File: {ds.file}')
print(f'{ds.metadata}')
ts = ds.get_metric("start")
if ts:
ts_size = len(ts)
print(f'Timestamp count {ts_size}. Start:{ts[0]} Last: {ts[-1]}')
op_counter_names = get_prefixed_metrics_names('serverStatus.opcounters', ds)
cpu = get_prefixed_metrics_names('systemMetrics.cpu', ds)
disk = get_prefixed_metrics_names('systemMetrics.disks.nvme1n1', ds)
xxx = ds.get_metric_list_numpy(['systemMetrics.cpu.iowait_ms', 'xxx', 'systemMetrics.cpu.num_cpus'])
disk_n = ds.get_metric_numpy('systemMetrics.disks.nvme1n1.writes')
files_read.append(file_name)
else:
print(f'No timestamps on this dataset.')
except FileNotFoundError as not_found:
print('Path not found.')
return files_read
if __name__ == "__main__":
files = open_files_in_dir('/somepath/diagnostic.data/',
'metrics.2022-11-13T21')
print(files)
```
[`cibuildwheel`]: https://cibuildwheel.readthedocs.io
Raw data
{
"_id": null,
"home_page": "",
"name": "pyftdc",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Jorge Imperial",
"author_email": "Jorge Imperial <jlimperial@protonmail.com>",
"download_url": "https://files.pythonhosted.org/packages/80/e0/61227a3bdfc771ff69cc472d7169c26cbbc9e0edaef1d221f978fa3db792/pyftdc-0.3.11.tar.gz",
"platform": null,
"description": "pyftdc\n==============\n\n\n| | Link | status |\n|--------------|------|------|\n| Conda builds | https://anaconda.org/jimper/pyftdc | |\n| Pypi builds | https://pypi.org/project/pyftdc/ | |\n\n\n\nA MongoDB FTDC files parser written in C++ that provides Python bindings using [pybind11](https://github.com/pybind/pybind11) and scikit-build.\n\n\n\n[gitter-badge]: https://badges.gitter.im/pybind/Lobby.svg\n[gitter-link]: https://gitter.im/pybind/Lobby\n[actions-badge]: https://github.com/pybind/pyftdc/workflows/Tests/badge.svg\n[actions-conda-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3AConda\n[actions-conda-badge]: https://github.com/pybind/pyftdc/workflows/Conda/badge.svg\n[actions-pip-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3APip\n[actions-pip-badge]: https://github.com/pybind/pyftdc/workflows/Pip/badge.svg\n[actions-wheels-link]: https://github.com/pybind/pyftdc/actions?query=workflow%3AWheels\n[actions-wheels-badge]: https://github.com/pybind/pyftdc/workflows/Wheels/badge.svg\n\n\n\nRequisites\n------------\n\nTo build the source distribution, you will need Python 3.8 or newer, git, python3-dev/python3-devel installed.\n\nPlease read the [build document](docs/build.md) for more details.\n\n\nInstallation\n------------\nYou can install the source distributions via pip or conda:\n\n- pip3 install pyftdc\n\n- conda install -c jimper pyftdc\n\nBuilding\n------------\n\n**Building on Unix (Ubuntu, Centos, macOS)**\n\n \n 1. clone this repository and change to the top level directory.\n ```\n git clone git@gitlab.com:jimper/mongo_ftdc.git \n cd mongo_ftdc\n ```\n \n 2. Install Python libraries to build binaries. Create a virtual environment to make your life easier.\n \n ```\n python3 -m venv venv\n source venv/bin/activate\n pip3 install --user-pep517 .\n ```\n \n You will now have built and installed in your virtual environment.\n \n\nAlternatively, you can use setup.py directly, but for that you will need to manually install the required libraries into your virtual environment by running\n\n \n cd mongo_ftdc\n pip install -r requirements.txt\n \n \nAfter which you can create a source distribution or a binary wheel:\n\n python3 setup.py sdist\n python3 setup.py bdist_wheel\n \nThese will reside in the _dist_ directory.\n\n\n**Building on Windows**\n \n Not tested yet, but 'It should work'(TM)\n \n\n\nLicense\n-------\n\nApache V2\n\nUsage\n---------\n\nThe module provides two classes, FTDCParser and Dataset. \nThe proper usage is to obtain a Dataset object from a file by calling FTDCParse.parse_file(), or a list of Dataset objects from the FTDCParse.parse_dir(), and then call the methods on these objects.\n\n\nFTDCParser \n - set_verbose(bool). \n \n Set verbosity flag.\n\n\n - parse_dir(dir, only_metadata, only_metrics_names, lazy)\n \n Returns a list of Dataset objects.\n\n\n - parse_file(file_path, only_metadata, only_metrics_names, lazy)\n \n Returns a single Dataset object.\n\n\n - get_parsed_file_info():\n \n Returns information on the last parsed file (absolute path, samples, start timestamp, end timestamp).\n\n\n - dump_file_as_json(input_file, output_file)\n \n Dumps the contents of an FTDC file to a JSON file. (WIP)\n\n\n - dump_file_as_csv(input_file, output_file)\n Dumps file contents to an FTDC file as CSV file. (WIP)\n\n \n - get_metric(metric_name, start, end, rated_metric).\n\n Returns a list of the values of a metric for the last file parsed. If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n \n - get_metrics_list(metric_name_list)\n \n Returns a list of lists of values for the metrics specified in the list metric_name_list for the last file parsed. If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\n - get_timestamps(start, end)\n\n Returns the timestamps of the last file parsed.\n\n \n - metadata.\n \n Contains a string with the metadata of the last file parsed.\n\n\n - get_metric_numpy(metric_name, start, end, rated_metric)\n\n Returns a list of the values of a metric for the last file parsed as a numpy array. If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\n - get_metrics_list_numpy(metric_names_list, start, end, rated_metric)\n\n Returns a list of numpy arrays of the values of a metric for the last file parsed . If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\nDataset\n - interinm\n \n Contains a boolean that tells if this is an interim FTDC file.\n\n\n - metadata\n\n Contains the metadata associated with this datased.\n\n\n - metrics_names\n\n Contains a list of the metrics names contained in the dataset.\n\n\n - file\n\n Contains the file name from which this dataset was parsed.\n\n\n - path\n\n Contains the full path to the file from which this dataset was parsed.\n\n\n- get_metric(metric_name, start, end, rated_metric)\n\n Returns a list of the values of a metric for this dataset. If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\n- get_metric_numpy(metric_name, start, end, rated_metric)\n\n Returns a list of the values of a metric for the last file parsed as a numpy array. If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\n- get_metrics_list_numpy(metric_names_list, start, end, rated_metric)\n\n Returns a list of numpy arrays of the values of a metric for the last file parsed . If rated_metric is true, will convert values to metrics per second. start and end are ignored.\n\n\n \n\nExample\n---------\n\n```python\nimport pyftdc\nimport numpy\n\ndef get_prefixed_metrics_names(param, ds):\n ops_counters = []\n for name in ds.metrics_names:\n if name.startswith(param):\n ops = ds.get_metric(name)\n ops_counters.append((name, ops))\n\n return ops_counters\n\n\ndef open_files_in_dir(dir_path, prefix):\n from os import listdir\n\n files_read = []\n try:\n dir_list = listdir(dir_path)\n for file_name in dir_list:\n if file_name.startswith(prefix):\n parser = pyftdc.FTDCParser()\n ds = parser.parse_file(dir_path + '/' + file_name)\n if ds:\n print(f'File: {ds.file}')\n print(f'{ds.metadata}')\n ts = ds.get_metric(\"start\")\n if ts:\n ts_size = len(ts)\n\n print(f'Timestamp count {ts_size}. Start:{ts[0]} Last: {ts[-1]}')\n\n op_counter_names = get_prefixed_metrics_names('serverStatus.opcounters', ds)\n cpu = get_prefixed_metrics_names('systemMetrics.cpu', ds)\n disk = get_prefixed_metrics_names('systemMetrics.disks.nvme1n1', ds)\n\n xxx = ds.get_metric_list_numpy(['systemMetrics.cpu.iowait_ms', 'xxx', 'systemMetrics.cpu.num_cpus'])\n disk_n = ds.get_metric_numpy('systemMetrics.disks.nvme1n1.writes')\n\n files_read.append(file_name)\n else:\n print(f'No timestamps on this dataset.')\n except FileNotFoundError as not_found:\n print('Path not found.')\n\n return files_read\n\n \n\nif __name__ == \"__main__\":\n files = open_files_in_dir('/somepath/diagnostic.data/',\n 'metrics.2022-11-13T21')\n print(files)\n\n```\n\n[`cibuildwheel`]: https://cibuildwheel.readthedocs.io\n",
"bugtrack_url": null,
"license": "",
"summary": "A parser for MongoDB FTDC files, with Python bindings.",
"version": "0.3.11",
"project_urls": {
"Bug Tracker": "https://gitlab.com/jimper/issues",
"Homepage": "https://gitlab.com/jimper/mongo_ftdc"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "80e061227a3bdfc771ff69cc472d7169c26cbbc9e0edaef1d221f978fa3db792",
"md5": "1e4b4613553e177bf04bf12be9a4f4a6",
"sha256": "801447320db3ef2cf32a9687448797a28689841e5b7fc8d6e3c59179dee25d50"
},
"downloads": -1,
"filename": "pyftdc-0.3.11.tar.gz",
"has_sig": false,
"md5_digest": "1e4b4613553e177bf04bf12be9a4f4a6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1337517,
"upload_time": "2023-12-27T21:35:41",
"upload_time_iso_8601": "2023-12-27T21:35:41.629808Z",
"url": "https://files.pythonhosted.org/packages/80/e0/61227a3bdfc771ff69cc472d7169c26cbbc9e0edaef1d221f978fa3db792/pyftdc-0.3.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-27 21:35:41",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "jimper",
"gitlab_project": "issues",
"lcname": "pyftdc"
}
