### *NTV-NumPy : A multidimensional semantic, compact and reversible format for interoperability*
For more information, see the [user guide](https://loco-philippe.github.io/ntv-numpy/docs/user_guide.html) or the [github repository](https://github.com/loco-philippe/ntv-numpy).
# Why a new format for multidimensional data ?
Each tool has a specific structure for processing multidimensional data with the following consequences:
- interfaces dedicated to each tool,
- partially processed data,
- no unified representation of data structures
The proposed format is based on the following principles:
- neutral format available for tabular or multidimensional tools (e.g. Numpy, pandas, xarray, scipp, astropy),
- taking into account a wide variety of data types as defined in [NTV](https://www.ietf.org/archive/id/draft-thomy-json-ntv-02.html) format,
- high interoperability: reversible (lossless round-trip) interface with tabular or multidimensional tools,
- reversible and compact JSON format (including categorical and sparse format),
- Ease of sharing and exchanging multidimensional and tabular data,
## main features
The NTV-Numpy converter uses this format to:
- provide lossless and reversible interfaces with multidimensional and tabular data processing tools,
- offer data exchange and sharing solutions with neutral or standardized formats (e.g. JSON, Numpy).
NTV-NumPy was developped originally in the [json-NTV project](https://github.com/loco-philippe/NTV)
## example
In the example below, a dataset available in JSON is shared with scipp or Xarray.
```mermaid
---
title: Example of interoperability
---
flowchart LR
A[Xarray] <--lossless--> B[Neutral\nXdataset]
B <--lossless--> C[NDData]
D[Scipp] <--lossless--> B
B <--lossless--> E[JSON]
```
### Data example
```python
In [1]: example = {
'example:xdataset': {
'var1': [['float[kg]', [2, 2], [10.1, 0.4, 3.4, 8.2]], ['x', 'y']],
'var1.variance': [[[2, 2], [0.1, 0.2, 0.3, 0.4]]],
'var1.mask1': [[[True, False]], ['x']],
'var1.mask2': [[[2, 2], [True, False, False, True]]],
'var2': [['var2.ntv'], ['x', 'y']],
'x': [['string', ['23F0AE', '578B98']], {'test': 21}],
'y': [['date', ['2021-01-01', '2022-02-02']]],
'ranking': [['month', [2, 2], [1, 2, 3, 4]], ['var1']],
'z': [['float', [10, 20]], ['x']],
'z.uncertainty': [[[0.1, 0.2]]],
'z_bis': [[['z1_bis', 'z2_bis']]],
'info': {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/example/'}
}
}
In [2]: from ntv_numpy import Xdataset
x_example = Xdataset.read_json(example)
x_example.info
Out[2]: {'name': 'example',
'xtype': 'group',
'data_vars': ['var1', 'var2'],
'data_arrays': ['z_bis'],
'dimensions': ['x', 'y'],
'coordinates': ['ranking', 'z'],
'additionals': ['var1.mask1', 'var1.mask2', 'var1.variance', 'z.uncertainty'],
'metadata': ['info'],
'validity': 'undefined',
'length': 4,
'width': 12}
```
The JSON representation is equivalent to the Xdataset entity (Json conversion reversible)
```python
In [3]: x_json = x_example.to_json()
x_example_json = Xdataset.read_json(x_json)
x_example_json == x_example
Out[2]: True
```
### Xarray interoperability
```python
In [4]: x_xarray = x_example.to_xarray()
print(x_xarray)
Out[4]: <xarray.Dataset> Size: 182B
Dimensions: (x: 2, y: 2)
Coordinates:
* x (x) <U6 48B '23F0AE' '578B98'
* y (y) datetime64[ns] 16B 2021-01-01 2022-02-02
ranking (x, y) int32 16B 1 2 3 4
z (x) float64 16B 10.0 20.0
var1.mask1 (x) bool 2B True False
var1.mask2 (x, y) bool 4B True False False True
var1.variance (x, y) float64 32B 0.1 0.2 0.3 0.4
z.uncertainty (x) float64 16B 0.1 0.2
Data variables:
var1 (x, y) float64 32B 10.1 0.4 3.4 8.2
Attributes:
info: {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/...
name: example
var2: [['var2.ntv'], ['x', 'y']]
z_bis: [['string', ['z1_bis', 'z2_bis']]]
```
Reversibility:
```python
In [3]: x_example_xr = Xdataset.from_xarray(x_xarray)
x_example_xr == x_example_json == x_example
Out[2]: True
```
### scipp interoperability
```python
In [4]: x_scipp = x_example.to_scipp()
print(x_scipp['example'])
Out[4]: <scipp.Dataset>
Dimensions: Sizes[x:string:2, y:date:2, ]
Coordinates:
* ranking:month int32 [dimensionless] (x:string, y:date) [1, 2, 3, 4]
* x:string string [dimensionless] (x:string) ["23F0AE", "578B98"]
* y:date datetime64 [ns] (y:date) [2021-01-01T00:00:00.000000000, 2022-02-02T00:00:00.000000000]
* z:float float64 [dimensionless] (x:string) [10, 20]
Data:
var1:float float64 [kg] (x:string, y:date) [10.1, 0.4, 3.4, 8.2] [0.1, 0.2, 0.3, 0.4]
Masks:
mask1:boolean bool [dimensionless] (x:string) [True, False]
mask2:boolean bool [dimensionless] (x:string, y:date) [True, False, False, True]
```
Reversibility:
```python
In [3]: x_example_sc = Xdataset.from_scipp(x_scipp)
x_example_sc == x_example_xr == x_example_json == x_example
Out[2]: True
```
### NDData interoperability
```python
In [1]: example = {
'example:xdataset': {
'data': [['float[erg/s]', [1,2,3,4]]],
'data.mask': [[[False, False, True, True]]],
'data.uncertainty': [['float64[std]', [1.0, 1.414, 1.732, 2.0]]],
'meta': {'object': 'fictional data.'},
'wcs': {'WCSAXES': 2, 'CRPIX1': 2048.0, 'CRPIX2': 1024.0, 'PC1_1': 1.2905625619716e-05,
'PC1_2': 5.9530912331034e-06, 'PC2_1': 5.0220581265601e-06, 'PC2_2': -1.2644774105568e-05,
'CDELT1': 1.0, 'CDELT2': 1.0, 'CUNIT1': 'deg', 'CUNIT2': 'deg', 'CTYPE1': 'RA---TAN',
'CTYPE2': 'DEC--TAN', 'CRVAL1': 5.63056810618, 'CRVAL2': -72.05457184279, 'LONPOLE': 180.0,
'LATPOLE': -72.05457184279, 'WCSNAME': 'IDC_qbu1641sj', 'MJDREF': 0.0, 'RADESYS': 'ICRS'},
'psf': [['float[erg/s]', [1,2,3,4]]]
}
}
n_example = Xdataset.read_json(example)
n_example.info
Out[4]: {'name': 'example',
'xtype': 'group',
'data_arrays': ['data', 'psf'],
'additionals': ['data.mask', 'data.uncertainty'],
'metadata': ['meta', 'wcs'],
'validity': 'valid',
'width': 6}
```
```python
In [4]: n_nddata = n_example.to_nddata()
n_nddata
Out[4]: NDData([1., 2., ——, ——], unit='erg / s')
```
Reversibility:
```python
In [5]: n_example_ndd = Xdataset.from_nddata(n_nddata)
n_example_ndd == n_example
Out[5]: True
```
## URI usage
In the example, only structural data is exchanged with json format.
```python
In [1]: example = {
'example:xdataset': {
'var1': [['float[kg]', [2, 2], 'var1.ntv'], ['x', 'y']],
'var1.variance': [[[2, 2], 'var1_variance.ntv']],
'var1.mask1': [['var1_mask1.ntv'], ['x']],
'var1.mask2': [[[2, 2], 'var1_mask2.ntv']],
'var2': [['var2.ntv'], ['x', 'y']],
'x': [['x.ntv'], {'test': 21}],
'y': [['date', 'y.ntv']],
'ranking': [['month', [2, 2], 'ranking.ntv'], ['var1']],
'z': [['float', 'z.ntv'], ['x']],
'z.uncertainty': [['z_uncertainty.ntv']],
'z_bis': [['z_bis.ntv']],
'info': {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/example/'}
}
}
```
The complete example can be rebuild with loading data (path + file name).
```python
In [5]: # simulation of reading files at the indicated "path"
var1 = np.array([10.1, 0.4, 3.4, 8.2])
var1_variance = Ndarray([0.1, 0.2, 0.3, 0.4], ntv_type='float')
var1_mask1 = np.array([True, False])
var1_mask2 = np.array([True, False, False, True])
var2 = Ndarray('var2.ntv')
x = np.array(['23F0AE', '578B98'])
y = np.array(['2021-01-01', '2022-02-02'], dtype='datetime64[D]')
ranking = np.array([1, 2, 3, 4])
z = np.array([10.0, 20.0])
z_uncertainty = np.array([0.1, 0.2])
z_bis = np.array(['z1_bis', 'z2_bis'])
array_data = [var1, var1_variance, var1_mask1, var1_mask2, var2, x, y, ranking, z, z_uncertainty, z_bis]
x_example_mixte_numpy = copy(x_example_mixte)
for data, xnda in zip(array_data, x_example_mixte_numpy.xnd):
xnda.set_ndarray(Ndarray(data))
x_example_mixte_numpy == x_example_mixte_json == x_example_sc == x_example_xr == x_example_json == x_example
Out[5]: True
```
Raw data
{
"_id": null,
"home_page": "https://github.com/loco-philippe/ntv-numpy/blob/main/README.md",
"name": "ntv-numpy",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.9",
"maintainer_email": null,
"keywords": "numpy, JSON-NTV, semantic JSON, development, environmental data, multidimensional",
"author": "Philippe Thomy",
"author_email": "philippe@loco-labs.io",
"download_url": "https://files.pythonhosted.org/packages/60/d3/cbd59ab1ea1da456f7c7e205fd8ae2335bca68b5dc51f9938105ce36356c/ntv_numpy-0.1.2.tar.gz",
"platform": null,
"description": "### *NTV-NumPy : A multidimensional semantic, compact and reversible format for interoperability*\r\n\r\nFor more information, see the [user guide](https://loco-philippe.github.io/ntv-numpy/docs/user_guide.html) or the [github repository](https://github.com/loco-philippe/ntv-numpy).\r\n\r\n# Why a new format for multidimensional data ?\r\n\r\nEach tool has a specific structure for processing multidimensional data with the following consequences:\r\n\r\n- interfaces dedicated to each tool,\r\n- partially processed data,\r\n- no unified representation of data structures\r\n\r\nThe proposed format is based on the following principles:\r\n\r\n- neutral format available for tabular or multidimensional tools (e.g. Numpy, pandas, xarray, scipp, astropy),\r\n- taking into account a wide variety of data types as defined in [NTV](https://www.ietf.org/archive/id/draft-thomy-json-ntv-02.html) format,\r\n- high interoperability: reversible (lossless round-trip) interface with tabular or multidimensional tools,\r\n- reversible and compact JSON format (including categorical and sparse format),\r\n- Ease of sharing and exchanging multidimensional and tabular data,\r\n\r\n## main features\r\n\r\nThe NTV-Numpy converter uses this format to:\r\n\r\n- provide lossless and reversible interfaces with multidimensional and tabular data processing tools,\r\n- offer data exchange and sharing solutions with neutral or standardized formats (e.g. JSON, Numpy).\r\n\r\nNTV-NumPy was developped originally in the [json-NTV project](https://github.com/loco-philippe/NTV)\r\n\r\n## example\r\n\r\nIn the example below, a dataset available in JSON is shared with scipp or Xarray.\r\n\r\n```mermaid\r\n---\r\ntitle: Example of interoperability\r\n---\r\nflowchart LR\r\n A[Xarray] <--lossless--> B[Neutral\\nXdataset]\r\n B <--lossless--> C[NDData]\r\n D[Scipp] <--lossless--> B\r\n B <--lossless--> E[JSON]\r\n```\r\n\r\n### Data example\r\n\r\n```python\r\nIn [1]: example = {\r\n 'example:xdataset': {\r\n 'var1': [['float[kg]', [2, 2], [10.1, 0.4, 3.4, 8.2]], ['x', 'y']],\r\n 'var1.variance': [[[2, 2], [0.1, 0.2, 0.3, 0.4]]],\r\n 'var1.mask1': [[[True, False]], ['x']],\r\n 'var1.mask2': [[[2, 2], [True, False, False, True]]],\r\n \r\n 'var2': [['var2.ntv'], ['x', 'y']], \r\n \r\n 'x': [['string', ['23F0AE', '578B98']], {'test': 21}],\r\n 'y': [['date', ['2021-01-01', '2022-02-02']]],\r\n \r\n 'ranking': [['month', [2, 2], [1, 2, 3, 4]], ['var1']],\r\n 'z': [['float', [10, 20]], ['x']],\r\n 'z.uncertainty': [[[0.1, 0.2]]],\r\n \r\n 'z_bis': [[['z1_bis', 'z2_bis']]],\r\n \r\n 'info': {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/example/'}\r\n }\r\n }\r\n\r\nIn [2]: from ntv_numpy import Xdataset\r\n\r\n x_example = Xdataset.read_json(example)\r\n x_example.info\r\nOut[2]: {'name': 'example',\r\n 'xtype': 'group',\r\n 'data_vars': ['var1', 'var2'],\r\n 'data_arrays': ['z_bis'],\r\n 'dimensions': ['x', 'y'],\r\n 'coordinates': ['ranking', 'z'],\r\n 'additionals': ['var1.mask1', 'var1.mask2', 'var1.variance', 'z.uncertainty'],\r\n 'metadata': ['info'],\r\n 'validity': 'undefined',\r\n 'length': 4,\r\n 'width': 12}\r\n```\r\n\r\nThe JSON representation is equivalent to the Xdataset entity (Json conversion reversible)\r\n\r\n```python\r\nIn [3]: x_json = x_example.to_json()\r\n x_example_json = Xdataset.read_json(x_json)\r\n x_example_json == x_example\r\nOut[2]: True\r\n```\r\n\r\n### Xarray interoperability\r\n\r\n```python\r\nIn [4]: x_xarray = x_example.to_xarray()\r\n print(x_xarray)\r\nOut[4]: <xarray.Dataset> Size: 182B\r\n Dimensions: (x: 2, y: 2)\r\n Coordinates:\r\n * x (x) <U6 48B '23F0AE' '578B98'\r\n * y (y) datetime64[ns] 16B 2021-01-01 2022-02-02\r\n ranking (x, y) int32 16B 1 2 3 4\r\n z (x) float64 16B 10.0 20.0\r\n var1.mask1 (x) bool 2B True False\r\n var1.mask2 (x, y) bool 4B True False False True\r\n var1.variance (x, y) float64 32B 0.1 0.2 0.3 0.4\r\n z.uncertainty (x) float64 16B 0.1 0.2\r\n Data variables:\r\n var1 (x, y) float64 32B 10.1 0.4 3.4 8.2\r\n Attributes:\r\n info: {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/...\r\n name: example\r\n var2: [['var2.ntv'], ['x', 'y']]\r\n z_bis: [['string', ['z1_bis', 'z2_bis']]]\r\n```\r\n\r\nReversibility:\r\n\r\n```python\r\nIn [3]: x_example_xr = Xdataset.from_xarray(x_xarray)\r\n x_example_xr == x_example_json == x_example\r\nOut[2]: True\r\n```\r\n\r\n### scipp interoperability\r\n\r\n```python\r\nIn [4]: x_scipp = x_example.to_scipp()\r\n print(x_scipp['example'])\r\nOut[4]: <scipp.Dataset>\r\nDimensions: Sizes[x:string:2, y:date:2, ]\r\nCoordinates:\r\n* ranking:month int32 [dimensionless] (x:string, y:date) [1, 2, 3, 4]\r\n* x:string string [dimensionless] (x:string) [\"23F0AE\", \"578B98\"]\r\n* y:date datetime64 [ns] (y:date) [2021-01-01T00:00:00.000000000, 2022-02-02T00:00:00.000000000]\r\n* z:float float64 [dimensionless] (x:string) [10, 20]\r\nData:\r\n var1:float float64 [kg] (x:string, y:date) [10.1, 0.4, 3.4, 8.2] [0.1, 0.2, 0.3, 0.4]\r\n Masks:\r\n mask1:boolean bool [dimensionless] (x:string) [True, False]\r\n mask2:boolean bool [dimensionless] (x:string, y:date) [True, False, False, True]\r\n```\r\n\r\nReversibility:\r\n\r\n```python\r\nIn [3]: x_example_sc = Xdataset.from_scipp(x_scipp)\r\n x_example_sc == x_example_xr == x_example_json == x_example\r\nOut[2]: True\r\n```\r\n\r\n### NDData interoperability\r\n\r\n```python\r\nIn [1]: example = {\r\n 'example:xdataset': {\r\n 'data': [['float[erg/s]', [1,2,3,4]]],\r\n 'data.mask': [[[False, False, True, True]]],\r\n 'data.uncertainty': [['float64[std]', [1.0, 1.414, 1.732, 2.0]]],\r\n 'meta': {'object': 'fictional data.'},\r\n 'wcs': {'WCSAXES': 2, 'CRPIX1': 2048.0, 'CRPIX2': 1024.0, 'PC1_1': 1.2905625619716e-05,\r\n 'PC1_2': 5.9530912331034e-06, 'PC2_1': 5.0220581265601e-06, 'PC2_2': -1.2644774105568e-05,\r\n 'CDELT1': 1.0, 'CDELT2': 1.0, 'CUNIT1': 'deg', 'CUNIT2': 'deg', 'CTYPE1': 'RA---TAN',\r\n 'CTYPE2': 'DEC--TAN', 'CRVAL1': 5.63056810618, 'CRVAL2': -72.05457184279, 'LONPOLE': 180.0,\r\n 'LATPOLE': -72.05457184279, 'WCSNAME': 'IDC_qbu1641sj', 'MJDREF': 0.0, 'RADESYS': 'ICRS'},\r\n 'psf': [['float[erg/s]', [1,2,3,4]]]\r\n }\r\n } \r\n n_example = Xdataset.read_json(example)\r\n n_example.info \r\nOut[4]: {'name': 'example',\r\n 'xtype': 'group',\r\n 'data_arrays': ['data', 'psf'],\r\n 'additionals': ['data.mask', 'data.uncertainty'],\r\n 'metadata': ['meta', 'wcs'],\r\n 'validity': 'valid',\r\n 'width': 6}\r\n```\r\n\r\n```python\r\nIn [4]: n_nddata = n_example.to_nddata()\r\n n_nddata\r\nOut[4]: NDData([1., 2., \u2014\u2014, \u2014\u2014], unit='erg / s')\r\n```\r\n\r\nReversibility:\r\n\r\n```python\r\nIn [5]: n_example_ndd = Xdataset.from_nddata(n_nddata)\r\n n_example_ndd == n_example\r\nOut[5]: True\r\n```\r\n\r\n## URI usage\r\n\r\nIn the example, only structural data is exchanged with json format.\r\n\r\n```python\r\nIn [1]: example = {\r\n 'example:xdataset': {\r\n 'var1': [['float[kg]', [2, 2], 'var1.ntv'], ['x', 'y']],\r\n 'var1.variance': [[[2, 2], 'var1_variance.ntv']],\r\n 'var1.mask1': [['var1_mask1.ntv'], ['x']],\r\n 'var1.mask2': [[[2, 2], 'var1_mask2.ntv']],\r\n \r\n 'var2': [['var2.ntv'], ['x', 'y']], \r\n \r\n 'x': [['x.ntv'], {'test': 21}],\r\n 'y': [['date', 'y.ntv']],\r\n \r\n 'ranking': [['month', [2, 2], 'ranking.ntv'], ['var1']],\r\n 'z': [['float', 'z.ntv'], ['x']],\r\n 'z.uncertainty': [['z_uncertainty.ntv']],\r\n \r\n 'z_bis': [['z_bis.ntv']],\r\n \r\n 'info': {'path': 'https://github.com/loco-philippe/ntv-numpy/tree/main/example/'}\r\n }\r\n }\r\n```\r\n\r\nThe complete example can be rebuild with loading data (path + file name).\r\n\r\n```python\r\nIn [5]: # simulation of reading files at the indicated \"path\"\r\n var1 = np.array([10.1, 0.4, 3.4, 8.2])\r\n var1_variance = Ndarray([0.1, 0.2, 0.3, 0.4], ntv_type='float')\r\n var1_mask1 = np.array([True, False])\r\n var1_mask2 = np.array([True, False, False, True])\r\n var2 = Ndarray('var2.ntv')\r\n x = np.array(['23F0AE', '578B98'])\r\n y = np.array(['2021-01-01', '2022-02-02'], dtype='datetime64[D]')\r\n ranking = np.array([1, 2, 3, 4])\r\n z = np.array([10.0, 20.0])\r\n z_uncertainty = np.array([0.1, 0.2])\r\n z_bis = np.array(['z1_bis', 'z2_bis'])\r\n\r\n array_data = [var1, var1_variance, var1_mask1, var1_mask2, var2, x, y, ranking, z, z_uncertainty, z_bis]\r\n\r\n x_example_mixte_numpy = copy(x_example_mixte)\r\n for data, xnda in zip(array_data, x_example_mixte_numpy.xnd):\r\n xnda.set_ndarray(Ndarray(data))\r\n\r\n x_example_mixte_numpy == x_example_mixte_json == x_example_sc == x_example_xr == x_example_json == x_example\r\nOut[5]: True\r\n```\r\n",
"bugtrack_url": null,
"license": null,
"summary": "NTV-NumPy : A multidimensional semantic, compact and reversible format for interoperability",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/loco-philippe/ntv-numpy/blob/main/README.md"
},
"split_keywords": [
"numpy",
" json-ntv",
" semantic json",
" development",
" environmental data",
" multidimensional"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2a2d77c783cc7cb88e1430faad8d91b46e9a6142728622e3788a7ae25bc8c3f2",
"md5": "5ad20bbb76cd4b080f0ff89ecbf213f2",
"sha256": "3c08c0fdb161fdfe8fd9745be134b6a522eca9d0f6a8ae90cd294fbd67656b31"
},
"downloads": -1,
"filename": "ntv_numpy-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5ad20bbb76cd4b080f0ff89ecbf213f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.9",
"size": 28958,
"upload_time": "2024-04-10T21:44:42",
"upload_time_iso_8601": "2024-04-10T21:44:42.877992Z",
"url": "https://files.pythonhosted.org/packages/2a/2d/77c783cc7cb88e1430faad8d91b46e9a6142728622e3788a7ae25bc8c3f2/ntv_numpy-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60d3cbd59ab1ea1da456f7c7e205fd8ae2335bca68b5dc51f9938105ce36356c",
"md5": "b4586904e4f8654c12fb65c876ed0e72",
"sha256": "cb02c86385c5fad95813c22c3ccb358876a364884cb36983a285be9c04f63f27"
},
"downloads": -1,
"filename": "ntv_numpy-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "b4586904e4f8654c12fb65c876ed0e72",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.9",
"size": 30524,
"upload_time": "2024-04-10T21:44:44",
"upload_time_iso_8601": "2024-04-10T21:44:44.708701Z",
"url": "https://files.pythonhosted.org/packages/60/d3/cbd59ab1ea1da456f7c7e205fd8ae2335bca68b5dc51f9938105ce36356c/ntv_numpy-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-10 21:44:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "loco-philippe",
"github_project": "ntv-numpy",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "ntv-numpy"
}
JSON
