# HDF5 Master config
This repository contains the XML file that describes the mapping between ICAT database and the master file in HDF5 format. It also contains a set of [tools](#tools) that will help with the maintenance of it.
:warning: The file `hdf5_cfg.xml` is not longer updated and only exists for backward compatibility with `pyicat-plus<=0.1.7`.
The new definitions file is `src/icat_esrf_definitions/hdf5_cfg.xml`. Changes should be applied to the new file only.
# Techniques
1. TOMO
2. FLUO
3. KMAP
4. MX
5. BIOSAXS
6. PTYCHO
7. MRT
# Tools
## Requirements
### Proxy
In case of need a proxy should be well setup:
```
export https_proxy=https://proxy.esrf.fr:3128
```
### python-icat
Some tools will need python-icat version 0.12.0 and version 0.13.1 will not work
#### Installing python-icat
Go to the URL :
```
https://icatproject.org/user-documentation/python-icat/
```
and download file called python-icat-0.12.0.tar.gz:
```
-rw-r--r-- 1 blissadm bliss 272576 Oct 10 2016 python-icat-0.12.0.tar.gz
c9034ad2a725ba1317a911940c577a8b python-icat-0.12.0.tar.gz
```
This file can be found on:
```
/segfs/bliss/projects/ICAT/python-icat
-rw-r--r-- 1 homsrego soft 272576 Feb 12 16:48 python-icat-0.12.0.tar.gz
```
Uncompress and install the tar by using pip:
```
cd Downloads
tar -xvzf python-icat-0.12.0.tar.gz
cd python-icat-0.12.0
pip install .
```
Test:
```
~/icat/python-icat % python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import icat
>>> icat.__version__
'0.12.0'
```
## List of tools
Set of tools to validate the XML file
1. [XML Syntax Validation](#XML-syntax-validation): Checks if XML is well formed
2. [Parameters list](#parameters-list): Prints a list of parameters from XML
3. [ICAT Parameter list](#icat-parameter-list): Prints a list of parameters from ICAT DB
4. [Metadata Status](#metadata-status): Compares both metadata parameters from XML file and DB
5. [Add parameters](#add-parameters): It reads the xml file and proposes to create the missing parameters in ICAT. Missing parameters are the params defined in the XML that are not in the ICAT DB
6. [List techniques](#list-techniques): It reads the xml file are returns all the subentries that correspond to the techniques
## XML Syntax Validation
This tool checks the syntax of the hdf5_cfg.xml file. It will raise an error if the XML is not well formed
### How to run it?
```
python tools/xml/validate.py hdf5_cfg.xml
```
#### Example
Success:
```
lindemaria:~/Software/metadata/hdf5-master-config % python tools/validate.py hdf5_cfg.xml
[PASSED] hdf5_cfg.xml is well-formed
```
Error:
```
lindemaria:~/Software/metadata/hdf5-master-config % python tools/validate.py hdf5_cfg.xml
[ERROR] hdf5_cfg.xml is mismatched tag: line 84, column 3
```
## Parameters list
This tool prints in the standard output the list of parameters set in the hdf5_cgf.xml file
### How to run it?
```
cd tools
python -m list.list ../hdf5_cfg.xml
```
#### Example
```
scanName
scanNumber
proposal
scanType
[..]
```
## ICAT Parameter list
This tool prints in the standard output the list of parameters set in ICAT.
### Configuration Files
icat.cfg should exist on this location:
```
./tools/icat/icat.cfg
```
It is the file that contains the connection string to ICAT.
An example of such file you can find icat.cfg.example where password has been hidden
```
[ovm-icat2]
url = https://ovm-icat2.esrf.fr/ICATService/ICAT?wsdl
auth = db
username = root
password = ************
idsurl = https://ovm-icat2:8181/ids
# uncomment, if your server does not have a trusted certificate
checkCert = No
```
Note that auth is a controlled value list of: db | esrf
### How to run it?
```
cd tools/icat/list
python list.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy ""
```
## Metadata Status
This tool will look for the parameters on ICAT DB and then will compare with the parameters defined in the xml making a summary of the current status
### Configuration
icat.cfg should exist that it is the file that contains the connection string to ICAT.
An example of such file is icat.cfg.example where password has been hidden
### How to run it?
```
cd tools/icat/status
python status.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy ""
```
### Example
```
lindemaria:tools/icat/status % python status.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy ""
[ERROR] These parameters exists in ../../../hdf5_cfg.xml but not in the DB
MRT_beamHeight
MRT_beamSize
MRT_dose
MRT_expoSpeed
MRT_expoStart
[INFO] -----------------------------------------------------------------------------------------------------------
[INFO] Summary of metadata on https://icat.esrf.fr/ICATService/ICAT?wsdl database and ../../../hdf5_cfg.xml file
[INFO] Total 267 parameters defined in the XML file
[INFO] Total 387 parameters defined in the ICAT DB
[INFO] -----------------------------------------------------------------------------------------------------------
```
### Add parameters
### Configuration
icat.cfg should exist that it is the file that contains the connection string to ICAT.
An example of such file is icat.cfg.example where password has been hidden
### How to run it?
```
cd tools/icat/addParameters
python addParameters.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy ""
```
### List techniques
### How to run it?
```
lindemaria:~/Software/metadata/hdf5-master-config % python tools/xml/getTechniques.py hdf5_cfg.xml
['SAXS', 'MX', 'PTYCHO', 'TOMO', 'MRT', 'HOLO', 'WAXS']
```
```
Raw data
{
"_id": null,
"home_page": "https://gitlab.esrf.fr/icat/hdf5-master-config/",
"name": "icat-esrf-definitions",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "ESRF",
"author_email": "wout.de_nolf@esrf.fr",
"download_url": "https://files.pythonhosted.org/packages/fa/5f/0ad12f7662c12b6ac5400b895010aedd65f4ea0170a12024079f5b1a673b/icat_esrf_definitions-2.1.1.tar.gz",
"platform": null,
"description": "# HDF5 Master config\n\nThis repository contains the XML file that describes the mapping between ICAT database and the master file in HDF5 format. It also contains a set of [tools](#tools) that will help with the maintenance of it.\n\n:warning: The file `hdf5_cfg.xml` is not longer updated and only exists for backward compatibility with `pyicat-plus<=0.1.7`.\nThe new definitions file is `src/icat_esrf_definitions/hdf5_cfg.xml`. Changes should be applied to the new file only.\n\n# Techniques\n\n1. TOMO\n2. FLUO\n3. KMAP\n4. MX\n5. BIOSAXS\n6. PTYCHO\n7. MRT\n\n\n# Tools\n\n## Requirements\n\n### Proxy \n\nIn case of need a proxy should be well setup:\n```\nexport https_proxy=https://proxy.esrf.fr:3128\n```\n### python-icat\n\nSome tools will need python-icat version 0.12.0 and version 0.13.1 will not work\n\n#### Installing python-icat\n\nGo to the URL :\n```\nhttps://icatproject.org/user-documentation/python-icat/\n```\nand download file called python-icat-0.12.0.tar.gz:\n```\n-rw-r--r-- 1 blissadm bliss 272576 Oct 10 2016 python-icat-0.12.0.tar.gz\nc9034ad2a725ba1317a911940c577a8b python-icat-0.12.0.tar.gz\n```\nThis file can be found on:\n```\n/segfs/bliss/projects/ICAT/python-icat\n-rw-r--r-- 1 homsrego soft 272576 Feb 12 16:48 python-icat-0.12.0.tar.gz\n\n\n```\n\nUncompress and install the tar by using pip:\n```\ncd Downloads\ntar -xvzf python-icat-0.12.0.tar.gz\ncd python-icat-0.12.0\npip install .\n```\n\nTest:\n```\n~/icat/python-icat % python \nPython 2.7.9 (default, Jun 29 2016, 13:08:31) \n[GCC 4.9.2] on linux2\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n>>> import icat\n>>> icat.__version__\n'0.12.0'\n```\n\n\n## List of tools\n\nSet of tools to validate the XML file\n\n1. [XML Syntax Validation](#XML-syntax-validation): Checks if XML is well formed\n2. [Parameters list](#parameters-list): Prints a list of parameters from XML\n3. [ICAT Parameter list](#icat-parameter-list): Prints a list of parameters from ICAT DB\n4. [Metadata Status](#metadata-status): Compares both metadata parameters from XML file and DB\n5. [Add parameters](#add-parameters): It reads the xml file and proposes to create the missing parameters in ICAT. Missing parameters are the params defined in the XML that are not in the ICAT DB\n6. [List techniques](#list-techniques): It reads the xml file are returns all the subentries that correspond to the techniques\n\n## XML Syntax Validation\n\t\nThis tool checks the syntax of the hdf5_cfg.xml file. It will raise an error if the XML is not well formed\n\n### How to run it?\n\n```\npython tools/xml/validate.py hdf5_cfg.xml \n```\n\n#### Example\n\n\tSuccess:\n\t```\n\tlindemaria:~/Software/metadata/hdf5-master-config % python tools/validate.py hdf5_cfg.xml \n\t[PASSED] hdf5_cfg.xml is well-formed\n\t```\n\n\tError:\n\t```\n\tlindemaria:~/Software/metadata/hdf5-master-config % python tools/validate.py hdf5_cfg.xml \n\t[ERROR] hdf5_cfg.xml is mismatched tag: line 84, column 3\n\t```\n\n\n\n## Parameters list\n\nThis tool prints in the standard output the list of parameters set in the hdf5_cgf.xml file\n\n### How to run it?\n\n```\ncd tools\npython -m list.list ../hdf5_cfg.xml\n```\n\n#### Example\n\n```\nscanName\nscanNumber\nproposal\nscanType\n[..]\n```\n\n## ICAT Parameter list\n\nThis tool prints in the standard output the list of parameters set in ICAT. \n\n### Configuration Files\n\nicat.cfg should exist on this location:\n```\n./tools/icat/icat.cfg\n```\n\nIt is the file that contains the connection string to ICAT. \nAn example of such file you can find icat.cfg.example where password has been hidden\n\t\n```\n[ovm-icat2]\nurl = https://ovm-icat2.esrf.fr/ICATService/ICAT?wsdl\nauth = db\nusername = root\npassword = ************\nidsurl = https://ovm-icat2:8181/ids\n# uncomment, if your server does not have a trusted certificate\ncheckCert = No\n```\n\t\nNote that auth is a controlled value list of: db | esrf\n \n\n\n### How to run it?\n```\ncd tools/icat/list\npython list.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy \"\"\n```\n\n\n## Metadata Status\n\nThis tool will look for the parameters on ICAT DB and then will compare with the parameters defined in the xml making a summary of the current status\n\n### Configuration\n\n\ticat.cfg should exist that it is the file that contains the connection string to ICAT.\n\tAn example of such file is icat.cfg.example where password has been hidden\n\n### How to run it?\n```\ncd tools/icat/status\npython status.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy \"\"\n```\n### Example\n```\nlindemaria:tools/icat/status % python status.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy \"\"\n\n\n[ERROR] These parameters exists in ../../../hdf5_cfg.xml but not in the DB\n\nMRT_beamHeight\nMRT_beamSize\nMRT_dose\nMRT_expoSpeed\nMRT_expoStart\n\n\n[INFO] -----------------------------------------------------------------------------------------------------------\n[INFO] Summary of metadata on https://icat.esrf.fr/ICATService/ICAT?wsdl database and ../../../hdf5_cfg.xml file\n[INFO] Total 267 parameters defined in the XML file\n[INFO] Total 387 parameters defined in the ICAT DB\n[INFO] -----------------------------------------------------------------------------------------------------------\n```\n\n### Add parameters\n\n### Configuration\n\n\ticat.cfg should exist that it is the file that contains the connection string to ICAT.\n\tAn example of such file is icat.cfg.example where password has been hidden\n\n### How to run it?\n```\ncd tools/icat/addParameters\npython addParameters.py -s ovm-icat2 --no-check-certificate -c ../icat.cfg --https-proxy \"\"\n```\n\n### List techniques\n### How to run it?\n```\nlindemaria:~/Software/metadata/hdf5-master-config % python tools/xml/getTechniques.py hdf5_cfg.xml \n['SAXS', 'MX', 'PTYCHO', 'TOMO', 'MRT', 'HOLO', 'WAXS']\n```\n```\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "ESRF ICAT definitions",
"version": "2.1.1",
"project_urls": {
"Documentation": "https://gitlab.esrf.fr/icat/hdf5-master-config/",
"Homepage": "https://gitlab.esrf.fr/icat/hdf5-master-config/",
"Source": "https://gitlab.esrf.fr/icat/hdf5-master-config/",
"Tracker": "https://gitlab.esrf.fr/icat/hdf5-master-config/issues/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fa5f0ad12f7662c12b6ac5400b895010aedd65f4ea0170a12024079f5b1a673b",
"md5": "bd30390f5c17ef5b4a0a395a360c54ee",
"sha256": "790bc7f133984ed56ff3fa990a71e628726c8692c5bd2951bfc1b4b1c94773f5"
},
"downloads": -1,
"filename": "icat_esrf_definitions-2.1.1.tar.gz",
"has_sig": false,
"md5_digest": "bd30390f5c17ef5b4a0a395a360c54ee",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 22328,
"upload_time": "2024-04-19T12:25:51",
"upload_time_iso_8601": "2024-04-19T12:25:51.453704Z",
"url": "https://files.pythonhosted.org/packages/fa/5f/0ad12f7662c12b6ac5400b895010aedd65f4ea0170a12024079f5b1a673b/icat_esrf_definitions-2.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-19 12:25:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "icat-esrf-definitions"
}
JSON