# IMCtermite
_IMCtermite_ provides access to the proprietary data format
_IMC2 Data Format_ with the file extension _.raw_ (or .dat) introduced and developed by
[imc Test & Measurement GmbH](https://www.imc-tm.de/). This data format is
employed i.a. by the measurement hardware
[imc CRONOSflex](https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/)
to dump and store data and the software packages
[imc Studio](https://www.imc-tm.de/produkte/messtechnik-software/imc-studio/ueberblick/)
& [imc FAMOS](https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/)
for measurement data control and analysis. Thanks to the integrated Python module,
the extracted measurement data can be stored in any open-source file format
accessible by Python like i.a. _csv_, _json_ or _parquet_.
On the [Record Evolution Platform](https://www.record-evolution.de/en/home-en/),
the library can be used both as a command line tool for interactive usage and as a
Python module to integrate the _.raw_ format into any ETL workflow.
## Overview
* [File format](#Fileformat)
* [Build and Installation](#Installation)
* [Usage and Examples](#Usage)
* [References](#References)
## File format
A file of the _IMC2 Data Format_ type with extension _.raw_ (or .dat) is a _mixed text/binary
file_ featuring a set of markers (keys) that indicate the start of various blocks
of data that provide meta information and the actual measurement data. Every single
marker is introduced by the character `"|" = 0x 7c` followed by two uppercase letters that
characterize the type of marker. Each block is further divided into several
parameters separated by commata `"," = 0x 2c` and terminated by a semicolon
`";" = 0x 3b`. For instance, the header - first 600 bytes - of a raw file may
look like this (in UTF-8 encoding):
```
|CF,2,1,1;|CK,1,3,1,1;
|NO,1,86,0,78,imc STUDIO 5.0 R10 (04.08.2017)@imc DEVICES 2.9R7 (25.7.2017)@imcDev__15190567,0,;
|CG,1,5,1,1,1; |CD,2, 63, 5.0000000000000001E-03,1,1,s,0,0,0, 0.0000000000000000E+00,1;
|NT,1,16,1,1,1980,0,0,0.0; |CC,1,3,1,1;|CP,1,16,1,4,7,32,0,0,1,0;
|CR,1,60,0, 1.0000000000000000E+00, 0.0000000000000000E+00,1,4,mbar;|CN,1,27,0,0,0,15,pressure_Vacuum,0,;
|Cb,1, 117,1,0, 1, 1, 0, 9608, 0, 9608,1, 2.0440300000000000E+03, 1.2416717060000000E+09,;
|CS,1, 9619, 1,�oD �nD6�nD)�nD�
```
Line breaks are introduced for readability. Most of the markers introduce
blocks of text, while only the last block identified by `|CS` contains binary data.
The format supports the storage of _multiple data sets (channels)_ in a single
file. The channels may be ordered in _multiplex_ mode (ordering w.r.t. time) or
_block_ mode (ordering w.r.t. to channels).
The markers (keys) are introduced by `"|" = 0x 7c` followed by two uppercase
letters. There are _two types_ of markers distinguished by the first letter:
1. _critical_ markers: introduced by `|C` featuring uppercase `C`
1. _noncritical_ markers: introduced by `|N` featuring uppercase `N`
The second letter represents further details of the specific key. Note that
while the _noncritical_ keys are optional, any _.raw_ file _cannot be_ correctly
decoded if any of the _critical_ markers are misinterpreted, invalid or damaged.
The second uppercase letter is followed by the first comma and the _version_
of the key starting from 1. After the next comma, an _(long) integer_ (in text
representation) specifies the length of the entire block, i.e. the number of
bytes between the following comma and the block-terminating semicolon. The further
structure of a block is not defined and may feature different numbers of additional
parameters. The format allows for any number of carriage returns (`CR = 0x0d`)
and line feeds (`LF = 0x 0a`) between keys, i.e. the block-terminating semicolon
and the vertical bar (pipe) of the next key. The following _critical markers_
are defined:
| marker | description |
|--------|-----------------------------------------------------------------------------------------------------|
| CF | format version and processor |
| CK | start of group of keys, no. parameters = 3, indicates (in)correct closure of the measurement series |
| CB | defines a group of channels |
| CT | text definition including group association index |
| CG | introduces group of components corresponding to CC keys |
| CD1,2 | old/new version of abscissa description |
| CZ | scaling of z-axis for segments |
| CC | start of a component |
| CP | information about buffer, datatype and samples of component |
| Cb | buffer description |
| CR | permissible range of values in component |
| CN | name and comment of channel |
| CS | raw binary data |
| CI | single numerical value (including unit) |
| Ca | add reference key |
Among the _noncritical_ markers, there are
| marker | description |
|--------|--------------------------------------------|
| NO | origin of data |
| NT | timestamp of trigger |
| ND | (color) display properties |
| NU | user defined key |
| Np | property of a channel |
| NE | extraction rule for channels from BUS data |
The format loosely defines some rules for the ordering of the markers in the
file stream. The rules for critical keys include: _CK_ has to follow up on _CF_,
_CK_ may be followed by any number of _CG_ blocks, each _CG_ has to be followed
by (any number of) component sequences comprised of the series _CC_ , _CP_,
(_CR_), (_ND_) and terminated by either _CS_ or the start of a new group,
component, text field or buffer.
## Installation
The _IMCtermite_ library may be employed both as a _CLI_ tool and a _python_
module.
### CLI tool
To build the CLI tool locally, use the default target `make` resulting
in the binary `imctermite`. To ensure system-wide availability, the installation
of the tool (in the default location `/usr/local/bin`) is done via
```
make install
````
which may require root permissions.
### Python
To integrate the library into a customized ETL toolchain, several cython targets
are available. For a local build that enables you to run the examples, use:
```
make cython-build
```
However, in a production environment, a proper installation of the module with
`make cython-install` is recommended for system-wide availability of the module.
#### Installation with pip
The package is also available in the [Python Package Index](https://pypi.org)
at [IMCtermite](https://pypi.org/project/IMCtermite/).
To install the latest version simply do
```Shell
python3 -m pip install IMCtermite
```
which provides binary wheels for multiple architectures on _Windows_ and _Linux_
and most _Python 3.x_ distributions. However, if your platform/architecture is
not supported you can still compile the source distribution yourself, which
requires _python3_setuptools_ and an up-to-date compiler supporting C++11
standard (e.g. _gcc version >= 10.2.0_).
## Usage
### CLI
The usage of the `imctermite` binary looks like this:
```
imctermite <raw-file> [options]
```
You have to provide a single _raw_ file and any option to specify what
to do with the data. All available options can be listed with `imctermite --help`:
```
Options:
-c, --listchannels list channels
-b, --listblocks list IMC key-blocks
-d, --output output directory to print channels
-s, --delimiter csv delimiter/separator char for output
-h, --help show this help message
-v, --version display version
```
For instance, to show a list of all channels included in `sample-data.raw`, you
do `imctermite sample-data.raw --listchannels`. No output files are
written by default. Output files are written only when an existing (!) directory
is provided as argument to the `--output` option. By default, every output file
is written using a `,` delimiter. You may provide any custom separator with the
option `--delimiter`. For example, in order to use `|`, the binary is called with
options `imctermite sample-data.raw -b -c -s '|'`.
### Python
Given the `IMCtermite` module is available, we can import it and declare an instance
of it by passing a _raw_ file to the constructor:
```Python
import IMCtermite
imcraw = IMCtermite.imctermite(b"sample/sampleA.raw")
```
An example of how to create an instance and obtain the list of channels is:
```Python
import IMCtermite
# declare and initialize instance of "imctermite" by passing a raw-file
try :
imcraw = IMCtermite.imctermite(b"samples/sampleA.raw")
except RuntimeError as e :
print("failed to load/parse raw-file: " + str(e))
# obtain list of channels as list of dictionaries (without data)
channels = imcraw.get_channels(False)
print(channels)
```
A more complete [example](python/examples/usage.py), including the methods for
obtaining the channels, i.a. their data and/or directly printing them to files,
can be found in the `python/examples` folder.
## References
### IMC
- https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/funktionen/im-und-export/
- https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/
- https://www.imc-tm.de/download-center/produkt-downloads/imc-famos/handbuecher
- https://www.imc-tm.de/fileadmin/Public/Downloads/Manuals/imc_FAMOS/imcGemeinsameKomponenten.pdf
- https://github.com/Apollo3zehn/ImcFamosFile
- https://apollo3zehn.github.io/ImcFamosFile/api/ImcFamosFile.FamosFileKeyType.html
### Cython
- https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html
### PyPI
- https://pypi.org/help/#apitoken
- https://sgoel.dev/posts/uploading-binary-wheels-to-pypi-from-github-actions/
- https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun
- https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml
- https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/
- https://github.com/actions/download-artifact#download-all-artifacts
- https://github.com/actions/download-artifact?tab=readme-ov-file#download-multiple-filtered-artifacts-to-the-same-directory
### iconv
- https://www.gnu.org/software/libiconv/
- https://vcpkg.io/en/packages.html
- https://vcpkg.io/en/getting-started
Raw data
{
"_id": null,
"home_page": "https://github.com/RecordEvolution/IMCtermite.git",
"name": "IMCtermite",
"maintainer": "Record Evolution GmbH",
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "IMC, raw, imcFAMOS, imcSTUDIO, imcCRONOS",
"author": "Record Evolution GmbH",
"author_email": "mario.fink@record-evolution.de",
"download_url": "https://files.pythonhosted.org/packages/46/81/48abd9936b18958483b1f0ed67c39cda5d421096adb325bc17a1488fc2b4/IMCtermite-2.1.10.tar.gz",
"platform": null,
"description": "# IMCtermite\n\n_IMCtermite_ provides access to the proprietary data format\n_IMC2 Data Format_ with the file extension _.raw_ (or .dat) introduced and developed by\n[imc Test & Measurement GmbH](https://www.imc-tm.de/). This data format is\nemployed i.a. by the measurement hardware\n[imc CRONOSflex](https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/)\nto dump and store data and the software packages\n[imc Studio](https://www.imc-tm.de/produkte/messtechnik-software/imc-studio/ueberblick/)\n& [imc FAMOS](https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/)\nfor measurement data control and analysis. Thanks to the integrated Python module,\nthe extracted measurement data can be stored in any open-source file format\naccessible by Python like i.a. _csv_, _json_ or _parquet_.\n\nOn the [Record Evolution Platform](https://www.record-evolution.de/en/home-en/),\nthe library can be used both as a command line tool for interactive usage and as a\nPython module to integrate the _.raw_ format into any ETL workflow.\n\n## Overview\n\n* [File format](#Fileformat)\n* [Build and Installation](#Installation)\n* [Usage and Examples](#Usage)\n* [References](#References)\n\n## File format\n\nA file of the _IMC2 Data Format_ type with extension _.raw_ (or .dat) is a _mixed text/binary\nfile_ featuring a set of markers (keys) that indicate the start of various blocks\nof data that provide meta information and the actual measurement data. Every single\nmarker is introduced by the character `\"|\" = 0x 7c` followed by two uppercase letters that\ncharacterize the type of marker. Each block is further divided into several\nparameters separated by commata `\",\" = 0x 2c` and terminated by a semicolon\n`\";\" = 0x 3b`. For instance, the header - first 600 bytes - of a raw file may\nlook like this (in UTF-8 encoding):\n\n```\n|CF,2,1,1;|CK,1,3,1,1;\n|NO,1,86,0,78,imc STUDIO 5.0 R10 (04.08.2017)@imc DEVICES 2.9R7 (25.7.2017)@imcDev__15190567,0,;\n|CG,1,5,1,1,1; |CD,2, 63, 5.0000000000000001E-03,1,1,s,0,0,0, 0.0000000000000000E+00,1;\n|NT,1,16,1,1,1980,0,0,0.0; |CC,1,3,1,1;|CP,1,16,1,4,7,32,0,0,1,0;\n|CR,1,60,0, 1.0000000000000000E+00, 0.0000000000000000E+00,1,4,mbar;|CN,1,27,0,0,0,15,pressure_Vacuum,0,;\n|Cb,1, 117,1,0, 1, 1, 0, 9608, 0, 9608,1, 2.0440300000000000E+03, 1.2416717060000000E+09,;\n|CS,1, 9619, 1,\ufffdoD\t\ufffdnD6\ufffdnD)\ufffdnD\ufffd\n```\n\nLine breaks are introduced for readability. Most of the markers introduce\nblocks of text, while only the last block identified by `|CS` contains binary data.\nThe format supports the storage of _multiple data sets (channels)_ in a single\nfile. The channels may be ordered in _multiplex_ mode (ordering w.r.t. time) or\n_block_ mode (ordering w.r.t. to channels).\n\nThe markers (keys) are introduced by `\"|\" = 0x 7c` followed by two uppercase\nletters. There are _two types_ of markers distinguished by the first letter:\n\n1. _critical_ markers: introduced by `|C` featuring uppercase `C`\n1. _noncritical_ markers: introduced by `|N` featuring uppercase `N` \n\nThe second letter represents further details of the specific key. Note that\nwhile the _noncritical_ keys are optional, any _.raw_ file _cannot be_ correctly\ndecoded if any of the _critical_ markers are misinterpreted, invalid or damaged.\nThe second uppercase letter is followed by the first comma and the _version_\nof the key starting from 1. After the next comma, an _(long) integer_ (in text\nrepresentation) specifies the length of the entire block, i.e. the number of\nbytes between the following comma and the block-terminating semicolon. The further\nstructure of a block is not defined and may feature different numbers of additional\nparameters. The format allows for any number of carriage returns (`CR = 0x0d`)\nand line feeds (`LF = 0x 0a`) between keys, i.e. the block-terminating semicolon\nand the vertical bar (pipe) of the next key. The following _critical markers_\nare defined:\n\n\n| marker | description |\n|--------|-----------------------------------------------------------------------------------------------------|\n| CF | format version and processor |\n| CK | start of group of keys, no. parameters = 3, indicates (in)correct closure of the measurement series |\n| CB | defines a group of channels |\n| CT | text definition including group association index |\n| CG | introduces group of components corresponding to CC keys |\n| CD1,2 | old/new version of abscissa description |\n| CZ | scaling of z-axis for segments |\n| CC | start of a component |\n| CP | information about buffer, datatype and samples of component |\n| Cb | buffer description |\n| CR | permissible range of values in component |\n| CN | name and comment of channel |\n| CS | raw binary data |\n| CI | single numerical value (including unit) |\n| Ca | add reference key |\n\nAmong the _noncritical_ markers, there are\n\n| marker | description |\n|--------|--------------------------------------------|\n| NO | origin of data |\n| NT | timestamp of trigger |\n| ND | (color) display properties |\n| NU | user defined key |\n| Np | property of a channel |\n| NE | extraction rule for channels from BUS data |\n\nThe format loosely defines some rules for the ordering of the markers in the\nfile stream. The rules for critical keys include: _CK_ has to follow up on _CF_,\n_CK_ may be followed by any number of _CG_ blocks, each _CG_ has to be followed\nby (any number of) component sequences comprised of the series _CC_ , _CP_,\n(_CR_), (_ND_) and terminated by either _CS_ or the start of a new group,\ncomponent, text field or buffer.\n\n## Installation\n\nThe _IMCtermite_ library may be employed both as a _CLI_ tool and a _python_\nmodule.\n\n### CLI tool\n\nTo build the CLI tool locally, use the default target `make` resulting\nin the binary `imctermite`. To ensure system-wide availability, the installation\nof the tool (in the default location `/usr/local/bin`) is done via\n\n```\nmake install\n````\n\nwhich may require root permissions.\n\n### Python\n\nTo integrate the library into a customized ETL toolchain, several cython targets\nare available. For a local build that enables you to run the examples, use:\n\n```\nmake cython-build\n```\n\nHowever, in a production environment, a proper installation of the module with\n`make cython-install` is recommended for system-wide availability of the module.\n\n#### Installation with pip\n\nThe package is also available in the [Python Package Index](https://pypi.org)\nat [IMCtermite](https://pypi.org/project/IMCtermite/).\nTo install the latest version simply do\n\n```Shell\npython3 -m pip install IMCtermite\n```\n\nwhich provides binary wheels for multiple architectures on _Windows_ and _Linux_\nand most _Python 3.x_ distributions. However, if your platform/architecture is\nnot supported you can still compile the source distribution yourself, which\nrequires _python3_setuptools_ and an up-to-date compiler supporting C++11\nstandard (e.g. _gcc version >= 10.2.0_).\n\n## Usage\n\n### CLI\n\nThe usage of the `imctermite` binary looks like this:\n\n```\nimctermite <raw-file> [options]\n```\n\nYou have to provide a single _raw_ file and any option to specify what\nto do with the data. All available options can be listed with `imctermite --help`:\n\n```\nOptions:\n\n -c, --listchannels list channels\n -b, --listblocks list IMC key-blocks\n -d, --output output directory to print channels\n -s, --delimiter csv delimiter/separator char for output\n -h, --help show this help message\n -v, --version display version\n```\n\nFor instance, to show a list of all channels included in `sample-data.raw`, you\ndo `imctermite sample-data.raw --listchannels`. No output files are\nwritten by default. Output files are written only when an existing (!) directory\nis provided as argument to the `--output` option. By default, every output file\nis written using a `,` delimiter. You may provide any custom separator with the\noption `--delimiter`. For example, in order to use `|`, the binary is called with\noptions `imctermite sample-data.raw -b -c -s '|'`.\n\n### Python\n\nGiven the `IMCtermite` module is available, we can import it and declare an instance\nof it by passing a _raw_ file to the constructor:\n\n```Python\nimport IMCtermite\n\nimcraw = IMCtermite.imctermite(b\"sample/sampleA.raw\")\n```\n\nAn example of how to create an instance and obtain the list of channels is:\n\n```Python\nimport IMCtermite\n\n# declare and initialize instance of \"imctermite\" by passing a raw-file\ntry :\n imcraw = IMCtermite.imctermite(b\"samples/sampleA.raw\")\nexcept RuntimeError as e :\n print(\"failed to load/parse raw-file: \" + str(e))\n\n# obtain list of channels as list of dictionaries (without data)\nchannels = imcraw.get_channels(False)\nprint(channels)\n```\n\nA more complete [example](python/examples/usage.py), including the methods for\nobtaining the channels, i.a. their data and/or directly printing them to files,\ncan be found in the `python/examples` folder.\n\n## References\n\n### IMC\n\n- https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/funktionen/im-und-export/\n- https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/\n- https://www.imc-tm.de/download-center/produkt-downloads/imc-famos/handbuecher\n- https://www.imc-tm.de/fileadmin/Public/Downloads/Manuals/imc_FAMOS/imcGemeinsameKomponenten.pdf\n- https://github.com/Apollo3zehn/ImcFamosFile\n- https://apollo3zehn.github.io/ImcFamosFile/api/ImcFamosFile.FamosFileKeyType.html\n\n### Cython\n\n- https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html\n\n### PyPI\n\n- https://pypi.org/help/#apitoken\n- https://sgoel.dev/posts/uploading-binary-wheels-to-pypi-from-github-actions/\n- https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun\n- https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml\n- https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/\n- https://github.com/actions/download-artifact#download-all-artifacts\n- https://github.com/actions/download-artifact?tab=readme-ov-file#download-multiple-filtered-artifacts-to-the-same-directory\n\n### iconv\n\n- https://www.gnu.org/software/libiconv/\n- https://vcpkg.io/en/packages.html\n- https://vcpkg.io/en/getting-started\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS and imcSTUDIO and facilitates its storage in open source file formats",
"version": "2.1.10",
"project_urls": {
"Homepage": "https://github.com/RecordEvolution/IMCtermite.git"
},
"split_keywords": [
"imc",
" raw",
" imcfamos",
" imcstudio",
" imccronos"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "468148abd9936b18958483b1f0ed67c39cda5d421096adb325bc17a1488fc2b4",
"md5": "c6171b8f83c7ef5dffe9fe1e1504120e",
"sha256": "c8811d3871bd425eceb0d8bf25c22152936f540a98615bade0d488e41a93f0f8"
},
"downloads": -1,
"filename": "IMCtermite-2.1.10.tar.gz",
"has_sig": false,
"md5_digest": "c6171b8f83c7ef5dffe9fe1e1504120e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 73255,
"upload_time": "2025-02-02T00:03:55",
"upload_time_iso_8601": "2025-02-02T00:03:55.799035Z",
"url": "https://files.pythonhosted.org/packages/46/81/48abd9936b18958483b1f0ed67c39cda5d421096adb325bc17a1488fc2b4/IMCtermite-2.1.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-02 00:03:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "RecordEvolution",
"github_project": "IMCtermite",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "imctermite"
}
JSON
