The km3io Python package
========================
.. image:: https://git.km3net.de/km3py/km3io/badges/master/pipeline.svg
:target: https://git.km3net.de/km3py/km3io/pipelines
.. image:: https://git.km3net.de/km3py/km3io/badges/master/coverage.svg
:target: https://km3py.pages.km3net.de/km3io/coverage
.. image:: https://git.km3net.de/examples/km3badges/-/raw/master/docs-latest-brightgreen.svg
:target: https://km3py.pages.km3net.de/km3io
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7382620.svg
:target: https://doi.org/10.5281/zenodo.7382620
This software provides a set of Python classes to read KM3NeT ROOT files
without having ROOT, Jpp or aanet installed. It only depends on Python 3.5+ and the amazing `uproot <https://github.com/scikit-hep/uproot>`__ package and gives you access to the data via `numpy <https://www.numpy.org>`__ and `awkward <https://awkward-array.readthedocs.io>`__ arrays.
It's very easy to use and according to the `uproot <https://github.com/scikit-hep/uproot>`__ benchmarks, it is able to outperform the original ROOT I/O performance.
Installation
============
Install km3io using pip::
pip install km3io
or conda::
conda install km3io
To get the latest (stable) development release::
pip install git+https://git.km3net.de/km3py/km3io.git
Docker::
docker run -it docker.km3net.de/km3io
Singularity::
wget https://sftp.km3net.de/singularity/km3io_v0.27.2.sif # pick the version you like
singularity shell km3io_v0.27.2.sif
**Reminder:** km3io is **not** dependent on aanet, ROOT or Jpp!
Questions
=========
If you have a question about km3io, please proceed as follows:
- Read the documentation below.
- Explore the `examples <https://km3py.pages.km3net.de/km3io/examples.html>`__ in the documentation.
- Haven't you found an answer to your question in the documentation, post a git issue with your question showing us an example of what you have tried first, and what you would like to do.
- Have you noticed a bug, please post it in a git issue, we appreciate your contribution.
Introduction
------------
Most of km3net data is stored in root files. These root files are created using
the `KM3NeT Dataformat library
<https://git.km3net.de/common/km3net-dataformat>`__ A ROOT file created with
`Jpp <https://git.km3net.de/common/jpp>`__ is an "online" file and all other
software usually produces "offline" files.
km3io is a Python package that provides access to offline files with its
``OfflineReader`` class and a special one to read gSeaGen files. All of these
ROOT files can be read without installing any other software like Jpp, aanet or
ROOT. km3io v1.1 and earlier also support the access to online files (events,
summaryslices and timeslices). This feature has been dropped due to a lack of
mainteinance power and inf favour of the `KM3io.jl <https://git.km3net.de/common/KM3io.jl>`__` Julia Package, which
provides high-performances access to all ROOT files and should also be
prioritised over ``km3io`` when performance matters (which does, most of the
time).
Data in km3io is returned as ``awkward.Array`` which is an advance Numpy-like container type to store
contiguous data for high performance computations.
Such an ``awkward.Array`` supports any level of nested arrays and records which can have different lengths, in contrast to Numpy where everything has to be rectangular.
The example is shown below shows the array which contains the ``dir_z`` values
of each track of the first 4 events. The type ``4 * var * float64`` means that
it has 4 subarrays with variable lengths of type ``float64``:
.. code-block:: python3
>>> import km3io
>>> from km3net_testdata import data_path
>>> f = km3io.OfflineReader(data_path("offline/numucc.root"))
>>> f[:4].tracks.dir_z
<Array [[0.213, 0.213, ... 0.229, 0.323]] type='4 * var * float64'>
The same concept applies to all other branches, including ``hits``, ``mc_hits``,
``mc_tracks``, ``t_sec`` etc.
Architecture overview
---------------------
``km3io`` utilises ``uproot`` behind the scenes and creates a lazy and thin
wrapper which offers convenient slicing and iterations by delaying the access to
the actual ROOT data branches to the very last moment. When using the iteration
functionality, the data is loaded in chunks and the iteration is done over e.g.
events in each chunk or a bunch of frames in case of the summaryslice reader.
The base class for the event-based readout is the ``km3io.rootio.EventReader``
class. When subclassing this class, the branches, aliases and nested branches
need to be defined in the static variables which are then used to mask unwanted
attributes. Especially in case of the Offline ROOT format, where the "one class
fits all" design was chosen, it is distracting that e.g. a `Hit` has many
attributes which make no sense depending on the context (MC hit, raw hit etc.).
By specifing the branches explicitely, the user API will only expose the
meaningful fields.
The online ROOT format support is partly still based on ``uproot3``.
Many of the utility functions are using Numba to achieve the best possible
performance. ``km3io`` does not offer alternative implementations, so Numba is a
strict dependency and an integral part of the implementation.
Offline files reader
--------------------
In general an offline file has two attributes to access data: the header and the events. Let's start with the header.
Reading the file header
"""""""""""""""""""""""
To read an offline file start with opening it with the ``OfflineReader``:
.. code-block:: python3
>>> import km3io
>>> from km3net_testdata import data_path
>>> f = km3io.OfflineReader(data_path("offline/numucc.root"))
Accessing is as easy as typing:
.. code-block:: python3
>>> f.header
<km3io.offline.Header at 0x7fcd81025990>
Printing it will give an overview of the structure:
.. code-block:: python3
>>> print(f.header)
MC Header:
DAQ(livetime=394)
PDF(i1=4, i2=58)
can(zmin=0, zmax=1027, r=888.4)
can_user: can_user(field_0=0.0, field_1=1027.0, field_2=888.4)
coord_origin(x=0, y=0, z=0)
cut_in(Emin=0, Emax=0, cosTmin=0, cosTmax=0)
cut_nu(Emin=100, Emax=100000000.0, cosTmin=-1, cosTmax=1)
cut_primary(Emin=0, Emax=0, cosTmin=0, cosTmax=0)
cut_seamuon(Emin=0, Emax=0, cosTmin=0, cosTmax=0)
decay: decay(field_0='doesnt', field_1='happen')
detector: NOT
drawing: Volume
genhencut(gDir=2000, Emin=0)
genvol(zmin=0, zmax=1027, r=888.4, volume=2649000000.0, numberOfEvents=100000)
kcut: 2
livetime(numberOfSeconds=0, errorOfSeconds=0)
model(interaction=1, muon=2, scattering=0, numberOfEnergyBins=1, field_4=12)
ngen: 100000.0
norma(primaryFlux=0, numberOfPrimaries=0)
nuflux: nuflux(field_0=0, field_1=3, field_2=0, field_3=0.5, field_4=0.0, field_5=1.0, field_6=3.0)
physics(program='GENHEN', version='7.2-220514', date=181116, time=1138)
seed(program='GENHEN', level=3, iseed=305765867, field_3=0, field_4=0)
simul(program='JSirene', version=11012, date='11/17/18', time=7)
sourcemode: diffuse
spectrum(alpha=-1.4)
start_run(run_id=1)
target: isoscalar
usedetfile: false
xlat_user: 0.63297
xparam: OFF
zed_user: zed_user(field_0=0.0, field_1=3450.0)
To read the values in the header one can call them directly, as the structures
are simple ``namedtuple``-like objects:
.. code-block:: python3
>>> f.header.DAQ.livetime
394
>>> f.header.cut_nu.Emin
100
>>> f.header.genvol.numberOfEvents
100000
Reading offline events
""""""""""""""""""""""
Events are at the top level of an offline file, so that each branch of an event
is directly accessible at the ``OfflineReader`` instance. The ``.keys()`` method
can be used to list the available attributes. Notice that some of them are aliases
for backwards compatibility (like ``mc_tracks`` and ``mc_trks``). Another
backwards compatibility feature is the ``f.events`` attribute which is simply
mapping everything to ``f``, so that ``f.events.mc_tracks`` is the same as
``f.mc_tracks``.
.. code-block:: python3
>>> f
OfflineReader (10 events)
>>> f.keys()
{'comment', 'det_id', 'flags', 'frame_index', 'hits', 'id', 'index',
'mc_hits', 'mc_id', 'mc_run_id', 'mc_t', 'mc_tracks', 'mc_trks',
'n_hits', 'n_mc_hits', 'n_mc_tracks', 'n_mc_trks', 'n_tracks',
'n_trks', 'overlays', 'run_id', 't_ns', 't_sec', 'tracks',
'trigger_counter', 'trigger_mask', 'trks', 'usr', 'usr_names',
'w', 'w2list', 'w3list'}
>>> f.tracks
<Branch [10] path='trks'>
>>> f.events.tracks
<Branch [10] path='trks'>
The ``[10]`` denotes that there are ``10`` events available, each containing a sub-array of ``tracks``.
Using <TAB> completion gives an overview of available data. Alternatively the attribute `fields`
can be used on event-branches and to see what is available for reading.
.. code-block:: python3
>>> f.tracks.fields
['id',
'pos_x',
'pos_y',
'pos_z',
'dir_x',
'dir_y',
'dir_z',
't',
'E',
'len',
'lik',
'rec_type',
'rec_stages',
'fitinf']
Reading the reconstructed values like energy and direction of an event can be done with:
.. code-block:: python3
>>> f.events.tracks.E
<Array [[117, 117, 0, 0, 0, ... 0, 0, 0, 0, 0]] type='10 * var * float64'>
The ``Array`` in this case is an `awkward <https://awkward-array.readthedocs.io>`__ array with the data type
``10 * var * float64`` which means that there are ``10`` sub-arrays with ``var``iable lengths of type ``float64``.
Awkward arrays allow high-performance access to arrays which are not rectangular (in contrast to ``numpy``).
Read the documention of AwkwardArray to learn how to work with these structures efficiently. One example
to retrieve the energy of the very first reconstructed track for the first three events is:
.. code-block:: python3
>>> f.events.tracks.E[:3, 0]
<Array [117, 4.4e+03, 8.37] type='3 * float64'>
Online files reader
-------------------
The support to read online ROOT files has been dropped in ``km3io`` v1.2.
Raw data
{
"_id": null,
"home_page": "https://git.km3net.de/km3py/km3io",
"name": "km3io",
"maintainer": "Tamas Gal",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "tgal@km3net.de",
"keywords": "neutrino, astroparticle, physics, HEP, root",
"author": "\"Tamas Gal and Zineb Aly\"",
"author_email": "tgal@km3net.de",
"download_url": "https://files.pythonhosted.org/packages/4e/65/d9e8c384c8d1a9b997dc71ff7211fa046b3414a91204f83eaa2827ee14b0/km3io-1.2.3.tar.gz",
"platform": null,
"description": "The km3io Python package\n========================\n\n.. image:: https://git.km3net.de/km3py/km3io/badges/master/pipeline.svg\n :target: https://git.km3net.de/km3py/km3io/pipelines\n\n.. image:: https://git.km3net.de/km3py/km3io/badges/master/coverage.svg\n :target: https://km3py.pages.km3net.de/km3io/coverage\n\n.. image:: https://git.km3net.de/examples/km3badges/-/raw/master/docs-latest-brightgreen.svg\n :target: https://km3py.pages.km3net.de/km3io\n\n.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7382620.svg\n :target: https://doi.org/10.5281/zenodo.7382620\n\nThis software provides a set of Python classes to read KM3NeT ROOT files\nwithout having ROOT, Jpp or aanet installed. It only depends on Python 3.5+ and the amazing `uproot <https://github.com/scikit-hep/uproot>`__ package and gives you access to the data via `numpy <https://www.numpy.org>`__ and `awkward <https://awkward-array.readthedocs.io>`__ arrays.\n\nIt's very easy to use and according to the `uproot <https://github.com/scikit-hep/uproot>`__ benchmarks, it is able to outperform the original ROOT I/O performance. \n\nInstallation\n============\n\nInstall km3io using pip::\n\n pip install km3io \n\nor conda::\n\n conda install km3io\n\nTo get the latest (stable) development release::\n\n pip install git+https://git.km3net.de/km3py/km3io.git\n\nDocker::\n\n docker run -it docker.km3net.de/km3io\n\nSingularity::\n\n wget https://sftp.km3net.de/singularity/km3io_v0.27.2.sif # pick the version you like\n singularity shell km3io_v0.27.2.sif\n\n**Reminder:** km3io is **not** dependent on aanet, ROOT or Jpp!\n\nQuestions\n=========\n\nIf you have a question about km3io, please proceed as follows:\n\n- Read the documentation below.\n- Explore the `examples <https://km3py.pages.km3net.de/km3io/examples.html>`__ in the documentation.\n- Haven't you found an answer to your question in the documentation, post a git issue with your question showing us an example of what you have tried first, and what you would like to do.\n- Have you noticed a bug, please post it in a git issue, we appreciate your contribution.\n\n\nIntroduction\n------------\n\nMost of km3net data is stored in root files. These root files are created using\nthe `KM3NeT Dataformat library\n<https://git.km3net.de/common/km3net-dataformat>`__ A ROOT file created with\n`Jpp <https://git.km3net.de/common/jpp>`__ is an \"online\" file and all other\nsoftware usually produces \"offline\" files.\n\nkm3io is a Python package that provides access to offline files with its\n``OfflineReader`` class and a special one to read gSeaGen files. All of these\nROOT files can be read without installing any other software like Jpp, aanet or\nROOT. km3io v1.1 and earlier also support the access to online files (events,\nsummaryslices and timeslices). This feature has been dropped due to a lack of\nmainteinance power and inf favour of the `KM3io.jl <https://git.km3net.de/common/KM3io.jl>`__` Julia Package, which\nprovides high-performances access to all ROOT files and should also be\nprioritised over ``km3io`` when performance matters (which does, most of the\ntime).\n\nData in km3io is returned as ``awkward.Array`` which is an advance Numpy-like container type to store\ncontiguous data for high performance computations.\nSuch an ``awkward.Array`` supports any level of nested arrays and records which can have different lengths, in contrast to Numpy where everything has to be rectangular.\n\nThe example is shown below shows the array which contains the ``dir_z`` values\nof each track of the first 4 events. The type ``4 * var * float64`` means that\nit has 4 subarrays with variable lengths of type ``float64``:\n\n.. code-block:: python3\n\n >>> import km3io\n >>> from km3net_testdata import data_path\n >>> f = km3io.OfflineReader(data_path(\"offline/numucc.root\"))\n >>> f[:4].tracks.dir_z\n <Array [[0.213, 0.213, ... 0.229, 0.323]] type='4 * var * float64'>\n\nThe same concept applies to all other branches, including ``hits``, ``mc_hits``,\n``mc_tracks``, ``t_sec`` etc.\n\nArchitecture overview\n---------------------\n\n``km3io`` utilises ``uproot`` behind the scenes and creates a lazy and thin\nwrapper which offers convenient slicing and iterations by delaying the access to\nthe actual ROOT data branches to the very last moment. When using the iteration\nfunctionality, the data is loaded in chunks and the iteration is done over e.g.\nevents in each chunk or a bunch of frames in case of the summaryslice reader.\n\nThe base class for the event-based readout is the ``km3io.rootio.EventReader``\nclass. When subclassing this class, the branches, aliases and nested branches\nneed to be defined in the static variables which are then used to mask unwanted\nattributes. Especially in case of the Offline ROOT format, where the \"one class\nfits all\" design was chosen, it is distracting that e.g. a `Hit` has many\nattributes which make no sense depending on the context (MC hit, raw hit etc.).\nBy specifing the branches explicitely, the user API will only expose the\nmeaningful fields.\n\nThe online ROOT format support is partly still based on ``uproot3``.\n\nMany of the utility functions are using Numba to achieve the best possible\nperformance. ``km3io`` does not offer alternative implementations, so Numba is a\nstrict dependency and an integral part of the implementation.\n\n\nOffline files reader\n--------------------\n\nIn general an offline file has two attributes to access data: the header and the events. Let's start with the header.\n\nReading the file header\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nTo read an offline file start with opening it with the ``OfflineReader``:\n\n.. code-block:: python3\n\n >>> import km3io\n >>> from km3net_testdata import data_path\n >>> f = km3io.OfflineReader(data_path(\"offline/numucc.root\"))\n\nAccessing is as easy as typing:\n\n.. code-block:: python3\n\n >>> f.header\n <km3io.offline.Header at 0x7fcd81025990>\n\nPrinting it will give an overview of the structure:\n\n.. code-block:: python3\n\n >>> print(f.header)\n MC Header:\n DAQ(livetime=394)\n PDF(i1=4, i2=58)\n can(zmin=0, zmax=1027, r=888.4)\n can_user: can_user(field_0=0.0, field_1=1027.0, field_2=888.4)\n coord_origin(x=0, y=0, z=0)\n cut_in(Emin=0, Emax=0, cosTmin=0, cosTmax=0)\n cut_nu(Emin=100, Emax=100000000.0, cosTmin=-1, cosTmax=1)\n cut_primary(Emin=0, Emax=0, cosTmin=0, cosTmax=0)\n cut_seamuon(Emin=0, Emax=0, cosTmin=0, cosTmax=0)\n decay: decay(field_0='doesnt', field_1='happen')\n detector: NOT\n drawing: Volume\n genhencut(gDir=2000, Emin=0)\n genvol(zmin=0, zmax=1027, r=888.4, volume=2649000000.0, numberOfEvents=100000)\n kcut: 2\n livetime(numberOfSeconds=0, errorOfSeconds=0)\n model(interaction=1, muon=2, scattering=0, numberOfEnergyBins=1, field_4=12)\n ngen: 100000.0\n norma(primaryFlux=0, numberOfPrimaries=0)\n nuflux: nuflux(field_0=0, field_1=3, field_2=0, field_3=0.5, field_4=0.0, field_5=1.0, field_6=3.0)\n physics(program='GENHEN', version='7.2-220514', date=181116, time=1138)\n seed(program='GENHEN', level=3, iseed=305765867, field_3=0, field_4=0)\n simul(program='JSirene', version=11012, date='11/17/18', time=7)\n sourcemode: diffuse\n spectrum(alpha=-1.4)\n start_run(run_id=1)\n target: isoscalar\n usedetfile: false\n xlat_user: 0.63297\n xparam: OFF\n zed_user: zed_user(field_0=0.0, field_1=3450.0)\n\nTo read the values in the header one can call them directly, as the structures\nare simple ``namedtuple``-like objects:\n\n.. code-block:: python3\n\n >>> f.header.DAQ.livetime\n 394\n >>> f.header.cut_nu.Emin\n 100\n >>> f.header.genvol.numberOfEvents\n 100000\n\n\nReading offline events\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nEvents are at the top level of an offline file, so that each branch of an event\nis directly accessible at the ``OfflineReader`` instance. The ``.keys()`` method\ncan be used to list the available attributes. Notice that some of them are aliases\nfor backwards compatibility (like ``mc_tracks`` and ``mc_trks``). Another\nbackwards compatibility feature is the ``f.events`` attribute which is simply\nmapping everything to ``f``, so that ``f.events.mc_tracks`` is the same as\n``f.mc_tracks``.\n\n.. code-block:: python3\n\n >>> f\n OfflineReader (10 events)\n >>> f.keys()\n {'comment', 'det_id', 'flags', 'frame_index', 'hits', 'id', 'index',\n 'mc_hits', 'mc_id', 'mc_run_id', 'mc_t', 'mc_tracks', 'mc_trks',\n 'n_hits', 'n_mc_hits', 'n_mc_tracks', 'n_mc_trks', 'n_tracks',\n 'n_trks', 'overlays', 'run_id', 't_ns', 't_sec', 'tracks',\n 'trigger_counter', 'trigger_mask', 'trks', 'usr', 'usr_names',\n 'w', 'w2list', 'w3list'}\n >>> f.tracks\n <Branch [10] path='trks'>\n >>> f.events.tracks\n <Branch [10] path='trks'>\n\nThe ``[10]`` denotes that there are ``10`` events available, each containing a sub-array of ``tracks``.\n\nUsing <TAB> completion gives an overview of available data. Alternatively the attribute `fields`\ncan be used on event-branches and to see what is available for reading.\n\n.. code-block:: python3\n\n >>> f.tracks.fields\n ['id',\n 'pos_x',\n 'pos_y',\n 'pos_z',\n 'dir_x',\n 'dir_y',\n 'dir_z',\n 't',\n 'E',\n 'len',\n 'lik',\n 'rec_type',\n 'rec_stages',\n 'fitinf']\n\n\nReading the reconstructed values like energy and direction of an event can be done with:\n\n.. code-block:: python3\n\n >>> f.events.tracks.E\n <Array [[117, 117, 0, 0, 0, ... 0, 0, 0, 0, 0]] type='10 * var * float64'>\n\nThe ``Array`` in this case is an `awkward <https://awkward-array.readthedocs.io>`__ array with the data type\n``10 * var * float64`` which means that there are ``10`` sub-arrays with ``var``iable lengths of type ``float64``.\nAwkward arrays allow high-performance access to arrays which are not rectangular (in contrast to ``numpy``).\nRead the documention of AwkwardArray to learn how to work with these structures efficiently. One example\nto retrieve the energy of the very first reconstructed track for the first three events is:\n\n.. code-block:: python3\n\n >>> f.events.tracks.E[:3, 0]\n <Array [117, 4.4e+03, 8.37] type='3 * float64'>\n\nOnline files reader\n-------------------\n\nThe support to read online ROOT files has been dropped in ``km3io`` v1.2.\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "\"KM3NeT I/O library without ROOT\"",
"version": "1.2.3",
"project_urls": {
"Homepage": "https://git.km3net.de/km3py/km3io"
},
"split_keywords": [
"neutrino",
" astroparticle",
" physics",
" hep",
" root"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b4597b7567482c66f9e4e6b7e6ee4bc7cb0e5bce7b68ae27aa329d2c891ad0b5",
"md5": "fe7990c5eeba1609dd6fb0d3c286ca37",
"sha256": "b8aaeaa56c6186b63932db803764fd7102cc868131aa933d6d89b09c08f6ac0e"
},
"downloads": -1,
"filename": "km3io-1.2.3-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "fe7990c5eeba1609dd6fb0d3c286ca37",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.9",
"size": 29413,
"upload_time": "2024-10-04T20:15:27",
"upload_time_iso_8601": "2024-10-04T20:15:27.333576Z",
"url": "https://files.pythonhosted.org/packages/b4/59/7b7567482c66f9e4e6b7e6ee4bc7cb0e5bce7b68ae27aa329d2c891ad0b5/km3io-1.2.3-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4e65d9e8c384c8d1a9b997dc71ff7211fa046b3414a91204f83eaa2827ee14b0",
"md5": "efddde7b678a82140c84ffa40cbb4e59",
"sha256": "ffbeebe0a16c061f1784a45568396ad4627813b895587e090bf7223a4a9bdd78"
},
"downloads": -1,
"filename": "km3io-1.2.3.tar.gz",
"has_sig": false,
"md5_digest": "efddde7b678a82140c84ffa40cbb4e59",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 297889,
"upload_time": "2024-10-04T20:15:29",
"upload_time_iso_8601": "2024-10-04T20:15:29.982359Z",
"url": "https://files.pythonhosted.org/packages/4e/65/d9e8c384c8d1a9b997dc71ff7211fa046b3414a91204f83eaa2827ee14b0/km3io-1.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-04 20:15:29",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "km3io"
}
JSON