KM3NeT database library
=======================
.. image:: https://git.km3net.de/km3py/km3db/badges/master/pipeline.svg
:target: https://git.km3net.de/km3py/km3db/pipelines
.. image:: https://git.km3net.de/km3py/km3db/badges/master/coverage.svg
:target: https://km3py.pages.km3net.de/km3db/coverage
.. image:: https://git.km3net.de/examples/km3badges/-/raw/master/docs-latest-brightgreen.svg
:target: https://km3py.pages.km3net.de/km3db
``km3db`` is a lightweight library to access the web API of the KM3NeT Oracle
database (https://km3netdbweb.in2p3.fr). It requires Python 2.7 or later and
comes with a small set of command line utilities which can be used in
shell scripts.
Installation
------------
Tagged releases are available on the Python Package Index repository (https://pypi.org)
and can easily be installed with the ``pip`` command::
pip install km3db
Database Access with Cookies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The KM3NeT DB credentials should be used to obtain a session cookie which is
then passed to the database web API for all future requests. The cookie file
can be stored as a file and the default location is ``~/.km3netdb_cookie``.
If you want to use a different file, set the ``KM3NET_DB_COOKIE_FILE`` to the
desired file and each request made with ``km3db`` will use that cookie. You
can also specify ``KM3NET_DB_COOKIE`` as a string value, which then will be used
as a session ID cookie. This is useful if you work in environments where you can't
or don't want to store files.
See the section below how to use the ``km3dbcookie`` command line tool explicity
to obtain a session cookie from the command line. Notice that cookies are
automatically handled behind the scenes if you use any of the ``km3db``
functionality.
Python Classes
--------------
The three important classes are ``DBManager``, ``StreamDS`` and ``CLBMap``.
``DBManager``
~~~~~~~~~~~~~
The ``DBManager`` class manages the authentication and cookie management and
low level access to the database::
>>> import km3db
>>> db = km3db.DBManager()
It tries to figure out the easiest way to authenticate with the database gateway.
If launched on the Lyon CC, GitLab CI or the KM3NeT JupyterHub service, it will
automatically use the corresponding session cookies.
If not operating on whitelisted hosts, the environment variables ``KM3NET_DB_USERNAME``
and ``KM3NET_DB_PASSWORD`` will be used. If those are not set, it will look for a
cookie in ``~/.km3netdb_cookie``. As a last resort, it will prompt the user to
enter the username and password manually.
After a successful authentication, a cookie file with the session cookie will be
stored in the above mentioned file for future authentications.
``StreamDS``
~~~~~~~~~~~~
The ``StreamDS`` class is specifically designed to access the Stream Data Service
entrypoint of the database, which is meant to provide large datasets, potentially
exceeding multiples of GB::
>>> import km3db
>>> sds = km3db.StreamDS()
>>> print(sds.detectors())
OID SERIALNUMBER LOCATIONID CITY FIRSTRUN LASTRUN
D_DU1CPPM 2 A00070004 Marseille 2 10
A00350276 3 A00070003 Napoli 0 0
...
...
D1DU039CT 59 A02181273 Catania 408 480
D0DU040CE 60 A01288502 Caserta 0 0
>>> print(sds.get("detectors")) # alternative way to call it
...
In km3pipe v8 and below, the `StreamDS` class always returned `pandas.DataFrames`
by default. This has been changed in `km3db` and by default, only the raw ASCII
output is returned, as delivered by the database.
One can however change the output container type back to `pandas.DataFrame` by
passing `container="pd"` to either the `StreamDS()` constructor or to the
`.get()` function itself. Another supported container type is `namedtuple` from
the Python standard library (`collections.namedtuple`), available via
`container="nt"`::
>>> sds = km3db.StreamDS(container="pd")
>>> type(sds.detectors())
pandas.core.frame.DataFrame
# pandas DataFrame only on a specific call
>>> sds = km3db.StreamDS()
>>> type(sds.get("detectors", container="pd"))
pandas.core.frame.DataFrame
# namedtuple
>>> sds.get("detectors", container="nt")[0]
Detectors(oid='D_DU1CPPM', serialnumber=2, locationid='A00070004', city='Marseille', firstrun=2, lastrun=10)
``CLBMap``
~~~~~~~~~~
The ``CLBMap`` is a powerful helper class which makes it easy to query detector
configurations and CLB::
>>> import km3db
>>> clbmap = km3db.CLBMap("D_ORCA003")
>>> clb = clbmap.omkeys[(1, 13)]
>>> clb
Clbmap(det_oid='D_ORCA003', du=1, floor=13, serial_number=374, upi='3.4.3.2/V2-2-1/2.374', dom_id=808949902)
>>> clb.dom_id
808949902
>>> clb.upi
'3.4.3.2/V2-2-1/2.374'
Command Line Utilities
----------------------
The following command line utilities will be accessible after installing ``km3db``.
``km3dbcookie``
~~~~~~~~~~~~~~~~~~
The ``km3netdbcookie`` command can be used to obtain a session cookie using the
KM3NeT DB credentials::
$ km3dbcookie -h
Generate a cookie for the KM3NeT Oracle Web API.
Usage:
km3dbcookie [-B | -C]
km3dbcookie (-h | --help)
km3dbcookie --version
Options:
-B Request the cookie for a class B network (12.23.X.Y).
-C Request the cookie for a class C network (12.23.45.Y).
-h --help Show this screen.
Example:
$ km3dbcookie -B
Please enter your KM3NeT DB username: tgal
Password:
Cookie saved as '/Users/tamasgal/.km3netdb_cookie'
$ cat /Users/tamasgal/.km3netdb_cookie
.in2p3.fr TRUE / TRUE 0 sid _tgal_131.188_70b78042c03a434594b041073484ce23
``detx``
~~~~~~~~~~~~
The ``detx`` command can be used to retrieve calibration information from the
database formatted as DETX, which is its main offline representation format::
$ detx -h
Retrieves DETX files from the database.
Usage:
detx [options] DET_ID
detx DET_ID RUN
detx (-h | --help)
detx --version
Options:
DET_ID The detector ID (e.g. 49)
RUN The run ID.
-c CALIBR_ID Geometrical calibration ID (eg. A01466417)
-t T0_SET Time calibration ID (eg. A01466431)
-o OUT Output folder or filename.
-h --help Show this screen.
Example:
detx 49 8220 # retrieve the calibrated DETX for run 8220 of ORCA6
``streamds``
~~~~~~~~~~~~
The ``streamds`` command provides access to the "Stream Data Service" which was
designed to deal with large datasets potentially exceeding multiple GB in size.
The help output explains all the available functionality of the tool::
$ streamds -h
Access the KM3NeT StreamDS DataBase service.
Usage:
streamds
streamds list
streamds info STREAM
streamds get [-f FORMAT -o OUTFILE -g GROUPBY] STREAM [PARAMETERS...]
streamds upload [-q -x] CSV_FILE
streamds (-h | --help)
streamds --version
Options:
STREAM Name of the stream.
PARAMETERS List of parameters separated by space (e.g. detid=29).
CSV_FILE Whitespace separated data for the runsummary tables.
-f FORMAT Usually 'txt' for ASCII or 'text' for UTF-8 [default: txt].
-o OUTFILE Output file: supported formats '.csv' and '.h5'.
-g COLUMN Group dataset by the name of the given row when writing HDF5.
-q Test run! When uploading, a TEST_ prefix will be added to the data.
-x Do not verify the SSL certificate.
-h --help Show this screen.
For example, a list of available detectors::
> streamds get detectors
OID SERIALNUMBER LOCATIONID CITY FIRSTRUN LASTRUN
D_DU1CPPM 2 A00070004 Marseille 2 10
A00350276 3 A00070003 Napoli 0 0
D_DU2NAPO 5 A00070003 Napoli 98 428
D_TESTDET 6 A00070002 Fisciano 3 35
D_ARCA001 7 A00073795 Italy 1 2763
FR_INFRAS 8 A00073796 France 1600 3202
D_DU003NA 9 A00070003 Napoli 1 242
D_DU004NA 12 A00070003 Napoli 243 342
D_DU001MA 13 A00070004 Marseille 1 1922
D_ARCA003 14 A00073795 Italy 1 6465
To write the database output to a file, use the ``-o`` option, e.g.
``streamds get detectors -o detectors.csv``. The currently supported
filetypes are ``.csv`` and ``.h5``. In case of ``.h5``, the data can
be grouped by providing ``-g COLUMN``, which will split up the
output and write distinct HDF5 dataset. It's useful to group large
datasets by e.g. ``RUN``, however, only numerical datatypes are supported
currently::
> streamds get toashort detid=D0ORCA010 minrun=13000 maxrun=13005 -g RUN -o KM3NeT_00000100_toashort.h5
Database output written to 'KM3NeT_00000100_toashort.h5'.
``km3db``
~~~~~~~~~
The ``km3db`` command gives direct access to database URLs and is mainly a
debugging tool::
$ km3db -h
Command line access to the KM3NeT DB web API.
Usage:
km3db URL
km3db (-h | --help)
km3db --version
Options:
URL The URL, starting from the database website's root.
-h --help Show this screen.
Example:
km3db "streamds/runs.txt?detid=D_ARCA003"
The URL parameter is simply the string which comes right after
``https://km3netdbweb.in2p3.fr/``.
Raw data
{
"_id": null,
"home_page": "https://git.km3net.de/km3py/km3db",
"name": "km3db",
"maintainer": null,
"docs_url": null,
"requires_python": ">=2.7",
"maintainer_email": null,
"keywords": null,
"author": "Tamas Gal",
"author_email": "tgal@km3net.de",
"download_url": "https://files.pythonhosted.org/packages/51/cd/ad9715104b6b0c833e481d3abb035d411efa769fdb40c5aa40e44c92a194/km3db-0.14.3.tar.gz",
"platform": "any",
"description": "KM3NeT database library\n=======================\n\n.. image:: https://git.km3net.de/km3py/km3db/badges/master/pipeline.svg\n :target: https://git.km3net.de/km3py/km3db/pipelines\n\n.. image:: https://git.km3net.de/km3py/km3db/badges/master/coverage.svg\n :target: https://km3py.pages.km3net.de/km3db/coverage\n\n.. image:: https://git.km3net.de/examples/km3badges/-/raw/master/docs-latest-brightgreen.svg\n :target: https://km3py.pages.km3net.de/km3db\n\n\n``km3db`` is a lightweight library to access the web API of the KM3NeT Oracle\ndatabase (https://km3netdbweb.in2p3.fr). It requires Python 2.7 or later and\ncomes with a small set of command line utilities which can be used in\nshell scripts.\n\nInstallation\n------------\n\nTagged releases are available on the Python Package Index repository (https://pypi.org)\nand can easily be installed with the ``pip`` command::\n\n pip install km3db\n\nDatabase Access with Cookies\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe KM3NeT DB credentials should be used to obtain a session cookie which is\nthen passed to the database web API for all future requests. The cookie file\ncan be stored as a file and the default location is ``~/.km3netdb_cookie``.\nIf you want to use a different file, set the ``KM3NET_DB_COOKIE_FILE`` to the\ndesired file and each request made with ``km3db`` will use that cookie. You\ncan also specify ``KM3NET_DB_COOKIE`` as a string value, which then will be used\nas a session ID cookie. This is useful if you work in environments where you can't\nor don't want to store files.\n\nSee the section below how to use the ``km3dbcookie`` command line tool explicity\nto obtain a session cookie from the command line. Notice that cookies are\nautomatically handled behind the scenes if you use any of the ``km3db``\nfunctionality.\n\nPython Classes\n--------------\n\nThe three important classes are ``DBManager``, ``StreamDS`` and ``CLBMap``.\n\n``DBManager``\n~~~~~~~~~~~~~\nThe ``DBManager`` class manages the authentication and cookie management and\nlow level access to the database::\n\n >>> import km3db\n >>> db = km3db.DBManager()\n\nIt tries to figure out the easiest way to authenticate with the database gateway.\nIf launched on the Lyon CC, GitLab CI or the KM3NeT JupyterHub service, it will\nautomatically use the corresponding session cookies.\nIf not operating on whitelisted hosts, the environment variables ``KM3NET_DB_USERNAME``\nand ``KM3NET_DB_PASSWORD`` will be used. If those are not set, it will look for a\ncookie in ``~/.km3netdb_cookie``. As a last resort, it will prompt the user to\nenter the username and password manually.\nAfter a successful authentication, a cookie file with the session cookie will be\nstored in the above mentioned file for future authentications.\n\n``StreamDS``\n~~~~~~~~~~~~\nThe ``StreamDS`` class is specifically designed to access the Stream Data Service\nentrypoint of the database, which is meant to provide large datasets, potentially\nexceeding multiples of GB::\n\n >>> import km3db\n >>> sds = km3db.StreamDS()\n >>> print(sds.detectors())\n OID\tSERIALNUMBER\tLOCATIONID\tCITY\tFIRSTRUN\tLASTRUN\n D_DU1CPPM\t2\tA00070004\tMarseille\t2\t10\n A00350276\t3\tA00070003\tNapoli\t0\t0\n ...\n ...\n D1DU039CT\t59\tA02181273\tCatania\t408\t480\n D0DU040CE\t60\tA01288502\tCaserta\t0\t0\n >>> print(sds.get(\"detectors\")) # alternative way to call it\n ...\n\nIn km3pipe v8 and below, the `StreamDS` class always returned `pandas.DataFrames`\nby default. This has been changed in `km3db` and by default, only the raw ASCII\noutput is returned, as delivered by the database.\n\nOne can however change the output container type back to `pandas.DataFrame` by\npassing `container=\"pd\"` to either the `StreamDS()` constructor or to the\n`.get()` function itself. Another supported container type is `namedtuple` from\nthe Python standard library (`collections.namedtuple`), available via\n`container=\"nt\"`::\n\n >>> sds = km3db.StreamDS(container=\"pd\")\n >>> type(sds.detectors())\n pandas.core.frame.DataFrame\n\n # pandas DataFrame only on a specific call\n >>> sds = km3db.StreamDS()\n >>> type(sds.get(\"detectors\", container=\"pd\"))\n pandas.core.frame.DataFrame\n\n # namedtuple\n >>> sds.get(\"detectors\", container=\"nt\")[0]\n Detectors(oid='D_DU1CPPM', serialnumber=2, locationid='A00070004', city='Marseille', firstrun=2, lastrun=10)\n\n``CLBMap``\n~~~~~~~~~~\nThe ``CLBMap`` is a powerful helper class which makes it easy to query detector\nconfigurations and CLB::\n\n >>> import km3db\n >>> clbmap = km3db.CLBMap(\"D_ORCA003\")\n >>> clb = clbmap.omkeys[(1, 13)]\n >>> clb\n Clbmap(det_oid='D_ORCA003', du=1, floor=13, serial_number=374, upi='3.4.3.2/V2-2-1/2.374', dom_id=808949902)\n >>> clb.dom_id\n 808949902\n >>> clb.upi\n '3.4.3.2/V2-2-1/2.374'\n\nCommand Line Utilities\n----------------------\n\nThe following command line utilities will be accessible after installing ``km3db``.\n\n``km3dbcookie``\n~~~~~~~~~~~~~~~~~~\n\nThe ``km3netdbcookie`` command can be used to obtain a session cookie using the\nKM3NeT DB credentials::\n\n $ km3dbcookie -h\n Generate a cookie for the KM3NeT Oracle Web API.\n\n Usage:\n km3dbcookie [-B | -C]\n km3dbcookie (-h | --help)\n km3dbcookie --version\n\n Options:\n -B Request the cookie for a class B network (12.23.X.Y).\n -C Request the cookie for a class C network (12.23.45.Y).\n -h --help Show this screen.\n\n Example:\n\n $ km3dbcookie -B\n Please enter your KM3NeT DB username: tgal\n Password:\n Cookie saved as '/Users/tamasgal/.km3netdb_cookie'\n $ cat /Users/tamasgal/.km3netdb_cookie\n .in2p3.fr\tTRUE\t/\tTRUE\t0\tsid\t_tgal_131.188_70b78042c03a434594b041073484ce23\n\n``detx``\n~~~~~~~~~~~~\n\nThe ``detx`` command can be used to retrieve calibration information from the\ndatabase formatted as DETX, which is its main offline representation format::\n\n $ detx -h\n Retrieves DETX files from the database.\n\n Usage:\n detx [options] DET_ID\n detx DET_ID RUN\n detx (-h | --help)\n detx --version\n\n Options:\n DET_ID The detector ID (e.g. 49)\n RUN The run ID.\n -c CALIBR_ID Geometrical calibration ID (eg. A01466417)\n -t T0_SET Time calibration ID (eg. A01466431)\n -o OUT Output folder or filename.\n -h --help Show this screen.\n\n Example:\n\n detx 49 8220 # retrieve the calibrated DETX for run 8220 of ORCA6\n\n``streamds``\n~~~~~~~~~~~~\n\nThe ``streamds`` command provides access to the \"Stream Data Service\" which was\ndesigned to deal with large datasets potentially exceeding multiple GB in size.\nThe help output explains all the available functionality of the tool::\n\n $ streamds -h\n Access the KM3NeT StreamDS DataBase service.\n\n Usage:\n streamds\n streamds list\n streamds info STREAM\n streamds get [-f FORMAT -o OUTFILE -g GROUPBY] STREAM [PARAMETERS...]\n streamds upload [-q -x] CSV_FILE\n streamds (-h | --help)\n streamds --version\n\n Options:\n STREAM Name of the stream.\n PARAMETERS List of parameters separated by space (e.g. detid=29).\n CSV_FILE Whitespace separated data for the runsummary tables.\n -f FORMAT Usually 'txt' for ASCII or 'text' for UTF-8 [default: txt].\n -o OUTFILE Output file: supported formats '.csv' and '.h5'.\n -g COLUMN Group dataset by the name of the given row when writing HDF5.\n -q Test run! When uploading, a TEST_ prefix will be added to the data.\n -x Do not verify the SSL certificate.\n -h --help Show this screen.\n\nFor example, a list of available detectors::\n\n > streamds get detectors\n OID\tSERIALNUMBER\tLOCATIONID\tCITY\tFIRSTRUN\tLASTRUN\n D_DU1CPPM\t2\tA00070004\tMarseille\t2\t10\n A00350276\t3\tA00070003\tNapoli\t0\t0\n D_DU2NAPO\t5\tA00070003\tNapoli\t98\t428\n D_TESTDET\t6\tA00070002\tFisciano\t3\t35\n D_ARCA001\t7\tA00073795\tItaly\t1\t2763\n FR_INFRAS\t8\tA00073796\tFrance\t1600\t3202\n D_DU003NA\t9\tA00070003\tNapoli\t1\t242\n D_DU004NA\t12\tA00070003\tNapoli\t243\t342\n D_DU001MA\t13\tA00070004\tMarseille\t1\t1922\n D_ARCA003\t14\tA00073795\tItaly\t1\t6465\n\nTo write the database output to a file, use the ``-o`` option, e.g.\n``streamds get detectors -o detectors.csv``. The currently supported\nfiletypes are ``.csv`` and ``.h5``. In case of ``.h5``, the data can\nbe grouped by providing ``-g COLUMN``, which will split up the\noutput and write distinct HDF5 dataset. It's useful to group large\ndatasets by e.g. ``RUN``, however, only numerical datatypes are supported\ncurrently::\n\n > streamds get toashort detid=D0ORCA010 minrun=13000 maxrun=13005 -g RUN -o KM3NeT_00000100_toashort.h5\n Database output written to 'KM3NeT_00000100_toashort.h5'.\n\n\n``km3db``\n~~~~~~~~~\n\nThe ``km3db`` command gives direct access to database URLs and is mainly a\ndebugging tool::\n\n $ km3db -h\n Command line access to the KM3NeT DB web API.\n\n Usage:\n km3db URL\n km3db (-h | --help)\n km3db --version\n\n Options:\n URL The URL, starting from the database website's root.\n -h --help Show this screen.\n\n Example:\n\n km3db \"streamds/runs.txt?detid=D_ARCA003\"\n\nThe URL parameter is simply the string which comes right after\n``https://km3netdbweb.in2p3.fr/``.\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "KM3NeT database library",
"version": "0.14.3",
"project_urls": {
"Homepage": "https://git.km3net.de/km3py/km3db"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8bea2fd941d43cd5353eb30c205db3ad91aa13c8cce7f9bc123e15de0eb6d74e",
"md5": "0b6cb0b69770402cca326eedef70e571",
"sha256": "d3b044e232ec8726876cc42d2194be2b166e1c553171d1f7f2ad981d2506c1ac"
},
"downloads": -1,
"filename": "km3db-0.14.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0b6cb0b69770402cca326eedef70e571",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=2.7",
"size": 25993,
"upload_time": "2024-11-10T21:56:46",
"upload_time_iso_8601": "2024-11-10T21:56:46.626738Z",
"url": "https://files.pythonhosted.org/packages/8b/ea/2fd941d43cd5353eb30c205db3ad91aa13c8cce7f9bc123e15de0eb6d74e/km3db-0.14.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "51cdad9715104b6b0c833e481d3abb035d411efa769fdb40c5aa40e44c92a194",
"md5": "0ccd9f683a1a96c910974d1d998df963",
"sha256": "249400901095107fe08118db38b02626b9a46f351dda2a3cb3eaec82b8560322"
},
"downloads": -1,
"filename": "km3db-0.14.3.tar.gz",
"has_sig": false,
"md5_digest": "0ccd9f683a1a96c910974d1d998df963",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.7",
"size": 70106,
"upload_time": "2024-11-10T21:56:48",
"upload_time_iso_8601": "2024-11-10T21:56:48.822117Z",
"url": "https://files.pythonhosted.org/packages/51/cd/ad9715104b6b0c833e481d3abb035d411efa769fdb40c5aa40e44c92a194/km3db-0.14.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-10 21:56:48",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "km3db"
}
JSON