km3db


Namekm3db JSON
Version 0.5.2 PyPI version JSON
download
home_pagehttps://git.km3net.de/km3py/km3db
SummaryKM3NeT database library
upload_time2021-02-23 17:54:56
maintainer
docs_urlNone
authorTamas Gal
requires_python>=2.7
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            KM3NeT database library
=======================

.. image:: https://git.km3net.de/km3py/km3db/badges/master/pipeline.svg
    :target: https://git.km3net.de/km3py/km3db/pipelines

.. image:: https://git.km3net.de/km3py/km3db/badges/master/coverage.svg
    :target: https://km3py.pages.km3net.de/km3db/coverage

.. image:: https://examples.pages.km3net.de/km3badges/docs-latest-brightgreen.svg
    :target: https://km3py.pages.km3net.de/km3db


``km3db`` is a lightweight library to access the web API of the KM3NeT Oracle
database (https://km3netdbweb.in2p3.fr). It requires Python 2.7 or later and
comes with a small set of command line utilities which can be used in
shell scripts.

Installation
------------

Tagged releases are available on the Python Package Index repository (https://pypi.org)
and can easily be installed with the ``pip`` command::

  pip install km3db

Python Classes
--------------

The three important classes are ``DBManager``, ``StreamDS`` and ``CLBMap``.

``DBManager``
~~~~~~~~~~~~~
The ``DBManager`` class manages the authentication and cookie management and
low level access to the database::

  >>> import km3db
  >>> db = km3db.DBManager()

It tries to figure out the easiest way to authenticate with the database gateway.
If launched on the Lyon CC, GitLab CI or the KM3NeT JupyterHub service, it will
automatically use the corresponding session cookies.
If not operating on whitelisted hosts, the environment variables ``KM3NET_DB_USERNAME``
and ``KM3NET_DB_PASSWORD`` will be used. If those are not set, it will look for a
cookie in ``~/.km3netdb_cookie``. As a last resort, it will prompt the user to
enter the username and password manually.
After a successful authentication, a cookie file with the session cookie will be
stored in the above mentioned file for future authentications.

``StreamDS``
~~~~~~~~~~~~
The ``StreamDS`` class is specifically designed to access the Stream Data Service
entrypoint of the database, which is meant to provide large datasets, potentially
exceeding multiples of GB::

  >>> import km3db
  >>> sds = km3db.StreamDS()
  >>> print(sds.detectors())
  OID	SERIALNUMBER	LOCATIONID	CITY	FIRSTRUN	LASTRUN
  D_DU1CPPM	2	A00070004	Marseille	2	10
  A00350276	3	A00070003	Napoli	0	0
  ...
  ...
  D1DU039CT	59	A02181273	Catania	408	480
  D0DU040CE	60	A01288502	Caserta	0	0
  >>> print(sds.get("detectors"))  # alternative way to call it
  ...

In km3pipe v8 and below, the `StreamDS` class always returned `pandas.DataFrames`
by default. This has been changed in `km3db` and by default, only the raw ASCII
output is returned, as delivered by the database.

One can however change the output container type back to `pandas.DataFrame` by
passing `container="pd"` to either the `StreamDS()` constructor or to the
`.get()` function itself. Another supported container type is `namedtuple` from
the Python standard library (`collections.namedtuple`), available via
`container="nt"`::

   >>> sds = km3db.StreamDS(container="pd")
   >>> type(sds.detectors())
   pandas.core.frame.DataFrame

   # pandas DataFrame only on a specific call
   >>> sds = km3db.StreamDS()
   >>> type(sds.get("detectors", container="pd"))
   pandas.core.frame.DataFrame

   # namedtuple
   >>> sds.get("detectors", container="nt")[0]
   Detectors(oid='D_DU1CPPM', serialnumber=2, locationid='A00070004', city='Marseille', firstrun=2, lastrun=10)

``CLBMap``
~~~~~~~~~~
The ``CLBMap`` is a powerful helper class which makes it easy to query detector
configurations and CLB::

  >>> import km3db
  >>> clbmap = km3db.CLBMap("D_ORCA003")
  >>> clb = clbmap.omkeys[(1, 13)]
  >>> clb
  Clbmap(det_oid='D_ORCA003', du=1, floor=13, serial_number=374, upi='3.4.3.2/V2-2-1/2.374', dom_id=808949902)
  >>> clb.dom_id
  808949902
  >>> clb.upi
  '3.4.3.2/V2-2-1/2.374'

Command Line Utilities
----------------------

The following command line utilities will be accessible after installing ``km3db``.

``detx``
~~~~~~~~~~~~

The ``detx`` command can be used to retrieve calibration information from the
database formatted as DETX, which is its main offline representation format::

  $ detx -h
  Retrieves DETX files from the database.

  Usage:
      detx [options] DET_ID
      detx DET_ID RUN
      detx (-h | --help)
      detx --version

  Options:
      DET_ID        The detector ID (e.g. 49)
      RUN           The run ID.
      -c CALIBR_ID  Geometrical calibration ID (eg. A01466417)
      -t T0_SET     Time calibration ID (eg. A01466431)
      -o OUT        Output folder or filename.
      -h --help     Show this screen.

  Example:

      detx 49 8220  # retrieve the calibrated DETX for run 8220 of ORCA6

``streamds``
~~~~~~~~~~~~

The ``streamds`` command provides access to the "Stream Data Service" which was
designed to deal with large datasets potentially exceeding multiple GB in size.
The help output explains all the available functionality of the tool::

  $ streamds -h
  Access the KM3NeT StreamDS DataBase service.

  Usage:
      streamds
      streamds list
      streamds info STREAM
      streamds get [-f FORMAT] STREAM [PARAMETERS...]
      streamds (-h | --help)
      streamds --version

  Options:
      STREAM      Name of the stream.
      PARAMETERS  List of parameters separated by space (e.g. detid=29).
      -f FORMAT   Usually 'txt' for ASCII or 'text' for UTF-8 [default: txt].
      -x          Do not verify the SSL certificate.
      -h --help   Show this screen.

For example, a list of available detectors::

  > streamds get detectors
  OID	SERIALNUMBER	LOCATIONID	CITY	FIRSTRUN	LASTRUN
  D_DU1CPPM	2	A00070004	Marseille	2	10
  A00350276	3	A00070003	Napoli	0	0
  D_DU2NAPO	5	A00070003	Napoli	98	428
  D_TESTDET	6	A00070002	Fisciano	3	35
  D_ARCA001	7	A00073795	Italy	1	2763
  FR_INFRAS	8	A00073796	France	1600	3202
  D_DU003NA	9	A00070003	Napoli	1	242
  D_DU004NA	12	A00070003	Napoli	243	342
  D_DU001MA	13	A00070004	Marseille	1	1922
  D_ARCA003	14	A00073795	Italy	1	6465


``km3db``
~~~~~~~~~

The ``km3db`` command gives direct access to database URLs and is mainly a
debugging tool::

  $ km3db -h
  Command line access to the KM3NeT DB web API.

  Usage:
      km3db URL
      km3db (-h | --help)
      km3db --version

  Options:
      URL         The URL, starting from the database website's root.
      -h --help   Show this screen.

  Example:

      km3db "streamds/runs.txt?detid=D_ARCA003"

The URL parameter is simply the string which comes right after
``https://km3netdbweb.in2p3.fr/``.



            

Raw data

            {
    "_id": null,
    "home_page": "https://git.km3net.de/km3py/km3db",
    "name": "km3db",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=2.7",
    "maintainer_email": "",
    "keywords": "",
    "author": "Tamas Gal",
    "author_email": "tgal@km3net.de",
    "download_url": "https://files.pythonhosted.org/packages/18/84/d772edd06d90d21a86d6c17c7b6fd60807192de11f57d71ff8f7b7e84430/km3db-0.5.2.tar.gz",
    "platform": "any",
    "description": "KM3NeT database library\n=======================\n\n.. image:: https://git.km3net.de/km3py/km3db/badges/master/pipeline.svg\n    :target: https://git.km3net.de/km3py/km3db/pipelines\n\n.. image:: https://git.km3net.de/km3py/km3db/badges/master/coverage.svg\n    :target: https://km3py.pages.km3net.de/km3db/coverage\n\n.. image:: https://examples.pages.km3net.de/km3badges/docs-latest-brightgreen.svg\n    :target: https://km3py.pages.km3net.de/km3db\n\n\n``km3db`` is a lightweight library to access the web API of the KM3NeT Oracle\ndatabase (https://km3netdbweb.in2p3.fr). It requires Python 2.7 or later and\ncomes with a small set of command line utilities which can be used in\nshell scripts.\n\nInstallation\n------------\n\nTagged releases are available on the Python Package Index repository (https://pypi.org)\nand can easily be installed with the ``pip`` command::\n\n  pip install km3db\n\nPython Classes\n--------------\n\nThe three important classes are ``DBManager``, ``StreamDS`` and ``CLBMap``.\n\n``DBManager``\n~~~~~~~~~~~~~\nThe ``DBManager`` class manages the authentication and cookie management and\nlow level access to the database::\n\n  >>> import km3db\n  >>> db = km3db.DBManager()\n\nIt tries to figure out the easiest way to authenticate with the database gateway.\nIf launched on the Lyon CC, GitLab CI or the KM3NeT JupyterHub service, it will\nautomatically use the corresponding session cookies.\nIf not operating on whitelisted hosts, the environment variables ``KM3NET_DB_USERNAME``\nand ``KM3NET_DB_PASSWORD`` will be used. If those are not set, it will look for a\ncookie in ``~/.km3netdb_cookie``. As a last resort, it will prompt the user to\nenter the username and password manually.\nAfter a successful authentication, a cookie file with the session cookie will be\nstored in the above mentioned file for future authentications.\n\n``StreamDS``\n~~~~~~~~~~~~\nThe ``StreamDS`` class is specifically designed to access the Stream Data Service\nentrypoint of the database, which is meant to provide large datasets, potentially\nexceeding multiples of GB::\n\n  >>> import km3db\n  >>> sds = km3db.StreamDS()\n  >>> print(sds.detectors())\n  OID\tSERIALNUMBER\tLOCATIONID\tCITY\tFIRSTRUN\tLASTRUN\n  D_DU1CPPM\t2\tA00070004\tMarseille\t2\t10\n  A00350276\t3\tA00070003\tNapoli\t0\t0\n  ...\n  ...\n  D1DU039CT\t59\tA02181273\tCatania\t408\t480\n  D0DU040CE\t60\tA01288502\tCaserta\t0\t0\n  >>> print(sds.get(\"detectors\"))  # alternative way to call it\n  ...\n\nIn km3pipe v8 and below, the `StreamDS` class always returned `pandas.DataFrames`\nby default. This has been changed in `km3db` and by default, only the raw ASCII\noutput is returned, as delivered by the database.\n\nOne can however change the output container type back to `pandas.DataFrame` by\npassing `container=\"pd\"` to either the `StreamDS()` constructor or to the\n`.get()` function itself. Another supported container type is `namedtuple` from\nthe Python standard library (`collections.namedtuple`), available via\n`container=\"nt\"`::\n\n   >>> sds = km3db.StreamDS(container=\"pd\")\n   >>> type(sds.detectors())\n   pandas.core.frame.DataFrame\n\n   # pandas DataFrame only on a specific call\n   >>> sds = km3db.StreamDS()\n   >>> type(sds.get(\"detectors\", container=\"pd\"))\n   pandas.core.frame.DataFrame\n\n   # namedtuple\n   >>> sds.get(\"detectors\", container=\"nt\")[0]\n   Detectors(oid='D_DU1CPPM', serialnumber=2, locationid='A00070004', city='Marseille', firstrun=2, lastrun=10)\n\n``CLBMap``\n~~~~~~~~~~\nThe ``CLBMap`` is a powerful helper class which makes it easy to query detector\nconfigurations and CLB::\n\n  >>> import km3db\n  >>> clbmap = km3db.CLBMap(\"D_ORCA003\")\n  >>> clb = clbmap.omkeys[(1, 13)]\n  >>> clb\n  Clbmap(det_oid='D_ORCA003', du=1, floor=13, serial_number=374, upi='3.4.3.2/V2-2-1/2.374', dom_id=808949902)\n  >>> clb.dom_id\n  808949902\n  >>> clb.upi\n  '3.4.3.2/V2-2-1/2.374'\n\nCommand Line Utilities\n----------------------\n\nThe following command line utilities will be accessible after installing ``km3db``.\n\n``detx``\n~~~~~~~~~~~~\n\nThe ``detx`` command can be used to retrieve calibration information from the\ndatabase formatted as DETX, which is its main offline representation format::\n\n  $ detx -h\n  Retrieves DETX files from the database.\n\n  Usage:\n      detx [options] DET_ID\n      detx DET_ID RUN\n      detx (-h | --help)\n      detx --version\n\n  Options:\n      DET_ID        The detector ID (e.g. 49)\n      RUN           The run ID.\n      -c CALIBR_ID  Geometrical calibration ID (eg. A01466417)\n      -t T0_SET     Time calibration ID (eg. A01466431)\n      -o OUT        Output folder or filename.\n      -h --help     Show this screen.\n\n  Example:\n\n      detx 49 8220  # retrieve the calibrated DETX for run 8220 of ORCA6\n\n``streamds``\n~~~~~~~~~~~~\n\nThe ``streamds`` command provides access to the \"Stream Data Service\" which was\ndesigned to deal with large datasets potentially exceeding multiple GB in size.\nThe help output explains all the available functionality of the tool::\n\n  $ streamds -h\n  Access the KM3NeT StreamDS DataBase service.\n\n  Usage:\n      streamds\n      streamds list\n      streamds info STREAM\n      streamds get [-f FORMAT] STREAM [PARAMETERS...]\n      streamds (-h | --help)\n      streamds --version\n\n  Options:\n      STREAM      Name of the stream.\n      PARAMETERS  List of parameters separated by space (e.g. detid=29).\n      -f FORMAT   Usually 'txt' for ASCII or 'text' for UTF-8 [default: txt].\n      -x          Do not verify the SSL certificate.\n      -h --help   Show this screen.\n\nFor example, a list of available detectors::\n\n  > streamds get detectors\n  OID\tSERIALNUMBER\tLOCATIONID\tCITY\tFIRSTRUN\tLASTRUN\n  D_DU1CPPM\t2\tA00070004\tMarseille\t2\t10\n  A00350276\t3\tA00070003\tNapoli\t0\t0\n  D_DU2NAPO\t5\tA00070003\tNapoli\t98\t428\n  D_TESTDET\t6\tA00070002\tFisciano\t3\t35\n  D_ARCA001\t7\tA00073795\tItaly\t1\t2763\n  FR_INFRAS\t8\tA00073796\tFrance\t1600\t3202\n  D_DU003NA\t9\tA00070003\tNapoli\t1\t242\n  D_DU004NA\t12\tA00070003\tNapoli\t243\t342\n  D_DU001MA\t13\tA00070004\tMarseille\t1\t1922\n  D_ARCA003\t14\tA00073795\tItaly\t1\t6465\n\n\n``km3db``\n~~~~~~~~~\n\nThe ``km3db`` command gives direct access to database URLs and is mainly a\ndebugging tool::\n\n  $ km3db -h\n  Command line access to the KM3NeT DB web API.\n\n  Usage:\n      km3db URL\n      km3db (-h | --help)\n      km3db --version\n\n  Options:\n      URL         The URL, starting from the database website's root.\n      -h --help   Show this screen.\n\n  Example:\n\n      km3db \"streamds/runs.txt?detid=D_ARCA003\"\n\nThe URL parameter is simply the string which comes right after\n``https://km3netdbweb.in2p3.fr/``.\n\n\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "KM3NeT database library",
    "version": "0.5.2",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ba3e2d497839a88b667f95c335d14074",
                "sha256": "8bcdd8a709c291787a4cd997cedc8a6afc2f6f974def47ac034474a699e3cdbe"
            },
            "downloads": -1,
            "filename": "km3db-0.5.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ba3e2d497839a88b667f95c335d14074",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=2.7",
            "size": 18176,
            "upload_time": "2021-02-23T17:54:55",
            "upload_time_iso_8601": "2021-02-23T17:54:55.144913Z",
            "url": "https://files.pythonhosted.org/packages/14/a6/b7abc8278d3a1369f3f4cf175bfbdb7bf7f1cbcd66d87770db56e2191e9e/km3db-0.5.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "58434a4b9805daffbda9b8064083eadc",
                "sha256": "c0c2ada0e9ca9eae54a8340924b91c5418d211bd0e1ed71b91e10351cf6e9789"
            },
            "downloads": -1,
            "filename": "km3db-0.5.2.tar.gz",
            "has_sig": false,
            "md5_digest": "58434a4b9805daffbda9b8064083eadc",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7",
            "size": 56539,
            "upload_time": "2021-02-23T17:54:56",
            "upload_time_iso_8601": "2021-02-23T17:54:56.484265Z",
            "url": "https://files.pythonhosted.org/packages/18/84/d772edd06d90d21a86d6c17c7b6fd60807192de11f57d71ff8f7b7e84430/km3db-0.5.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-02-23 17:54:56",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "km3db"
}
        
Elapsed time: 0.23809s