########
influxio
########
.. start-badges
|ci-tests| |ci-coverage| |license| |pypi-downloads|
|python-versions| |status| |pypi-version|
.. |ci-tests| image:: https://github.com/daq-tools/influxio/actions/workflows/tests.yml/badge.svg
:target: https://github.com/daq-tools/influxio/actions/workflows/tests.yml
:alt: Build status
.. |ci-coverage| image:: https://codecov.io/gh/daq-tools/influxio/branch/main/graph/badge.svg
:target: https://app.codecov.io/gh/daq-tools/influxio
:alt: Coverage
.. |pypi-version| image:: https://img.shields.io/pypi/v/influxio.svg
:target: https://pypi.org/project/influxio/
:alt: PyPI Version
.. |python-versions| image:: https://img.shields.io/pypi/pyversions/influxio.svg
:target: https://pypi.org/project/influxio/
:alt: Python Version
.. |pypi-downloads| image:: https://static.pepy.tech/badge/influxio/month
:target: https://www.pepy.tech/projects/influxio
:alt: PyPI Downloads per month
.. |status| image:: https://img.shields.io/pypi/status/influxio.svg
:target: https://pypi.org/project/influxio/
:alt: Status
.. |license| image:: https://img.shields.io/pypi/l/influxio.svg
:target: https://pypi.org/project/influxio/
:alt: License
.. end-badges
*****
About
*****
You can use ``influxio`` to import and export data into/from InfluxDB.
It can be used both as a standalone program, and as a library.
``influxio`` is, amongst others, based on the excellent `dask`_, `fsspec`_,
`influxdb-client`_, `influx-line`_, `line-protocol-parser`_, `pandas`_,
and `SQLAlchemy`_ packages.
Please note that ``influxio`` is alpha-quality software, and a work in progress.
Contributions of all kinds are very welcome, in order to make it more solid.
Breaking changes should be expected until a 1.0 release, so version pinning
is recommended, especially when you use it as a library.
********
Synopsis
********
.. code-block:: shell
# Export from API to database.
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"sqlite://export.sqlite?table=demo"
# Export from data directory to line protocol format.
influxio copy \
"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
"file://export.lp"
**********
Quickstart
**********
If you are in a hurry, and want to run ``influxio`` without any installation,
just use the OCI image on Podman or Docker.
.. code-block:: shell
docker run --rm --network=host ghcr.io/daq-tools/influxio \
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"crate://crate@localhost:4200/testdrive/demo"
*****
Setup
*****
Install ``influxio`` from PyPI.
.. code-block:: shell
pip install influxio
*****
Usage
*****
This section outlines some example invocations of ``influxio``, both on the
command line, and per library use. Other than the resources available from
the web, testing data can be acquired from the repository's `testdata`_ folder.
Prerequisites
=============
For properly running some of the example invocations outlined below, you will
need an InfluxDB and a CrateDB server. The easiest way to spin up those
instances is to use Podman or Docker.
Please visit the ``docs/development.rst`` documentation to learn about how to
spin up corresponding sandbox instances on your workstation.
Command line use
================
Help
----
.. code-block:: shell
influxio --help
influxio info
influxio copy --help
Import
------
.. code-block:: shell
# From test data to API.
# Choose one of dummy, mixed, dateindex, wide.
influxio copy \
"testdata://dateindex/" \
"http://example:token@localhost:8086/testdrive/demo"
# With selected amount of rows.
influxio copy \
"testdata://dateindex/?rows=42" \
"http://example:token@localhost:8086/testdrive/demo"
# With selected amount of rows and columns (only supported by certain test data sources).
influxio copy \
"testdata://wide/?rows=42&columns=42" \
"http://example:token@localhost:8086/testdrive/demo"
# From line protocol file to InfluxDB API.
influxio copy \
"file://tests/testdata/basic.lp" \
"http://example:token@localhost:8086/testdrive/demo"
# From line protocol file to InfluxDB API.
influxio copy \
"https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
"http://example:token@localhost:8086/testdrive/demo"
# From line protocol file to any database supported by SQLAlchemy.
influxio copy \
"file://export.lp" \
"sqlite://export.sqlite?table=export"
Export from API
---------------
.. code-block:: shell
# From API to database file.
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"sqlite:///export.sqlite?table=demo"
# From API to database server.
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"crate://crate@localhost:4200/testdrive?table=demo"
# From API to line protocol file.
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"file://export.lp"
# From API to line protocol on stdout.
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"file://-?format=lp"
Export from data directory
--------------------------
.. code-block:: shell
# From InfluxDB data directory to line protocol file.
influxio copy \
"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
"file://export.lp"
# From InfluxDB data directory to line protocol file, compressed with gzip.
influxio copy \
"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
"file://export.lp.gz"
# From InfluxDB data directory to line protocol on stdout.
influxio copy \
"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
""file://-?format=lp"
OCI
---
OCI images are available on the GitHub Container Registry (GHCR). In order to
run them on Podman or Docker, invoke:
.. code-block:: shell
docker run --rm --network=host ghcr.io/daq-tools/influxio \
influxio copy \
"http://example:token@localhost:8086/testdrive/demo" \
"stdout://export.lp"
If you want to work with files on your filesystem, you will need to either
mount the working directory into the container using the ``--volume`` option,
or use the ``--interactive`` option to consume STDIN, like:
.. code-block:: shell
docker run --rm --volume=$(pwd):/data ghcr.io/daq-tools/influxio \
influxio copy "file:///data/export.lp" "sqlite:///data/export.sqlite?table=export"
cat export.lp | \
docker run --rm --interactive --network=host ghcr.io/daq-tools/influxio \
influxio copy "stdin://?format=lp" "crate://crate@localhost:4200/testdrive/export"
In order to always run the latest ``nightly`` development version, and to use a
shortcut for that, this section outlines how to use an alias for ``influxio``,
and a variable for storing the input URL. It may be useful to save a few
keystrokes on subsequent invocations.
.. code-block:: shell
docker pull ghcr.io/daq-tools/influxio:nightly
alias influxio="docker run --rm --interactive ghcr.io/daq-tools/influxio:nightly influxio"
SOURCE=https://github.com/daq-tools/influxio/raw/main/tests/testdata/basic.lp
TARGET=crate://crate@localhost:4200/testdrive/basic
influxio copy "${SOURCE}" "${TARGET}"
*******************
Project information
*******************
Development
===========
For installing the project from source, please follow the `development`_
documentation.
Prior art
=========
There are a few other projects which are aiming at similar goals.
- `InfluxDB Fetcher`_
- `influxdb-write-to-postgresql`_ (IW2PG)
- `Outflux`_
.. _dask: https://www.dask.org/
.. _development: doc/development.rst
.. _fsspec: https://pypi.org/project/fsspec/
.. _influx: https://docs.influxdata.com/influxdb/latest/reference/cli/influx/
.. _influx-line: https://github.com/functionoffunction/influx-line
.. _influxd: https://docs.influxdata.com/influxdb/latest/reference/cli/influxd/
.. _InfluxDB Fetcher: https://github.com/hgomez/influxdb
.. _InfluxDB line protocol: https://docs.influxdata.com/influxdb/latest/reference/syntax/line-protocol/
.. _influxdb-client: https://github.com/influxdata/influxdb-client-python
.. _influxdb-write-to-postgresql: https://github.com/eras/influxdb-write-to-postgresql
.. _line-protocol-parser: https://github.com/Penlect/line-protocol-parser
.. _list of other projects: doc/prior-art.rst
.. _Outflux: https://github.com/timescale/outflux
.. _pandas: https://pandas.pydata.org/
.. _SQLAlchemy: https://pypi.org/project/SQLAlchemy/
.. _testdata: https://github.com/daq-tools/influxio/tree/main/tests/testdata
Raw data
{
"_id": null,
"home_page": null,
"name": "influxio",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.8",
"maintainer_email": null,
"keywords": "export, ilp, import, influx, influxdb, line-protocol, lineprotocol, rdbms, sql, sqlalchemy, timeseries, timeseries-data, transfer",
"author": null,
"author_email": "Andreas Motl <andreas.motl@panodata.org>, Richard Pobering <richard.pobering@panodata.org>",
"download_url": "https://files.pythonhosted.org/packages/92/1e/b68aab39a96409a3df10e104519d251efd02cee90ac39329bd1ab76645e9/influxio-0.2.0.tar.gz",
"platform": null,
"description": "########\ninfluxio\n########\n\n.. start-badges\n\n|ci-tests| |ci-coverage| |license| |pypi-downloads|\n\n|python-versions| |status| |pypi-version|\n\n.. |ci-tests| image:: https://github.com/daq-tools/influxio/actions/workflows/tests.yml/badge.svg\n :target: https://github.com/daq-tools/influxio/actions/workflows/tests.yml\n :alt: Build status\n\n.. |ci-coverage| image:: https://codecov.io/gh/daq-tools/influxio/branch/main/graph/badge.svg\n :target: https://app.codecov.io/gh/daq-tools/influxio\n :alt: Coverage\n\n.. |pypi-version| image:: https://img.shields.io/pypi/v/influxio.svg\n :target: https://pypi.org/project/influxio/\n :alt: PyPI Version\n\n.. |python-versions| image:: https://img.shields.io/pypi/pyversions/influxio.svg\n :target: https://pypi.org/project/influxio/\n :alt: Python Version\n\n.. |pypi-downloads| image:: https://static.pepy.tech/badge/influxio/month\n :target: https://www.pepy.tech/projects/influxio\n :alt: PyPI Downloads per month\n\n.. |status| image:: https://img.shields.io/pypi/status/influxio.svg\n :target: https://pypi.org/project/influxio/\n :alt: Status\n\n.. |license| image:: https://img.shields.io/pypi/l/influxio.svg\n :target: https://pypi.org/project/influxio/\n :alt: License\n\n.. end-badges\n\n\n*****\nAbout\n*****\n\nYou can use ``influxio`` to import and export data into/from InfluxDB.\nIt can be used both as a standalone program, and as a library.\n\n``influxio`` is, amongst others, based on the excellent `dask`_, `fsspec`_,\n`influxdb-client`_, `influx-line`_, `line-protocol-parser`_, `pandas`_,\nand `SQLAlchemy`_ packages.\n\nPlease note that ``influxio`` is alpha-quality software, and a work in progress.\nContributions of all kinds are very welcome, in order to make it more solid.\nBreaking changes should be expected until a 1.0 release, so version pinning\nis recommended, especially when you use it as a library.\n\n\n********\nSynopsis\n********\n\n.. code-block:: shell\n\n # Export from API to database.\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"sqlite://export.sqlite?table=demo\"\n\n # Export from data directory to line protocol format.\n influxio copy \\\n \"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo\" \\\n \"file://export.lp\"\n\n\n**********\nQuickstart\n**********\n\nIf you are in a hurry, and want to run ``influxio`` without any installation,\njust use the OCI image on Podman or Docker.\n\n.. code-block:: shell\n\n docker run --rm --network=host ghcr.io/daq-tools/influxio \\\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"crate://crate@localhost:4200/testdrive/demo\"\n\n\n*****\nSetup\n*****\n\nInstall ``influxio`` from PyPI.\n\n.. code-block:: shell\n\n pip install influxio\n\n\n*****\nUsage\n*****\n\nThis section outlines some example invocations of ``influxio``, both on the\ncommand line, and per library use. Other than the resources available from\nthe web, testing data can be acquired from the repository's `testdata`_ folder.\n\nPrerequisites\n=============\n\nFor properly running some of the example invocations outlined below, you will\nneed an InfluxDB and a CrateDB server. The easiest way to spin up those\ninstances is to use Podman or Docker.\n\nPlease visit the ``docs/development.rst`` documentation to learn about how to\nspin up corresponding sandbox instances on your workstation.\n\nCommand line use\n================\n\nHelp\n----\n\n.. code-block:: shell\n\n influxio --help\n influxio info\n influxio copy --help\n\nImport\n------\n\n.. code-block:: shell\n\n # From test data to API.\n # Choose one of dummy, mixed, dateindex, wide.\n influxio copy \\\n \"testdata://dateindex/\" \\\n \"http://example:token@localhost:8086/testdrive/demo\"\n\n # With selected amount of rows.\n influxio copy \\\n \"testdata://dateindex/?rows=42\" \\\n \"http://example:token@localhost:8086/testdrive/demo\"\n\n # With selected amount of rows and columns (only supported by certain test data sources).\n influxio copy \\\n \"testdata://wide/?rows=42&columns=42\" \\\n \"http://example:token@localhost:8086/testdrive/demo\"\n\n # From line protocol file to InfluxDB API.\n influxio copy \\\n \"file://tests/testdata/basic.lp\" \\\n \"http://example:token@localhost:8086/testdrive/demo\"\n\n # From line protocol file to InfluxDB API.\n influxio copy \\\n \"https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp\" \\\n \"http://example:token@localhost:8086/testdrive/demo\"\n\n # From line protocol file to any database supported by SQLAlchemy.\n influxio copy \\\n \"file://export.lp\" \\\n \"sqlite://export.sqlite?table=export\"\n\n\nExport from API\n---------------\n\n.. code-block:: shell\n\n # From API to database file.\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"sqlite:///export.sqlite?table=demo\"\n\n # From API to database server.\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"crate://crate@localhost:4200/testdrive?table=demo\"\n\n # From API to line protocol file.\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"file://export.lp\"\n\n # From API to line protocol on stdout.\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"file://-?format=lp\"\n\n\nExport from data directory\n--------------------------\n\n.. code-block:: shell\n\n # From InfluxDB data directory to line protocol file.\n influxio copy \\\n \"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo\" \\\n \"file://export.lp\"\n\n # From InfluxDB data directory to line protocol file, compressed with gzip.\n influxio copy \\\n \"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo\" \\\n \"file://export.lp.gz\"\n\n # From InfluxDB data directory to line protocol on stdout.\n influxio copy \\\n \"file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo\" \\\n \"\"file://-?format=lp\"\n\n\nOCI\n---\n\nOCI images are available on the GitHub Container Registry (GHCR). In order to\nrun them on Podman or Docker, invoke:\n\n.. code-block:: shell\n\n docker run --rm --network=host ghcr.io/daq-tools/influxio \\\n influxio copy \\\n \"http://example:token@localhost:8086/testdrive/demo\" \\\n \"stdout://export.lp\"\n\nIf you want to work with files on your filesystem, you will need to either\nmount the working directory into the container using the ``--volume`` option,\nor use the ``--interactive`` option to consume STDIN, like:\n\n.. code-block:: shell\n\n docker run --rm --volume=$(pwd):/data ghcr.io/daq-tools/influxio \\\n influxio copy \"file:///data/export.lp\" \"sqlite:///data/export.sqlite?table=export\"\n\n cat export.lp | \\\n docker run --rm --interactive --network=host ghcr.io/daq-tools/influxio \\\n influxio copy \"stdin://?format=lp\" \"crate://crate@localhost:4200/testdrive/export\"\n\nIn order to always run the latest ``nightly`` development version, and to use a\nshortcut for that, this section outlines how to use an alias for ``influxio``,\nand a variable for storing the input URL. It may be useful to save a few\nkeystrokes on subsequent invocations.\n\n.. code-block:: shell\n\n docker pull ghcr.io/daq-tools/influxio:nightly\n alias influxio=\"docker run --rm --interactive ghcr.io/daq-tools/influxio:nightly influxio\"\n SOURCE=https://github.com/daq-tools/influxio/raw/main/tests/testdata/basic.lp\n TARGET=crate://crate@localhost:4200/testdrive/basic\n\n influxio copy \"${SOURCE}\" \"${TARGET}\"\n\n\n*******************\nProject information\n*******************\n\nDevelopment\n===========\nFor installing the project from source, please follow the `development`_\ndocumentation.\n\nPrior art\n=========\nThere are a few other projects which are aiming at similar goals.\n\n- `InfluxDB Fetcher`_\n- `influxdb-write-to-postgresql`_ (IW2PG)\n- `Outflux`_\n\n\n.. _dask: https://www.dask.org/\n.. _development: doc/development.rst\n.. _fsspec: https://pypi.org/project/fsspec/\n.. _influx: https://docs.influxdata.com/influxdb/latest/reference/cli/influx/\n.. _influx-line: https://github.com/functionoffunction/influx-line\n.. _influxd: https://docs.influxdata.com/influxdb/latest/reference/cli/influxd/\n.. _InfluxDB Fetcher: https://github.com/hgomez/influxdb\n.. _InfluxDB line protocol: https://docs.influxdata.com/influxdb/latest/reference/syntax/line-protocol/\n.. _influxdb-client: https://github.com/influxdata/influxdb-client-python\n.. _influxdb-write-to-postgresql: https://github.com/eras/influxdb-write-to-postgresql\n.. _line-protocol-parser: https://github.com/Penlect/line-protocol-parser\n.. _list of other projects: doc/prior-art.rst\n.. _Outflux: https://github.com/timescale/outflux\n.. _pandas: https://pandas.pydata.org/\n.. _SQLAlchemy: https://pypi.org/project/SQLAlchemy/\n.. _testdata: https://github.com/daq-tools/influxio/tree/main/tests/testdata\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Import and export data into/from InfluxDB",
"version": "0.2.0",
"project_urls": {
"changelog": "https://github.com/daq-tools/influxio/blob/main/CHANGES.rst",
"documentation": "https://github.com/daq-tools/influxio",
"homepage": "https://github.com/daq-tools/influxio",
"repository": "https://github.com/daq-tools/influxio"
},
"split_keywords": [
"export",
" ilp",
" import",
" influx",
" influxdb",
" line-protocol",
" lineprotocol",
" rdbms",
" sql",
" sqlalchemy",
" timeseries",
" timeseries-data",
" transfer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1cd55a72c04f31d61a7ba312df993f2e7901468fa3c2f3eccba59e7f21f0f23d",
"md5": "41e8060eef179fc5844f4028849d0e9b",
"sha256": "5d0b6e0a7779cdf065d373ba0a32d95f06934e1866e1ee8b06e67e99b197f22d"
},
"downloads": -1,
"filename": "influxio-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "41e8060eef179fc5844f4028849d0e9b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.8",
"size": 20735,
"upload_time": "2024-04-10T01:31:25",
"upload_time_iso_8601": "2024-04-10T01:31:25.196984Z",
"url": "https://files.pythonhosted.org/packages/1c/d5/5a72c04f31d61a7ba312df993f2e7901468fa3c2f3eccba59e7f21f0f23d/influxio-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "921eb68aab39a96409a3df10e104519d251efd02cee90ac39329bd1ab76645e9",
"md5": "efd81a552e4162ca0dae86e7dc67c285",
"sha256": "79525a7cebe353c1f100ab7c8dd1c02167179a01b64f9fe86b4222939976860f"
},
"downloads": -1,
"filename": "influxio-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "efd81a552e4162ca0dae86e7dc67c285",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.8",
"size": 23839,
"upload_time": "2024-04-10T01:31:28",
"upload_time_iso_8601": "2024-04-10T01:31:28.500707Z",
"url": "https://files.pythonhosted.org/packages/92/1e/b68aab39a96409a3df10e104519d251efd02cee90ac39329bd1ab76645e9/influxio-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-10 01:31:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "daq-tools",
"github_project": "influxio",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "influxio"
}
JSON
