<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Apache IoTDB
[](https://github.com/apache/iotdb/actions/workflows/client-python.yml)
[](https://github.com/apache/iotdb/releases)
[](https://www.apache.org/licenses/LICENSE-2.0.html)
[](https://iotdb.apache.org/)
Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for
data management and analysis, deployable on the edge and the cloud. Due to its light-weight
architecture, high performance and rich feature set together with its deep integration with
Apache Hadoop, Spark and Flink, Apache IoTDB can meet the requirements of massive data storage,
high-speed data ingestion and complex data analysis in the IoT industrial fields.
## Python Native API
### Requirements
You have to install thrift (>=0.13) before using the package.
### How to use (Example)
First, download the latest package: `pip3 install apache-iotdb`
*Notice: If you are installing Python API v0.13.0, DO NOT install by `pip install apache-iotdb==0.13.0`, use `pip install apache-iotdb==0.13.0.post1` instead!*
You can get an example of using the package to read and write data at here: [Example](https://github.com/apache/iotdb/blob/master/client-py/SessionExample.py)
An example of aligned timeseries: [Aligned Timeseries Session Example](https://github.com/apache/iotdb/blob/master/client-py/SessionAlignedTimeseriesExample.py)
(you need to add `import iotdb` in the head of the file)
Or:
```python
from iotdb.Session import Session
ip = "127.0.0.1"
port_ = "6667"
username_ = "root"
password_ = "root"
session = Session(ip, port_, username_, password_)
session.open(False)
zone = session.get_time_zone()
session.close()
```
### Initialization
* Initialize a Session
```python
session = Session(ip, port_, username_, password_, fetch_size=1024, zone_id="UTC+8")
```
* Open a session, with a parameter to specify whether to enable RPC compression
```python
session.open(enable_rpc_compression=False)
```
Notice: this RPC compression status of client must comply with that of IoTDB server
* Close a Session
```python
session.close()
```
### Data Definition Interface (DDL Interface)
#### DATABASE Management
* CREATE DATABASE
```python
session.set_storage_group(group_name)
```
* Delete one or several databases
```python
session.delete_storage_group(group_name)
session.delete_storage_groups(group_name_lst)
```
#### Timeseries Management
* Create one or multiple timeseries
```python
session.create_time_series(ts_path, data_type, encoding, compressor,
props=None, tags=None, attributes=None, alias=None)
session.create_multi_time_series(
ts_path_lst, data_type_lst, encoding_lst, compressor_lst,
props_lst=None, tags_lst=None, attributes_lst=None, alias_lst=None
)
```
* Create aligned timeseries
```python
session.create_aligned_time_series(
device_id, measurements_lst, data_type_lst, encoding_lst, compressor_lst
)
```
Attention: Alias of measurements are **not supported** currently.
* Delete one or several timeseries
```python
session.delete_time_series(paths_list)
```
* Check whether the specific timeseries exists
```python
session.check_time_series_exists(path)
```
### Data Manipulation Interface (DML Interface)
#### Insert
It is recommended to use insertTablet to help improve write efficiency.
* Insert a Tablet，which is multiple rows of a device, each row has the same measurements
* **Better Write Performance**
* **Support null values**: fill the null value with any value, and then mark the null value via BitMap (from v0.13)
We have two implementations of Tablet in Python API.
* Normal Tablet
```python
values_ = [
[False, 10, 11, 1.1, 10011.1, "test01"],
[True, 100, 11111, 1.25, 101.0, "test02"],
[False, 100, 1, 188.1, 688.25, "test03"],
[True, 0, 0, 0, 6.25, "test04"],
]
timestamps_ = [1, 2, 3, 4]
tablet_ = Tablet(
device_id, measurements_, data_types_, values_, timestamps_
)
session.insert_tablet(tablet_)
values_ = [
[None, 10, 11, 1.1, 10011.1, "test01"],
[True, None, 11111, 1.25, 101.0, "test02"],
[False, 100, None, 188.1, 688.25, "test03"],
[True, 0, 0, 0, None, None],
]
timestamps_ = [16, 17, 18, 19]
tablet_ = Tablet(
device_id, measurements_, data_types_, values_, timestamps_
)
session.insert_tablet(tablet_)
```
* Numpy Tablet
Comparing with Tablet, Numpy Tablet is using [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) to record data.
With less memory footprint and time cost of serialization, the insert performance will be better.
**Notice**
1. time and value columns in Tablet are ndarray.
2. recommended to use the specific dtypes to each ndarray, see the example below
(if not, the default dtypes are also ok).
```python
import numpy as np
data_types_ = [
TSDataType.BOOLEAN,
TSDataType.INT32,
TSDataType.INT64,
TSDataType.FLOAT,
TSDataType.DOUBLE,
TSDataType.TEXT,
]
np_values_ = [
np.array([False, True, False, True], TSDataType.BOOLEAN.np_dtype()),
np.array([10, 100, 100, 0], TSDataType.INT32.np_dtype()),
np.array([11, 11111, 1, 0], TSDataType.INT64.np_dtype()),
np.array([1.1, 1.25, 188.1, 0], TSDataType.FLOAT.np_dtype()),
np.array([10011.1, 101.0, 688.25, 6.25], TSDataType.DOUBLE.np_dtype()),
np.array(["test01", "test02", "test03", "test04"], TSDataType.TEXT.np_dtype()),
]
np_timestamps_ = np.array([1, 2, 3, 4], TSDataType.INT64.np_dtype())
np_tablet_ = NumpyTablet(
device_id, measurements_, data_types_, np_values_, np_timestamps_
)
session.insert_tablet(np_tablet_)
# insert one numpy tablet with none into the database.
np_values_ = [
np.array([False, True, False, True], TSDataType.BOOLEAN.np_dtype()),
np.array([10, 100, 100, 0], TSDataType.INT32.np_dtype()),
np.array([11, 11111, 1, 0], TSDataType.INT64.np_dtype()),
np.array([1.1, 1.25, 188.1, 0], TSDataType.FLOAT.np_dtype()),
np.array([10011.1, 101.0, 688.25, 6.25], TSDataType.DOUBLE.np_dtype()),
np.array(["test01", "test02", "test03", "test04"], TSDataType.TEXT.np_dtype()),
]
np_timestamps_ = np.array([98, 99, 100, 101], TSDataType.INT64.np_dtype())
np_bitmaps_ = []
for i in range(len(measurements_)):
np_bitmaps_.append(BitMap(len(np_timestamps_)))
np_bitmaps_[0].mark(0)
np_bitmaps_[1].mark(1)
np_bitmaps_[2].mark(2)
np_bitmaps_[4].mark(3)
np_bitmaps_[5].mark(3)
np_tablet_with_none = NumpyTablet(
device_id, measurements_, data_types_, np_values_, np_timestamps_, np_bitmaps_
)
session.insert_tablet(np_tablet_with_none)
```
* Insert multiple Tablets
```python
session.insert_tablets(tablet_lst)
```
* Insert a Record
```python
session.insert_record(device_id, timestamp, measurements_, data_types_, values_)
```
* Insert multiple Records
```python
session.insert_records(
device_ids_, time_list_, measurements_list_, data_type_list_, values_list_
)
```
* Insert multiple Records that belong to the same device.
With type info the server has no need to do type inference, which leads a better performance
```python
session.insert_records_of_one_device(device_id, time_list, measurements_list, data_types_list, values_list)
```
#### Insert with type inference
When the data is of String type, we can use the following interface to perform type inference based on the value of the value itself. For example, if value is "true" , it can be automatically inferred to be a boolean type. If value is "3.2" , it can be automatically inferred as a flout type. Without type information, server has to do type inference, which may cost some time.
* Insert a Record, which contains multiple measurement value of a device at a timestamp
```python
session.insert_str_record(device_id, timestamp, measurements, string_values)
```
#### Insert of Aligned Timeseries
The Insert of aligned timeseries uses interfaces like insert_aligned_XXX, and others are similar to the above interfaces:
* insert_aligned_record
* insert_aligned_records
* insert_aligned_records_of_one_device
* insert_aligned_tablet
* insert_aligned_tablets
### IoTDB-SQL Interface
* Execute query statement
```python
session.execute_query_statement(sql)
```
* Execute non query statement
```python
session.execute_non_query_statement(sql)
```
* Execute statement
```python
session.execute_statement(sql)
```
### Device Template
#### Create Device Template
The step for creating a metadata template is as follows
1. Create the template class
2. Adding child Node，InternalNode and MeasurementNode can be chose
3. Execute create device template function
```python
template = Template(name=template_name, share_time=True)
i_node_gps = InternalNode(name="GPS", share_time=False)
i_node_v = InternalNode(name="vehicle", share_time=True)
m_node_x = MeasurementNode("x", TSDataType.FLOAT, TSEncoding.RLE, Compressor.SNAPPY)
i_node_gps.add_child(m_node_x)
i_node_v.add_child(m_node_x)
template.add_template(i_node_gps)
template.add_template(i_node_v)
template.add_template(m_node_x)
session.create_schema_template(template)
```
#### Modify Device Template nodes
Modify nodes in a template, the template must be already created. These are functions that add or delete some measurement nodes.
* add node in template
```python
session.add_measurements_in_template(template_name, measurements_path, data_types, encodings, compressors, is_aligned)
```
* delete node in template
```python
session.delete_node_in_template(template_name, path)
```
#### Set Device Template
```python
session.set_schema_template(template_name, prefix_path)
```
#### Uset Device Template
```python
session.unset_schema_template(template_name, prefix_path)
```
#### Show Device Template
* Show all device templates
```python
session.show_all_templates()
```
* Count all nodes in templates
```python
session.count_measurements_in_template(template_name)
```
* Judge whether the path is measurement or not in templates, This measurement must be in the template
```python
session.count_measurements_in_template(template_name, path)
```
* Judge whether the path is exist or not in templates, This path may not belong to the template
```python
session.is_path_exist_in_template(template_name, path)
```
* Show nodes under in device template
```python
session.show_measurements_in_template(template_name)
```
* Show the path prefix where a device template is set
```python
session.show_paths_template_set_on(template_name)
```
* Show the path prefix where a device template is used (i.e. the time series has been created)
```python
session.show_paths_template_using_on(template_name)
```
#### Drop Device Template
Delete an existing metadata template，dropping an already set template is not supported
```python
session.drop_schema_template("template_python")
```
### Pandas Support
To easily transform a query result to a [Pandas Dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)
the SessionDataSet has a method `.todf()` which consumes the dataset and transforms it to a pandas dataframe.
Example:
```python
from iotdb.Session import Session
ip = "127.0.0.1"
port_ = "6667"
username_ = "root"
password_ = "root"
session = Session(ip, port_, username_, password_)
session.open(False)
result = session.execute_query_statement("SELECT * FROM root.*")
# Transform to Pandas Dataset
df = result.todf()
session.close()
# Now you can work with the dataframe
df = ...
```
### IoTDB Testcontainer
The Test Support is based on the lib `testcontainers` (https://testcontainers-python.readthedocs.io/en/latest/index.html) which you need to install in your project if you want to use the feature.
To start (and stop) an IoTDB Database in a Docker container simply do:
```python
class MyTestCase(unittest.TestCase):
def test_something(self):
with IoTDBContainer() as c:
session = Session("localhost", c.get_exposed_port(6667), "root", "root")
session.open(False)
result = session.execute_query_statement("SHOW TIMESERIES")
print(result)
session.close()
```
by default it will load the image `apache/iotdb:latest`, if you want a specific version just pass it like e.g. `IoTDBContainer("apache/iotdb:0.12.0")` to get version `0.12.0` running.
### IoTDB DBAPI
IoTDB DBAPI implements the Python DB API 2.0 specification (https://peps.python.org/pep-0249/), which defines a common
interface for accessing databases in Python.
#### Examples
+ Initialization
The initialized parameters are consistent with the session part (except for the sqlalchemy_mode).
```python
from iotdb.dbapi import connect
ip = "127.0.0.1"
port_ = "6667"
username_ = "root"
password_ = "root"
conn = connect(ip, port_, username_, password_,fetch_size=1024,zone_id="UTC+8",sqlalchemy_mode=False)
cursor = conn.cursor()
```
+ simple SQL statement execution
```python
cursor.execute("SELECT * FROM root.*")
for row in cursor.fetchall():
print(row)
```
+ execute SQL with parameter
IoTDB DBAPI supports pyformat style parameters
```python
cursor.execute("SELECT * FROM root.* WHERE time < %(time)s",{"time":"2017-11-01T00:08:00.000"})
for row in cursor.fetchall():
print(row)
```
+ execute SQL with parameter sequences
```python
seq_of_parameters = [
{"timestamp": 1, "temperature": 1},
{"timestamp": 2, "temperature": 2},
{"timestamp": 3, "temperature": 3},
{"timestamp": 4, "temperature": 4},
{"timestamp": 5, "temperature": 5},
]
sql = "insert into root.cursor(timestamp,temperature) values(%(timestamp)s,%(temperature)s)"
cursor.executemany(sql,seq_of_parameters)
```
+ close the connection and cursor
```python
cursor.close()
conn.close()
```
### IoTDB SQLAlchemy Dialect (Experimental)
The SQLAlchemy dialect of IoTDB is written to adapt to Apache Superset.
This part is still being improved.
Please do not use it in the production environment!
#### Mapping of the metadata
The data model used by SQLAlchemy is a relational data model, which describes the relationships between different entities through tables.
While the data model of IoTDB is a hierarchical data model, which organizes the data through a tree structure.
In order to adapt IoTDB to the dialect of SQLAlchemy, the original data model in IoTDB needs to be reorganized.
Converting the data model of IoTDB into the data model of SQLAlchemy.
The metadata in the IoTDB are：
1. Database
2. Path
3. Entity
4. Measurement
The metadata in the SQLAlchemy are：
1. Schema
2. Table
3. Column
The mapping relationship between them is：
| The metadata in the SQLAlchemy | The metadata in the IoTDB |
| -------------------- | ---------------------------------------------- |
| Schema | Database |
| Table | Path ( from database to entity ) + Entity |
| Column | Measurement |
The following figure shows the relationship between the two more intuitively:
#### Data type mapping
| data type in IoTDB | data type in SQLAlchemy |
|--------------------|-------------------------|
| BOOLEAN | Boolean |
| INT32 | Integer |
| INT64 | BigInteger |
| FLOAT | Float |
| DOUBLE | Float |
| TEXT | Text |
| LONG | BigInteger |
#### Example
+ execute statement
```python
from sqlalchemy import create_engine
engine = create_engine("iotdb://root:root@127.0.0.1:6667")
connect = engine.connect()
result = connect.execute("SELECT ** FROM root")
for row in result.fetchall():
print(row)
```
+ ORM (now only simple queries are supported)
```python
from sqlalchemy import create_engine, Column, Float, BigInteger, MetaData
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
metadata = MetaData(
schema='root.factory'
)
Base = declarative_base(metadata=metadata)
class Device(Base):
__tablename__ = "room2.device1"
Time = Column(BigInteger, primary_key=True)
temperature = Column(Float)
status = Column(Float)
engine = create_engine("iotdb://root:root@127.0.0.1:6667")
DbSession = sessionmaker(bind=engine)
session = DbSession()
res = session.query(Device.status).filter(Device.temperature > 1)
for row in res:
print(row)
```
## Developers
### Introduction
This is an example of how to connect to IoTDB with python, using the thrift rpc interfaces. Things are almost the same on Windows or Linux, but pay attention to the difference like path separator.
### Prerequisites
Python3.7 or later is preferred.
You have to install Thrift (0.11.0 or later) to compile our thrift file into python code. Below is the official tutorial of installation, eventually, you should have a thrift executable.
```
http://thrift.apache.org/docs/install/
```
Before starting you need to install `requirements_dev.txt` in your python environment, e.g. by calling
```shell
pip install -r requirements_dev.txt
```
### Compile the thrift library and Debug
In the root of IoTDB's source code folder, run `mvn clean generate-sources -pl client-py -am`.
This will automatically delete and repopulate the folder `iotdb/thrift` with the generated thrift files.
This folder is ignored from git and should **never be pushed to git!**
**Notice** Do not upload `iotdb/thrift` to the git repo.
### Session Client & Example
We packed up the Thrift interface in `client-py/src/iotdb/Session.py` (similar with its Java counterpart), also provided an example file `client-py/src/SessionExample.py` of how to use the session module. please read it carefully.
Or, another simple example:
```python
from iotdb.Session import Session
ip = "127.0.0.1"
port_ = "6667"
username_ = "root"
password_ = "root"
session = Session(ip, port_, username_, password_)
session.open(False)
zone = session.get_time_zone()
session.close()
```
### Tests
Please add your custom tests in `tests` folder.
To run all defined tests just type `pytest .` in the root folder.
**Notice** Some tests need docker to be started on your system as a test instance is started in a docker container using [testcontainers](https://testcontainers-python.readthedocs.io/en/latest/index.html).
### Futher Tools
[black](https://pypi.org/project/black/) and [flake8](https://pypi.org/project/flake8/) are installed for autoformatting and linting.
Both can be run by `black .` or `flake8 .` respectively.
## Releasing
To do a release just ensure that you have the right set of generated thrift files.
Then run linting and auto-formatting.
Then, ensure that all tests work (via `pytest .`).
Then you are good to go to do a release!
### Preparing your environment
First, install all necessary dev dependencies via `pip install -r requirements_dev.txt`.
### Doing the Release
There is a convenient script `release.sh` to do all steps for a release.
Namely, these are
* Remove all transient directories from last release (if exists)
* (Re-)generate all generated sources via mvn
* Run Linting (flake8)
* Run Tests via pytest
* Build
* Release to pypi
Raw data
{
"_id": null,
"home_page": "https://github.com/apache/iotdb",
"name": "apache-iotdb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Apache Software Foundation",
"author_email": "dev@iotdb.apache.org",
"download_url": "https://files.pythonhosted.org/packages/98/0a/a5c8bede42a5dcf2f2eadb8b37a406a1ff2fb4e4f78df149d7a63cac058f/apache-iotdb-1.3.0.tar.gz",
"platform": null,
"description": "<!--\n\n Licensed to the Apache Software Foundation (ASF) under one\n or more contributor license agreements. See the NOTICE file\n distributed with this work for additional information\n regarding copyright ownership. The ASF licenses this file\n to you under the Apache License, Version 2.0 (the\n \"License\"); you may not use this file except in compliance\n with the License. You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing,\n software distributed under the License is distributed on an\n \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n KIND, either express or implied. See the License for the\n specific language governing permissions and limitations\n under the License.\n\n-->\n\n# Apache IoTDB\n\n[](https://github.com/apache/iotdb/actions/workflows/client-python.yml)\n[](https://github.com/apache/iotdb/releases)\n[](https://www.apache.org/licenses/LICENSE-2.0.html)\n\n\n\n[](https://iotdb.apache.org/)\n\n\nApache IoTDB (Database for Internet of Things) is an IoT native database with high performance for \ndata management and analysis, deployable on the edge and the cloud. Due to its light-weight \narchitecture, high performance and rich feature set together with its deep integration with \nApache Hadoop, Spark and Flink, Apache IoTDB can meet the requirements of massive data storage, \nhigh-speed data ingestion and complex data analysis in the IoT industrial fields.\n\n## Python Native API\n\n### Requirements\n\nYou have to install thrift (>=0.13) before using the package.\n\n\n\n### How to use (Example)\n\nFirst, download the latest package: `pip3 install apache-iotdb`\n\n*Notice: If you are installing Python API v0.13.0, DO NOT install by `pip install apache-iotdb==0.13.0`, use `pip install apache-iotdb==0.13.0.post1` instead!* \n\nYou can get an example of using the package to read and write data at here: [Example](https://github.com/apache/iotdb/blob/master/client-py/SessionExample.py)\n\nAn example of aligned timeseries: [Aligned Timeseries Session Example](https://github.com/apache/iotdb/blob/master/client-py/SessionAlignedTimeseriesExample.py)\n\n(you need to add `import iotdb` in the head of the file)\n\nOr:\n\n```python\nfrom iotdb.Session import Session\n\nip = \"127.0.0.1\"\nport_ = \"6667\"\nusername_ = \"root\"\npassword_ = \"root\"\nsession = Session(ip, port_, username_, password_)\nsession.open(False)\nzone = session.get_time_zone()\nsession.close()\n```\n\n### Initialization\n\n* Initialize a Session\n\n```python\nsession = Session(ip, port_, username_, password_, fetch_size=1024, zone_id=\"UTC+8\")\n```\n\n* Open a session, with a parameter to specify whether to enable RPC compression\n\n```python\nsession.open(enable_rpc_compression=False)\n```\n\nNotice: this RPC compression status of client must comply with that of IoTDB server\n\n* Close a Session\n\n```python\nsession.close()\n```\n\n### Data Definition Interface (DDL Interface)\n\n#### DATABASE Management\n\n* CREATE DATABASE\n\n```python\nsession.set_storage_group(group_name)\n```\n\n* Delete one or several databases\n\n```python\nsession.delete_storage_group(group_name)\nsession.delete_storage_groups(group_name_lst)\n```\n#### Timeseries Management\n\n* Create one or multiple timeseries\n\n```python\nsession.create_time_series(ts_path, data_type, encoding, compressor,\n props=None, tags=None, attributes=None, alias=None)\n \nsession.create_multi_time_series(\n ts_path_lst, data_type_lst, encoding_lst, compressor_lst,\n props_lst=None, tags_lst=None, attributes_lst=None, alias_lst=None\n)\n```\n\n* Create aligned timeseries\n\n```python\nsession.create_aligned_time_series(\n device_id, measurements_lst, data_type_lst, encoding_lst, compressor_lst\n)\n```\n\nAttention: Alias of measurements are **not supported** currently.\n\n* Delete one or several timeseries\n\n```python\nsession.delete_time_series(paths_list)\n```\n\n* Check whether the specific timeseries exists\n\n```python\nsession.check_time_series_exists(path)\n```\n\n### Data Manipulation Interface (DML Interface)\n\n#### Insert\n\nIt is recommended to use insertTablet to help improve write efficiency.\n\n* Insert a Tablet\uff0cwhich is multiple rows of a device, each row has the same measurements\n * **Better Write Performance**\n * **Support null values**: fill the null value with any value, and then mark the null value via BitMap (from v0.13)\n\n\nWe have two implementations of Tablet in Python API.\n\n* Normal Tablet\n\n```python\nvalues_ = [\n [False, 10, 11, 1.1, 10011.1, \"test01\"],\n [True, 100, 11111, 1.25, 101.0, \"test02\"],\n [False, 100, 1, 188.1, 688.25, \"test03\"],\n [True, 0, 0, 0, 6.25, \"test04\"],\n]\ntimestamps_ = [1, 2, 3, 4]\ntablet_ = Tablet(\n device_id, measurements_, data_types_, values_, timestamps_\n)\nsession.insert_tablet(tablet_)\n\nvalues_ = [\n [None, 10, 11, 1.1, 10011.1, \"test01\"],\n [True, None, 11111, 1.25, 101.0, \"test02\"],\n [False, 100, None, 188.1, 688.25, \"test03\"],\n [True, 0, 0, 0, None, None],\n]\ntimestamps_ = [16, 17, 18, 19]\ntablet_ = Tablet(\n device_id, measurements_, data_types_, values_, timestamps_\n)\nsession.insert_tablet(tablet_)\n```\n* Numpy Tablet\n\nComparing with Tablet, Numpy Tablet is using [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) to record data.\nWith less memory footprint and time cost of serialization, the insert performance will be better.\n\n**Notice**\n1. time and value columns in Tablet are ndarray.\n2. recommended to use the specific dtypes to each ndarray, see the example below \n(if not, the default dtypes are also ok).\n\n```python\nimport numpy as np\ndata_types_ = [\n TSDataType.BOOLEAN,\n TSDataType.INT32,\n TSDataType.INT64,\n TSDataType.FLOAT,\n TSDataType.DOUBLE,\n TSDataType.TEXT,\n]\nnp_values_ = [\n np.array([False, True, False, True], TSDataType.BOOLEAN.np_dtype()),\n np.array([10, 100, 100, 0], TSDataType.INT32.np_dtype()),\n np.array([11, 11111, 1, 0], TSDataType.INT64.np_dtype()),\n np.array([1.1, 1.25, 188.1, 0], TSDataType.FLOAT.np_dtype()),\n np.array([10011.1, 101.0, 688.25, 6.25], TSDataType.DOUBLE.np_dtype()),\n np.array([\"test01\", \"test02\", \"test03\", \"test04\"], TSDataType.TEXT.np_dtype()),\n]\nnp_timestamps_ = np.array([1, 2, 3, 4], TSDataType.INT64.np_dtype())\nnp_tablet_ = NumpyTablet(\n device_id, measurements_, data_types_, np_values_, np_timestamps_\n)\nsession.insert_tablet(np_tablet_)\n\n# insert one numpy tablet with none into the database.\nnp_values_ = [\n np.array([False, True, False, True], TSDataType.BOOLEAN.np_dtype()),\n np.array([10, 100, 100, 0], TSDataType.INT32.np_dtype()),\n np.array([11, 11111, 1, 0], TSDataType.INT64.np_dtype()),\n np.array([1.1, 1.25, 188.1, 0], TSDataType.FLOAT.np_dtype()),\n np.array([10011.1, 101.0, 688.25, 6.25], TSDataType.DOUBLE.np_dtype()),\n np.array([\"test01\", \"test02\", \"test03\", \"test04\"], TSDataType.TEXT.np_dtype()),\n]\nnp_timestamps_ = np.array([98, 99, 100, 101], TSDataType.INT64.np_dtype())\nnp_bitmaps_ = []\nfor i in range(len(measurements_)):\n np_bitmaps_.append(BitMap(len(np_timestamps_)))\nnp_bitmaps_[0].mark(0)\nnp_bitmaps_[1].mark(1)\nnp_bitmaps_[2].mark(2)\nnp_bitmaps_[4].mark(3)\nnp_bitmaps_[5].mark(3)\nnp_tablet_with_none = NumpyTablet(\n device_id, measurements_, data_types_, np_values_, np_timestamps_, np_bitmaps_\n)\nsession.insert_tablet(np_tablet_with_none)\n```\n\n* Insert multiple Tablets\n\n```python\nsession.insert_tablets(tablet_lst)\n```\n\n* Insert a Record\n\n```python\nsession.insert_record(device_id, timestamp, measurements_, data_types_, values_)\n```\n\n* Insert multiple Records\n\n```python\nsession.insert_records(\n device_ids_, time_list_, measurements_list_, data_type_list_, values_list_\n)\n```\n\n* Insert multiple Records that belong to the same device.\n With type info the server has no need to do type inference, which leads a better performance\n\n\n```python\nsession.insert_records_of_one_device(device_id, time_list, measurements_list, data_types_list, values_list)\n```\n\n#### Insert with type inference\n\nWhen the data is of String type, we can use the following interface to perform type inference based on the value of the value itself. For example, if value is \"true\" , it can be automatically inferred to be a boolean type. If value is \"3.2\" , it can be automatically inferred as a flout type. Without type information, server has to do type inference, which may cost some time.\n\n* Insert a Record, which contains multiple measurement value of a device at a timestamp\n\n```python\nsession.insert_str_record(device_id, timestamp, measurements, string_values)\n```\n\n#### Insert of Aligned Timeseries\n\nThe Insert of aligned timeseries uses interfaces like insert_aligned_XXX, and others are similar to the above interfaces:\n\n* insert_aligned_record\n* insert_aligned_records\n* insert_aligned_records_of_one_device\n* insert_aligned_tablet\n* insert_aligned_tablets\n\n\n### IoTDB-SQL Interface\n\n* Execute query statement\n\n```python\nsession.execute_query_statement(sql)\n```\n\n* Execute non query statement\n\n```python\nsession.execute_non_query_statement(sql)\n```\n\n* Execute statement\n\n```python\nsession.execute_statement(sql)\n```\n\n### Device Template\n#### Create Device Template\nThe step for creating a metadata template is as follows\n1. Create the template class\n2. Adding child Node\uff0cInternalNode and MeasurementNode can be chose\n3. Execute create device template function\n\n```python\ntemplate = Template(name=template_name, share_time=True)\n\ni_node_gps = InternalNode(name=\"GPS\", share_time=False)\ni_node_v = InternalNode(name=\"vehicle\", share_time=True)\nm_node_x = MeasurementNode(\"x\", TSDataType.FLOAT, TSEncoding.RLE, Compressor.SNAPPY)\n\ni_node_gps.add_child(m_node_x)\ni_node_v.add_child(m_node_x)\n\ntemplate.add_template(i_node_gps)\ntemplate.add_template(i_node_v)\ntemplate.add_template(m_node_x)\n\nsession.create_schema_template(template)\n```\n#### Modify Device Template nodes\nModify nodes in a template, the template must be already created. These are functions that add or delete some measurement nodes.\n* add node in template\n```python\nsession.add_measurements_in_template(template_name, measurements_path, data_types, encodings, compressors, is_aligned)\n```\n\n* delete node in template\n```python\nsession.delete_node_in_template(template_name, path)\n```\n\n#### Set Device Template\n```python\nsession.set_schema_template(template_name, prefix_path)\n```\n\n#### Uset Device Template\n```python\nsession.unset_schema_template(template_name, prefix_path)\n```\n\n#### Show Device Template\n* Show all device templates\n```python\nsession.show_all_templates()\n```\n* Count all nodes in templates\n```python\nsession.count_measurements_in_template(template_name)\n```\n\n* Judge whether the path is measurement or not in templates, This measurement must be in the template\n```python\nsession.count_measurements_in_template(template_name, path)\n```\n\n* Judge whether the path is exist or not in templates, This path may not belong to the template\n```python\nsession.is_path_exist_in_template(template_name, path)\n```\n\n* Show nodes under in device template\n```python\nsession.show_measurements_in_template(template_name)\n```\n\n* Show the path prefix where a device template is set\n```python\nsession.show_paths_template_set_on(template_name)\n```\n\n* Show the path prefix where a device template is used (i.e. the time series has been created)\n```python\nsession.show_paths_template_using_on(template_name)\n```\n\n#### Drop Device Template\nDelete an existing metadata template\uff0cdropping an already set template is not supported\n```python\nsession.drop_schema_template(\"template_python\")\n```\n\n\n### Pandas Support\n\nTo easily transform a query result to a [Pandas Dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)\nthe SessionDataSet has a method `.todf()` which consumes the dataset and transforms it to a pandas dataframe.\n\nExample:\n\n```python\nfrom iotdb.Session import Session\n\nip = \"127.0.0.1\"\nport_ = \"6667\"\nusername_ = \"root\"\npassword_ = \"root\"\nsession = Session(ip, port_, username_, password_)\nsession.open(False)\nresult = session.execute_query_statement(\"SELECT * FROM root.*\")\n\n# Transform to Pandas Dataset\ndf = result.todf()\n\nsession.close()\n\n# Now you can work with the dataframe\ndf = ...\n```\n\n\n### IoTDB Testcontainer\n\nThe Test Support is based on the lib `testcontainers` (https://testcontainers-python.readthedocs.io/en/latest/index.html) which you need to install in your project if you want to use the feature.\n\nTo start (and stop) an IoTDB Database in a Docker container simply do:\n```python\nclass MyTestCase(unittest.TestCase):\n\n def test_something(self):\n with IoTDBContainer() as c:\n session = Session(\"localhost\", c.get_exposed_port(6667), \"root\", \"root\")\n session.open(False)\n result = session.execute_query_statement(\"SHOW TIMESERIES\")\n print(result)\n session.close()\n```\n\nby default it will load the image `apache/iotdb:latest`, if you want a specific version just pass it like e.g. `IoTDBContainer(\"apache/iotdb:0.12.0\")` to get version `0.12.0` running.\n\n\n### IoTDB DBAPI\n\nIoTDB DBAPI implements the Python DB API 2.0 specification (https://peps.python.org/pep-0249/), which defines a common\ninterface for accessing databases in Python.\n\n#### Examples\n+ Initialization\n\nThe initialized parameters are consistent with the session part (except for the sqlalchemy_mode).\n```python\nfrom iotdb.dbapi import connect\n\nip = \"127.0.0.1\"\nport_ = \"6667\"\nusername_ = \"root\"\npassword_ = \"root\"\nconn = connect(ip, port_, username_, password_,fetch_size=1024,zone_id=\"UTC+8\",sqlalchemy_mode=False)\ncursor = conn.cursor()\n```\n+ simple SQL statement execution\n```python\ncursor.execute(\"SELECT * FROM root.*\")\nfor row in cursor.fetchall():\n print(row)\n```\n\n+ execute SQL with parameter\n\nIoTDB DBAPI supports pyformat style parameters\n```python\ncursor.execute(\"SELECT * FROM root.* WHERE time < %(time)s\",{\"time\":\"2017-11-01T00:08:00.000\"})\nfor row in cursor.fetchall():\n print(row)\n```\n\n+ execute SQL with parameter sequences\n```python\nseq_of_parameters = [\n {\"timestamp\": 1, \"temperature\": 1},\n {\"timestamp\": 2, \"temperature\": 2},\n {\"timestamp\": 3, \"temperature\": 3},\n {\"timestamp\": 4, \"temperature\": 4},\n {\"timestamp\": 5, \"temperature\": 5},\n]\nsql = \"insert into root.cursor(timestamp,temperature) values(%(timestamp)s,%(temperature)s)\"\ncursor.executemany(sql,seq_of_parameters)\n```\n\n+ close the connection and cursor\n```python\ncursor.close()\nconn.close()\n```\n\n### IoTDB SQLAlchemy Dialect (Experimental)\nThe SQLAlchemy dialect of IoTDB is written to adapt to Apache Superset.\nThis part is still being improved.\nPlease do not use it in the production environment!\n#### Mapping of the metadata\nThe data model used by SQLAlchemy is a relational data model, which describes the relationships between different entities through tables.\nWhile the data model of IoTDB is a hierarchical data model, which organizes the data through a tree structure.\nIn order to adapt IoTDB to the dialect of SQLAlchemy, the original data model in IoTDB needs to be reorganized.\nConverting the data model of IoTDB into the data model of SQLAlchemy.\n\nThe metadata in the IoTDB are\uff1a\n\n1. Database\n2. Path\n3. Entity\n4. Measurement\n\nThe metadata in the SQLAlchemy are\uff1a\n1. Schema\n2. Table\n3. Column\n\nThe mapping relationship between them is\uff1a\n\n| The metadata in the SQLAlchemy | The metadata in the IoTDB |\n| -------------------- | ---------------------------------------------- |\n| Schema | Database |\n| Table | Path ( from database to entity ) + Entity |\n| Column | Measurement |\n\nThe following figure shows the relationship between the two more intuitively:\n\n\n\n#### Data type mapping\n| data type in IoTDB | data type in SQLAlchemy |\n|--------------------|-------------------------|\n| BOOLEAN | Boolean |\n| INT32 | Integer |\n| INT64 | BigInteger |\n| FLOAT | Float |\n| DOUBLE | Float |\n| TEXT | Text |\n| LONG | BigInteger |\n#### Example\n\n+ execute statement\n\n```python\nfrom sqlalchemy import create_engine\n\nengine = create_engine(\"iotdb://root:root@127.0.0.1:6667\")\nconnect = engine.connect()\nresult = connect.execute(\"SELECT ** FROM root\")\nfor row in result.fetchall():\n print(row)\n```\n\n+ ORM (now only simple queries are supported)\n\n```python\nfrom sqlalchemy import create_engine, Column, Float, BigInteger, MetaData\nfrom sqlalchemy.ext.declarative import declarative_base\nfrom sqlalchemy.orm import sessionmaker\n\nmetadata = MetaData(\n schema='root.factory'\n)\nBase = declarative_base(metadata=metadata)\n\n\nclass Device(Base):\n __tablename__ = \"room2.device1\"\n Time = Column(BigInteger, primary_key=True)\n temperature = Column(Float)\n status = Column(Float)\n\n\nengine = create_engine(\"iotdb://root:root@127.0.0.1:6667\")\n\nDbSession = sessionmaker(bind=engine)\nsession = DbSession()\n\nres = session.query(Device.status).filter(Device.temperature > 1)\n\nfor row in res:\n print(row)\n```\n\n\n## Developers\n\n### Introduction\n\nThis is an example of how to connect to IoTDB with python, using the thrift rpc interfaces. Things are almost the same on Windows or Linux, but pay attention to the difference like path separator.\n\n\n\n### Prerequisites\n\nPython3.7 or later is preferred.\n\nYou have to install Thrift (0.11.0 or later) to compile our thrift file into python code. Below is the official tutorial of installation, eventually, you should have a thrift executable.\n\n```\nhttp://thrift.apache.org/docs/install/\n```\n\nBefore starting you need to install `requirements_dev.txt` in your python environment, e.g. by calling\n```shell\npip install -r requirements_dev.txt\n```\n\n\n\n### Compile the thrift library and Debug\n\nIn the root of IoTDB's source code folder, run `mvn clean generate-sources -pl client-py -am`.\n\nThis will automatically delete and repopulate the folder `iotdb/thrift` with the generated thrift files.\nThis folder is ignored from git and should **never be pushed to git!**\n\n**Notice** Do not upload `iotdb/thrift` to the git repo.\n\n\n\n\n### Session Client & Example\n\nWe packed up the Thrift interface in `client-py/src/iotdb/Session.py` (similar with its Java counterpart), also provided an example file `client-py/src/SessionExample.py` of how to use the session module. please read it carefully.\n\n\nOr, another simple example:\n\n```python\nfrom iotdb.Session import Session\n\nip = \"127.0.0.1\"\nport_ = \"6667\"\nusername_ = \"root\"\npassword_ = \"root\"\nsession = Session(ip, port_, username_, password_)\nsession.open(False)\nzone = session.get_time_zone()\nsession.close()\n```\n\n\n\n### Tests\n\nPlease add your custom tests in `tests` folder.\n\nTo run all defined tests just type `pytest .` in the root folder.\n\n**Notice** Some tests need docker to be started on your system as a test instance is started in a docker container using [testcontainers](https://testcontainers-python.readthedocs.io/en/latest/index.html).\n\n\n\n### Futher Tools\n\n[black](https://pypi.org/project/black/) and [flake8](https://pypi.org/project/flake8/) are installed for autoformatting and linting.\nBoth can be run by `black .` or `flake8 .` respectively.\n\n\n\n## Releasing\n\nTo do a release just ensure that you have the right set of generated thrift files.\nThen run linting and auto-formatting.\nThen, ensure that all tests work (via `pytest .`).\nThen you are good to go to do a release!\n\n\n\n### Preparing your environment\n\nFirst, install all necessary dev dependencies via `pip install -r requirements_dev.txt`.\n\n\n\n### Doing the Release\n\nThere is a convenient script `release.sh` to do all steps for a release.\nNamely, these are\n\n* Remove all transient directories from last release (if exists)\n* (Re-)generate all generated sources via mvn\n* Run Linting (flake8)\n* Run Tests via pytest\n* Build\n* Release to pypi\n\n\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Apache IoTDB client API",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/apache/iotdb"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f3831292d28c772d573c1c2d191a1bcec031c5ee48fc0e0549ab6e0046142519",
"md5": "83225625b5e0eec2a0af691d48ba1ce4",
"sha256": "b56001b30f216649301eb5ae65ef523146f98089b903446c4cdaa5375686ffb8"
},
"downloads": -1,
"filename": "apache_iotdb-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "83225625b5e0eec2a0af691d48ba1ce4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 127594,
"upload_time": "2024-01-02T01:34:31",
"upload_time_iso_8601": "2024-01-02T01:34:31.347271Z",
"url": "https://files.pythonhosted.org/packages/f3/83/1292d28c772d573c1c2d191a1bcec031c5ee48fc0e0549ab6e0046142519/apache_iotdb-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "980aa5c8bede42a5dcf2f2eadb8b37a406a1ff2fb4e4f78df149d7a63cac058f",
"md5": "147a85ff01f507a95579cfc28a5c3c9b",
"sha256": "2206073dc2e745dc45bd5047b1bf73c0779bc634400809ffef52884413525ef1"
},
"downloads": -1,
"filename": "apache-iotdb-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "147a85ff01f507a95579cfc28a5c3c9b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 88203,
"upload_time": "2024-01-02T01:34:50",
"upload_time_iso_8601": "2024-01-02T01:34:50.237694Z",
"url": "https://files.pythonhosted.org/packages/98/0a/a5c8bede42a5dcf2f2eadb8b37a406a1ff2fb4e4f78df149d7a63cac058f/apache-iotdb-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-02 01:34:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "apache",
"github_project": "iotdb",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "apache-iotdb"
}
JSON
