| Name | kola JSON |
| Version |
1.6.0
JSON |
| download |
| home_page | None |
| Summary | a Python Polars interface to kdb+/q |
| upload_time | 2025-02-15 02:40:57 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | None |
| keywords |
q
kdb
polars
dataframe
arrow
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# kola
a Python [Polars](https://pola-rs.github.io/polars/) Interface to kdb+/q
## Basic Data Type Map
### Deserialization
#### Atom
| k type | n | size | python type | note |
| ----------- | --- | ---- | ----------- | --------------------------- |
| `boolean` | 1 | 1 | `bool` | |
| `guid` | 2 | 16 | `str` | |
| `byte` | 4 | 1 | `int` | |
| `short` | 5 | 2 | `int` | |
| `int` | 6 | 4 | `int` | |
| `long` | 7 | 8 | `int` | |
| `real` | 8 | 4 | `float` | |
| `float` | 9 | 8 | `float` | |
| `char` | 10 | 1 | `str` | |
| `string` | 10 | 1 | `str` | |
| `symbol` | 11 | \* | `str` | |
| `timestamp` | 12 | 8 | `datetime` | |
| `month` | 13 | 4 | `-` | |
| `date` | 14 | 4 | `date` | 0001.01.01 - 9999.12.31 |
| `datetime` | 15 | 8 | `datetime` | |
| `timespan` | 16 | 8 | `timedelta` | |
| `minute` | 17 | 4 | `time` | 00:00 - 23:59 |
| `second` | 18 | 4 | `time` | 00:00:00 - 23:59:59 |
| `time` | 19 | 4 | `time` | 00:00:00.000 - 23:59:59.999 |
#### Composite Data Type
| k type | n | size | python type |
| ---------------- | --- | ---- | ------------------------ |
| `boolean list` | 1 | 1 | `pl.Boolean` |
| `guid list` | 2 | 16 | `pl.List(pl.Binary(16))` |
| `byte list` | 4 | 1 | `pl.Uint8` |
| `short list` | 5 | 2 | `pl.Int16` |
| `int list` | 6 | 4 | `pl.Int32` |
| `long list` | 7 | 8 | `pl.Int64` |
| `real list` | 8 | 4 | `pl.Float32` |
| `float list` | 9 | 8 | `pl.Float64` |
| `char list` | 10 | 1 | `pl.Utf8` |
| `string list` | 10 | 1 | `pl.Utf8` |
| `symbol list` | 11 | \* | `pl.Categorical` |
| `timestamp list` | 12 | 8 | `pl.Datetime` |
| `month list` | 13 | 4 | `-` |
| `date list` | 14 | 4 | `pl.Date` |
| `datetime list` | 15 | 8 | `pl.Datetime` |
| `timespan list` | 16 | 8 | `pl.Duration` |
| `minute list` | 17 | 4 | `pl.Time` |
| `second list` | 18 | 4 | `pl.Time` |
| `time list` | 19 | 4 | `pl.Time` |
| `table` | 98 | \* | `pl.DataFrame` |
| `dictionary` | 99 | \* | `-` |
| `keyed table` | 99 | \* | `pl.DataFrame` |
> performance is impacted by converting guid to string, deserialize the uuid to 16 fixed binary list, use .hex() to convert binary to string if required
> real/float 0n is mapped to Polars null not NaN
> short/int/long 0Nh/i/j, 0Wh/i/j and -0Wh/i/j are mapped to null
```
df.with_columns([
(pl.col("uuid").apply(lambda u: u.hex()))
])
```
### Serialization
#### Basic Data Type
| python type | k type | note |
| ----------- | ----------- | --------------------------- |
| `bool` | `boolean` | |
| `int` | `long` | |
| `float` | `float` | |
| `str` | `symbol` | |
| `bytes` | `string` | |
| `datetime` | `timestamp` | |
| `date` | `date` | 0001.01.01 - 9999.12.31 |
| `datetime` | `datetime` | |
| `timedelta` | `timespan` | |
| `time` | `time` | 00:00:00.000 - 23:59:59.999 |
#### Dictionary, Series and DataFrame
| python type | k type |
| ------------------------ | --------- |
| `dict` | dict |
| `pl.Boolean` | boolean |
| `pl.List(pl.Binary(16))` | guid |
| `pl.Uint8` | byte |
| `pl.Int16` | short |
| `pl.Int32` | int |
| `pl.Int64` | long |
| `pl.Float32` | real |
| `pl.Float64` | float |
| `pl.Utf8` | char |
| `pl.Categorical` | symbol |
| `pl.Datetime` | timestamp |
| `pl.Date` | date |
| `pl.Datetime` | datetime |
| `pl.Duration` | timespan |
| `pl.Time` | time |
| `pl.DataFrame` | table |
> Limited Support for dictionary as arguments, requires `string` as keys.
## Quick Start
### Create a Connection
```python
import polars as pl
import kola
q = kola.Q('localhost', 1800)
# with retries for IO Errors, 1s, 2s, 4s ...
q = kola.Q('localhost', 1800, retries=3)
# with read timeout error, 2s, "Resource temporarily unavailable"
q = kola.Q('localhost', 1800, retries=3, timeout=2)
```
### Connect(Optional)
Automatically connect when querying q process
```python
q.connect()
```
### Disconnect
Automatically disconnect if any IO error
```python
q.disconnect()
```
### String Query
```python
q.sync("select from trade where date=last date")
```
### Lambda Query
When the first string starts with `{` and ends with `}`, it is treated as a lambda.
```python
d = {"a": 1, "b": 2}
q.sync("{key x}", d)
```
### Functional Query
For functional query, `kola` supports Python [Basic Data Type](#basic-data-type), `pl.Series`, `pl.DataFrame` and Python Dictionary with string keys and Python [Basic Data Type](#basic-data-type) and `pl.Series` values.
```python
from datetime import date, time
q.sync(
".gw.query",
"table",
{
"date": date(2023, 11, 21),
"syms": pl.Series("", ["sym0", "sym1"], pl.Categorical),
# 09:00
"startTime": time(9),
# 11:30
"endTime": time(11, 30),
},
)
```
### Send DataFrame
```python
# pl_df is a Polars DataFrame
q.sync("upsert", "table", pl_df)
```
```python
# pd_df is a Pandas DataFrame, use pl.DateFrame to cast Pandas DataFrame
q.sync("upsert", "table", pl.DataFrame(pd_df))
```
### Async Query
```python
# pl_df is a Polars DataFrame
q.asyn("upsert", "table", pl_df)
```
### Subscribe
```python
from kola import QType
q.sync(".u.sub", pl.Series("", ["table1", "table2"], QType.Symbol), "")
# specify symbol filter
q.sync(
".u.sub",
pl.Series("", ["table1", "table2"], QType.Symbol),
pl.Series("", ["sym1", "sym2"], QType.Symbol),
)
while true:
# ("upd", "table", pl.Dataframe)
upd = self.q.receive()
print(upd)
```
### Generate IPC
```python
import polars as pl
from kola import generate_ipc
df = pl.DataFrame(
{
"sym": pl.Series("sym", ["a", "b", "c"], pl.Categorical),
"price": [1, 2, 3],
}
)
# without compression
buffer = generate_ipc("sync", False, ["upd", "table", df])
# with compression
buffer = generate_ipc("sync", True, ["upd", "table", df])
```
## Polars Documentations
Refer to
- [User Guide](https://pola-rs.github.io/polars/user-guide/)
- [API Reference](https://pola-rs.github.io/polars/py-polars/html/reference/index.html)
Raw data
{
"_id": null,
"home_page": null,
"name": "kola",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "q, kdb, polars, dataframe, arrow",
"author": null,
"author_email": "Jo Shinonome <jo.shinonome@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/04/e6/66a0daca1f505c7b90cbd231931e44a159e81aeb09b549f9fb08c8af2fac/kola-1.6.0.tar.gz",
"platform": null,
"description": "# kola\n\na Python [Polars](https://pola-rs.github.io/polars/) Interface to kdb+/q\n\n## Basic Data Type Map\n\n### Deserialization\n\n#### Atom\n\n| k type | n | size | python type | note |\n| ----------- | --- | ---- | ----------- | --------------------------- |\n| `boolean` | 1 | 1 | `bool` | |\n| `guid` | 2 | 16 | `str` | |\n| `byte` | 4 | 1 | `int` | |\n| `short` | 5 | 2 | `int` | |\n| `int` | 6 | 4 | `int` | |\n| `long` | 7 | 8 | `int` | |\n| `real` | 8 | 4 | `float` | |\n| `float` | 9 | 8 | `float` | |\n| `char` | 10 | 1 | `str` | |\n| `string` | 10 | 1 | `str` | |\n| `symbol` | 11 | \\* | `str` | |\n| `timestamp` | 12 | 8 | `datetime` | |\n| `month` | 13 | 4 | `-` | |\n| `date` | 14 | 4 | `date` | 0001.01.01 - 9999.12.31 |\n| `datetime` | 15 | 8 | `datetime` | |\n| `timespan` | 16 | 8 | `timedelta` | |\n| `minute` | 17 | 4 | `time` | 00:00 - 23:59 |\n| `second` | 18 | 4 | `time` | 00:00:00 - 23:59:59 |\n| `time` | 19 | 4 | `time` | 00:00:00.000 - 23:59:59.999 |\n\n#### Composite Data Type\n\n| k type | n | size | python type |\n| ---------------- | --- | ---- | ------------------------ |\n| `boolean list` | 1 | 1 | `pl.Boolean` |\n| `guid list` | 2 | 16 | `pl.List(pl.Binary(16))` |\n| `byte list` | 4 | 1 | `pl.Uint8` |\n| `short list` | 5 | 2 | `pl.Int16` |\n| `int list` | 6 | 4 | `pl.Int32` |\n| `long list` | 7 | 8 | `pl.Int64` |\n| `real list` | 8 | 4 | `pl.Float32` |\n| `float list` | 9 | 8 | `pl.Float64` |\n| `char list` | 10 | 1 | `pl.Utf8` |\n| `string list` | 10 | 1 | `pl.Utf8` |\n| `symbol list` | 11 | \\* | `pl.Categorical` |\n| `timestamp list` | 12 | 8 | `pl.Datetime` |\n| `month list` | 13 | 4 | `-` |\n| `date list` | 14 | 4 | `pl.Date` |\n| `datetime list` | 15 | 8 | `pl.Datetime` |\n| `timespan list` | 16 | 8 | `pl.Duration` |\n| `minute list` | 17 | 4 | `pl.Time` |\n| `second list` | 18 | 4 | `pl.Time` |\n| `time list` | 19 | 4 | `pl.Time` |\n| `table` | 98 | \\* | `pl.DataFrame` |\n| `dictionary` | 99 | \\* | `-` |\n| `keyed table` | 99 | \\* | `pl.DataFrame` |\n\n> performance is impacted by converting guid to string, deserialize the uuid to 16 fixed binary list, use .hex() to convert binary to string if required\n\n> real/float 0n is mapped to Polars null not NaN\n\n> short/int/long 0Nh/i/j, 0Wh/i/j and -0Wh/i/j are mapped to null\n\n```\ndf.with_columns([\n (pl.col(\"uuid\").apply(lambda u: u.hex()))\n ])\n```\n\n### Serialization\n\n#### Basic Data Type\n\n| python type | k type | note |\n| ----------- | ----------- | --------------------------- |\n| `bool` | `boolean` | |\n| `int` | `long` | |\n| `float` | `float` | |\n| `str` | `symbol` | |\n| `bytes` | `string` | |\n| `datetime` | `timestamp` | |\n| `date` | `date` | 0001.01.01 - 9999.12.31 |\n| `datetime` | `datetime` | |\n| `timedelta` | `timespan` | |\n| `time` | `time` | 00:00:00.000 - 23:59:59.999 |\n\n#### Dictionary, Series and DataFrame\n\n| python type | k type |\n| ------------------------ | --------- |\n| `dict` | dict |\n| `pl.Boolean` | boolean |\n| `pl.List(pl.Binary(16))` | guid |\n| `pl.Uint8` | byte |\n| `pl.Int16` | short |\n| `pl.Int32` | int |\n| `pl.Int64` | long |\n| `pl.Float32` | real |\n| `pl.Float64` | float |\n| `pl.Utf8` | char |\n| `pl.Categorical` | symbol |\n| `pl.Datetime` | timestamp |\n| `pl.Date` | date |\n| `pl.Datetime` | datetime |\n| `pl.Duration` | timespan |\n| `pl.Time` | time |\n| `pl.DataFrame` | table |\n\n> Limited Support for dictionary as arguments, requires `string` as keys.\n\n## Quick Start\n\n### Create a Connection\n\n```python\nimport polars as pl\nimport kola\nq = kola.Q('localhost', 1800)\n\n# with retries for IO Errors, 1s, 2s, 4s ...\nq = kola.Q('localhost', 1800, retries=3)\n\n# with read timeout error, 2s, \"Resource temporarily unavailable\"\nq = kola.Q('localhost', 1800, retries=3, timeout=2)\n```\n\n### Connect(Optional)\n\nAutomatically connect when querying q process\n\n```python\nq.connect()\n```\n\n### Disconnect\n\nAutomatically disconnect if any IO error\n\n```python\nq.disconnect()\n```\n\n### String Query\n\n```python\nq.sync(\"select from trade where date=last date\")\n```\n\n### Lambda Query\n\nWhen the first string starts with `{` and ends with `}`, it is treated as a lambda.\n\n```python\nd = {\"a\": 1, \"b\": 2}\nq.sync(\"{key x}\", d)\n```\n\n### Functional Query\n\nFor functional query, `kola` supports Python [Basic Data Type](#basic-data-type), `pl.Series`, `pl.DataFrame` and Python Dictionary with string keys and Python [Basic Data Type](#basic-data-type) and `pl.Series` values.\n\n```python\nfrom datetime import date, time\n\nq.sync(\n \".gw.query\",\n \"table\",\n {\n \"date\": date(2023, 11, 21),\n \"syms\": pl.Series(\"\", [\"sym0\", \"sym1\"], pl.Categorical),\n # 09:00\n \"startTime\": time(9),\n # 11:30\n \"endTime\": time(11, 30),\n },\n)\n```\n\n### Send DataFrame\n\n```python\n# pl_df is a Polars DataFrame\nq.sync(\"upsert\", \"table\", pl_df)\n```\n\n```python\n# pd_df is a Pandas DataFrame, use pl.DateFrame to cast Pandas DataFrame\nq.sync(\"upsert\", \"table\", pl.DataFrame(pd_df))\n```\n\n### Async Query\n\n```python\n# pl_df is a Polars DataFrame\nq.asyn(\"upsert\", \"table\", pl_df)\n```\n\n### Subscribe\n\n```python\nfrom kola import QType\n\nq.sync(\".u.sub\", pl.Series(\"\", [\"table1\", \"table2\"], QType.Symbol), \"\")\n\n# specify symbol filter\nq.sync(\n \".u.sub\",\n pl.Series(\"\", [\"table1\", \"table2\"], QType.Symbol),\n pl.Series(\"\", [\"sym1\", \"sym2\"], QType.Symbol),\n)\n\nwhile true:\n # (\"upd\", \"table\", pl.Dataframe)\n upd = self.q.receive()\n print(upd)\n```\n\n### Generate IPC\n\n```python\nimport polars as pl\nfrom kola import generate_ipc\n\ndf = pl.DataFrame(\n {\n \"sym\": pl.Series(\"sym\", [\"a\", \"b\", \"c\"], pl.Categorical),\n \"price\": [1, 2, 3],\n }\n)\n# without compression\nbuffer = generate_ipc(\"sync\", False, [\"upd\", \"table\", df])\n\n# with compression\nbuffer = generate_ipc(\"sync\", True, [\"upd\", \"table\", df])\n```\n\n## Polars Documentations\n\nRefer to\n\n- [User Guide](https://pola-rs.github.io/polars/user-guide/)\n- [API Reference](https://pola-rs.github.io/polars/py-polars/html/reference/index.html)\n\n",
"bugtrack_url": null,
"license": null,
"summary": "a Python Polars interface to kdb+/q",
"version": "1.6.0",
"project_urls": {
"Repository": "https://github.com/jshinonome/kola"
},
"split_keywords": [
"q",
" kdb",
" polars",
" dataframe",
" arrow"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4b650f7e618f462e05cb6b5c912deffd8ef1129527084cc79fc2e431bf48bdde",
"md5": "810990aa4382b5c280d59df4fe3c3b8e",
"sha256": "16ba0f3760286f36c258f91646e7fcf92f03d4e940b52e23dfba1f1d2f0408f1"
},
"downloads": -1,
"filename": "kola-1.6.0-cp310-cp310-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "810990aa4382b5c280d59df4fe3c3b8e",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 7200599,
"upload_time": "2025-02-15T02:40:41",
"upload_time_iso_8601": "2025-02-15T02:40:41.651716Z",
"url": "https://files.pythonhosted.org/packages/4b/65/0f7e618f462e05cb6b5c912deffd8ef1129527084cc79fc2e431bf48bdde/kola-1.6.0-cp310-cp310-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "700bb80b9b97ec7ca916e09f7d402b7d6e0336c31a3602824df28e2a65e6ddfa",
"md5": "44fe51609420389dad6e0b41e92f73c6",
"sha256": "f28cfb5ac4f13fc12dfafa19c3189e4509403144dd0477fd7366f41fbb83218b"
},
"downloads": -1,
"filename": "kola-1.6.0-cp311-cp311-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "44fe51609420389dad6e0b41e92f73c6",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 7200001,
"upload_time": "2025-02-15T02:40:44",
"upload_time_iso_8601": "2025-02-15T02:40:44.981173Z",
"url": "https://files.pythonhosted.org/packages/70/0b/b80b9b97ec7ca916e09f7d402b7d6e0336c31a3602824df28e2a65e6ddfa/kola-1.6.0-cp311-cp311-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c8d167065952769e493a82382f15a71fbe9f3f46e9ccf1009645fec61011af08",
"md5": "27658bd101ce0a284b54e9e78b9c9961",
"sha256": "70550ed62290b9b7e01d58945f07a00ea114e829b1e76aa2c493b6be54684a26"
},
"downloads": -1,
"filename": "kola-1.6.0-cp312-cp312-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "27658bd101ce0a284b54e9e78b9c9961",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 7199168,
"upload_time": "2025-02-15T02:40:47",
"upload_time_iso_8601": "2025-02-15T02:40:47.838629Z",
"url": "https://files.pythonhosted.org/packages/c8/d1/67065952769e493a82382f15a71fbe9f3f46e9ccf1009645fec61011af08/kola-1.6.0-cp312-cp312-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "696e3532da1b609bd804b86d6b7a59451e34756836a476ddd80e3ab5c6de1221",
"md5": "cb59a688e3c1f2e59e4ff9dbb6516e6f",
"sha256": "4ecfd2d9d9345a0b7930db2312fc88662da74bc81f3eab97c29e65c8c2897815"
},
"downloads": -1,
"filename": "kola-1.6.0-cp313-cp313-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "cb59a688e3c1f2e59e4ff9dbb6516e6f",
"packagetype": "bdist_wheel",
"python_version": "cp313",
"requires_python": ">=3.9",
"size": 7199354,
"upload_time": "2025-02-15T02:40:50",
"upload_time_iso_8601": "2025-02-15T02:40:50.707245Z",
"url": "https://files.pythonhosted.org/packages/69/6e/3532da1b609bd804b86d6b7a59451e34756836a476ddd80e3ab5c6de1221/kola-1.6.0-cp313-cp313-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7abbf18f6f741dff797daeb04c4e6daafbe98470ed13fe945d0eea836a7fcd6d",
"md5": "86204ac5ebd1ea7b93297f3ce27cac41",
"sha256": "127bb85502bc0806be52694cd532be0124b3fe869b3971f3e12b3078a4ee85ec"
},
"downloads": -1,
"filename": "kola-1.6.0-cp39-cp39-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "86204ac5ebd1ea7b93297f3ce27cac41",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 7202122,
"upload_time": "2025-02-15T02:40:54",
"upload_time_iso_8601": "2025-02-15T02:40:54.646241Z",
"url": "https://files.pythonhosted.org/packages/7a/bb/f18f6f741dff797daeb04c4e6daafbe98470ed13fe945d0eea836a7fcd6d/kola-1.6.0-cp39-cp39-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "04e666a0daca1f505c7b90cbd231931e44a159e81aeb09b549f9fb08c8af2fac",
"md5": "65069912a62614b110b02e7e454ce5c9",
"sha256": "602340d274a6ae9557145a922873259a0309fea23b1ee477d2cc62e8109cb200"
},
"downloads": -1,
"filename": "kola-1.6.0.tar.gz",
"has_sig": false,
"md5_digest": "65069912a62614b110b02e7e454ce5c9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 52493,
"upload_time": "2025-02-15T02:40:57",
"upload_time_iso_8601": "2025-02-15T02:40:57.503816Z",
"url": "https://files.pythonhosted.org/packages/04/e6/66a0daca1f505c7b90cbd231931e44a159e81aeb09b549f9fb08c8af2fac/kola-1.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-15 02:40:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jshinonome",
"github_project": "kola",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kola"
}
