| Name | dve-lumipy-testing JSON |
| Version |
1.0.289
JSON |
| download |
| home_page | None |
| Summary | Python library for Luminesce |
| upload_time | 2024-12-04 22:01:49 |
| maintainer | None |
| docs_url | None |
| author | FINBOURNE Technology |
| requires_python | >=3.6 |
| license | None |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Lumipy
[*loom-ee-pie*]
## Introduction
Lumipy is a python library that integrates Luminesce and the Python Data Science Stack.
It is designed to be used in Jupyter, but you can use it scripts and modules as well.
It has two components
* **Getting data:** a fluent syntax for scripting up queries using python code. This makes it easy to build complex queries and get your data back as pandas DataFrames.
* **Integration:** infrastructure to build providers in python. This allows you to build data sources and transforms such as ML models and connect them to Luminesce. They can then be used by other users from the Web UI, Power BI, etc.
Lumipy is designed to be as easy to use and as unobtrusive as possible.
You should have to do minimal imports and everything should be explorable from Jupyter through tab completion and `shift + tab`.
## Install
Lumipy is available from PyPI:
**LumiPy** Is our latest package which utilises the V2 Finbourne SDKs.
It is important to uninstall `dve-lumipy-preview` before installing `lumipy`. You can do this by running:
```
pip uninstall dve-lumipy-preview
```
We recommend using the --force-reinstall option to make this transition smoother. Please note that this will force
update all dependencies for lumipy and could affect your other Python projects.
```
pip install --force-reinstall lumipy
```
If you prefer not to update all dependencies, you can omit the `--force-reinstall` and use the regular pip install
command instead:
```
pip install lumipy
```
**Dve-Lumipy-Preview** uses the V1 Finbourne SDKs and is no longer maintained.
```
pip install dve-lumipy-preview
```
## Configure
Add a personal access token to your config. This first one will be the active one.
```python
import lumipy as lm
lm.config.add('fbn-prd', '<your PAT>')
```
If you add another domain and PAT you will need to switch to it.
```python
import lumipy as lm
lm.config.add('fbn-ci', '<your PAT>')
lm.config.domain = 'fbn-ci'
```
## Query
All built around the atlas object. This is the starting point for exploring your data sources and then using them.
Build your atlas with `lm.get_atlas`. If you don't supply credentials it will default to your active domain in the config.
If there is no active domain in your config it will fall back to env vars.
```python
import lumipy as lm
atlas = lm.get_atlas()
ins = atlas.lusid_instrument()
ins.select('^').limit(10).go()
```
You can also specify the domain here by a positional argument, e.g. `lm.get_atlas('fbn-ci')` will use `fbn-ci` and will override the active domain.
Client objects are created in the same way. You can submit raw SQL strings as queries using `run()`
```python
import lumipy as lm
client = lm.get_client()
client.run('select ^ from lusid.instrument limit 10')
```
You can create a `client` or `atlas` for a domain other than the active one by specifying it in `get_client` or `get_atlas`.
```python
import lumipy as lm
client = lm.get_client('fbn-prd')
atlas = lm.get_atlas('fbn-prd')
```
## Connect
Python providers are build by inheriting from a base class, `BaseProvider`, and implementing the `__init__` and `get_data` methods.
The former defines the 'shape' of the output data and the parameters it takes. The latter is where the provider actually does something.
This can be whatever you want as long as it returns a dataframe with the declared columns.
### Running Providers
This will run the required setup on the first startup.
Once that's finished it'll spin up a provider that returns Fisher's Irises dataset.
Try it out from the web GUI, or from an atlas in another notebook. Remember to get the atlas again once it's finished starting up.
This uses the built-in `PandasProvider` class to make a provider object, adds it to a `ProviderManager` and then starts it.
```python
import lumipy.provider as lp
p = lp.PandasProvider(
'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv',
'iris'
)
lp.ProviderManager(p).run()
```
This will also default to the active domain if none is specified as an argument to the provider manager.
You can run globally in the domain so other users can query your provider by setting `user='global'` and `whitelist_me=True` in
the `ProviderManager` constructor.
The setup consists of getting the dlls for the dotnet app (provider factory) and getting the pem files to run in the domain.
To run the setup on its own run the `lp.setup()` function. This takes the same arguments as `get_client` and `get_atlas`.
### Building Providers
The following example will simulate a set of coin flips. It has two columns `Label` and `Result`, and one
parameter `Probability` with a default value of 0.5.
Its name and column/param content are specified in `__init__`. The simulation of the coin flips
happens inside `get_data` where we draw numbers from a binomial distribution with the given probability and n = 1.
We also have a check for the probability value. If it's out of range an error will be thrown in python and reported back
in the progress log and query status.
Finally, the provider object is instantiated and given to a provider manager. The provider manager is then started up with
the `run()` method.
```python
import lumipy.provider as lp
from pandas import DataFrame
from typing import Union, Iterator
import numpy as np
class CoinFlips(lp.BaseProvider):
def __init__(self):
columns = [
lp.ColumnMeta('Label', lp.DType.Text),
lp.ColumnMeta('Result', lp.DType.Int),
]
params = [lp.ParamMeta('Probability', lp.DType.Double, default_value=0.5)]
super().__init__('test.coin.flips', columns, params)
def get_data(self, context) -> Union[DataFrame, Iterator[DataFrame]]:
# If no limit is given, default to 100 rows.
limit = context.limit()
if limit is None:
limit = 100
# Get param value from params dict. If it's out of bounds throw an error.
p = context.get('Probability')
if not 0 <= p <= 1:
raise ValueError(f'Probability must be between 0 and 1. Was {p}.')
# Generate the coin flips and return.
return DataFrame({'Label':f'Flip {i}', 'Result': np.random.binomial(1, p)} for i in range(limit))
coin_flips = CoinFlips()
lp.ProviderManager(coin_flips).run()
```
# CLI
Lumipy also contains a command line interface (CLI) app with five different functions.
You can view help for the CLI and each of the actions using `--help`. Try this to start with
```shell
$ lumipy --help
```
## Config
This lets you configure your domains and PATs. You can show, add, set, delete and deactivate domains.
To see all options and args run the following
```shell
$ lumipy config --help
```
### Config Examples
`set` Set a domain as the active one.
```shell
$ lumipy config set --domain=my-domain
```
`add` Add a domain and PAT to your config.
```shell
$ lumipy config add --domain=my-domain --token=<my token>
(--overwrite)
```
`show` Show a censored view of the config contents.
```shell
$ lumipy config show
```
`delete` Delete a domain from the config.
```shell
$ lumipy config delete --domain=my-domain
```
`deactivate` Deactivate the config so no domain is used by default.
```shell
$ lumipy config deactivate
```
## Run
This lets you run python providers. You can run prebuilt named sets, CSV files, python files containing provider objects,
or even a directory containing CSVs and py files.
### Run Examples
`.py` File
```shell
$ lumipy run path/to/my_providers.py
```
`.csv` File
```shell
$ lumipy run path/to/my_data.csv
```
Built-in Set
```shell
$ lumipy run demo
```
Directory
```shell
$ lumipy run path/to/dir
```
## Query
This command runs a SQL query, gets the result back, shows it on screen and then saves it as a CSV.
### Query Examples
Run a query (saves as CSV to a temp directory).
```shell
$ lumipy query --sql="select ^ from lusid.instrument limit 5"
```
Run a query to a defined location.
```shell
$ lumipy query --sql="select ^ from lusid.instrument limit 5" --save-to=/path/to/output.csv
```
## Setup
This lets you run the provider infrastructure setup on your machine.
### Setup Examples
Run the py providers setup. This will redownload the certs and get the latest dlls, overwriting any that are already there.
```shell
$ lumipy setup --domain=my-domain
```
## Test
This lets you run the Lumipy test suites.
### Test Examples
You can run `unit` tests, `integration` tests, `provider` tests, or `everything`.
```shell
$ lumipy test unit
```
## Windows Setup
To use LumiPy and run local providers it is recommended that you use an admin `powershell` terminal.
Install (or update) LumiPy using your `powerhsell` terminal.
### LumiPy (V2 Finbourne SDK)
```shell
$ pip install lumipy --upgrade
```
Verify that your install was succesful.
```shell
$ lumipy --help
```
Setup your config with a personal access token (PAT).
```shell
$ lumipy config add --domain=my-domain --token=my-pat-token
```
Ensure you can run local providers. To run these providers globally add `--user==global` and `--whitelist-me` to the command below.
```shell
$ lumipy run demo
```
### Testing Local Changes on Windows
To test your local `dve-lumipy` changes on Windows add `dve-lumipy` to your python path (inside your environment variables).
## Authenticating with the SDK (Lumipy)
Example using the `lumipy.client.get_client()` method:
```python
from lumipy.client import get_client
client = get_client()
```
### Recommended Method
Authenticate by setting up the PAT token via the CLI or directly in Python (see the Configure section above).
### Secrets File
Initialize `get_client` using a secrets file:
```python
client = get_client(api_secrets_file="secrets_file_path/secrets.json")
```
File structure should be:
```json
{
"api": {
"luminesceUrl": "https://fbn-ci.lusid.com/honeycomb/",
"clientId": "clientId",
"clientSecret": "clientSecret",
"appName": "appName",
"certificateFilename": "test_certificate.pem",
"accessToken": "personal access token"
},
"proxy": {
"address": "http://myproxy.com:8080",
"username": "proxyuser",
"password": "proxypass"
}
}
```
### Keyword Arguments
Initialize `get_client` with keyword arguments:
```python
client = get_client(username="myusername", ...)
```
Relevant keyword arguments include:
- token_url
- api_url
- username
- password
- client_id
- client_secret
- app_name
- certificate_filename
- proxy_address
- proxy_username
- proxy_password
- access_token
### Environment Variables
The following environment variables can also be set:
- FBN_TOKEN_URL
- FBN_LUMINESCE_API_URL
- FBN_USERNAME
- FBN_PASSWORD
- FBN_CLIENT_ID
- FBN_CLIENT_SECRET
- FBN_APP_NAME
- FBN_CLIENT_CERTIFICATE
- FBN_PROXY_ADDRESS
- FBN_PROXY_USERNAME
- FBN_PROXY_PASSWORD
- FBN_ACCESS_TOKEN
Raw data
{
"_id": null,
"home_page": null,
"name": "dve-lumipy-testing",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "FINBOURNE Technology",
"author_email": "engineering@finbourne.com",
"download_url": null,
"platform": null,
"description": "# Lumipy\n[*loom-ee-pie*]\n\n## Introduction\nLumipy is a python library that integrates Luminesce and the Python Data Science Stack. \nIt is designed to be used in Jupyter, but you can use it scripts and modules as well.\n\nIt has two components\n * **Getting data:** a fluent syntax for scripting up queries using python code. This makes it easy to build complex queries and get your data back as pandas DataFrames.\n * **Integration:** infrastructure to build providers in python. This allows you to build data sources and transforms such as ML models and connect them to Luminesce. They can then be used by other users from the Web UI, Power BI, etc.\n\nLumipy is designed to be as easy to use and as unobtrusive as possible. \nYou should have to do minimal imports and everything should be explorable from Jupyter through tab completion and `shift + tab`. \n\n\n## Install\nLumipy is available from PyPI:\n\n**LumiPy** Is our latest package which utilises the V2 Finbourne SDKs.\n\nIt is important to uninstall `dve-lumipy-preview` before installing `lumipy`. You can do this by running:\n\n```\npip uninstall dve-lumipy-preview\n```\n\nWe recommend using the --force-reinstall option to make this transition smoother. Please note that this will force \nupdate all dependencies for lumipy and could affect your other Python projects. \n\n```\npip install --force-reinstall lumipy\n```\n\nIf you prefer not to update all dependencies, you can omit the `--force-reinstall` and use the regular pip install \ncommand instead:\n\n```\npip install lumipy\n```\n\n**Dve-Lumipy-Preview** uses the V1 Finbourne SDKs and is no longer maintained.\n```\npip install dve-lumipy-preview\n```\n\n## Configure\nAdd a personal access token to your config. This first one will be the active one.\n```python\nimport lumipy as lm\n\nlm.config.add('fbn-prd', '<your PAT>')\n```\nIf you add another domain and PAT you will need to switch to it.\n```python\nimport lumipy as lm\n\nlm.config.add('fbn-ci', '<your PAT>')\nlm.config.domain = 'fbn-ci'\n```\n\n## Query\nAll built around the atlas object. This is the starting point for exploring your data sources and then using them.\n\nBuild your atlas with `lm.get_atlas`. If you don't supply credentials it will default to your active domain in the config. \nIf there is no active domain in your config it will fall back to env vars. \n```python\nimport lumipy as lm\n\natlas = lm.get_atlas()\n\nins = atlas.lusid_instrument()\nins.select('^').limit(10).go()\n```\nYou can also specify the domain here by a positional argument, e.g. `lm.get_atlas('fbn-ci')` will use `fbn-ci` and will override the active domain. \n\nClient objects are created in the same way. You can submit raw SQL strings as queries using `run()`\n```python\nimport lumipy as lm\n\nclient = lm.get_client()\nclient.run('select ^ from lusid.instrument limit 10')\n```\n\nYou can create a `client` or `atlas` for a domain other than the active one by specifying it in `get_client` or `get_atlas`. \n```python\nimport lumipy as lm\n\nclient = lm.get_client('fbn-prd')\natlas = lm.get_atlas('fbn-prd')\n```\n\n## Connect\n\nPython providers are build by inheriting from a base class, `BaseProvider`, and implementing the `__init__` and `get_data` methods.\nThe former defines the 'shape' of the output data and the parameters it takes. The latter is where the provider actually does something. \nThis can be whatever you want as long as it returns a dataframe with the declared columns. \n\n### Running Providers\n\nThis will run the required setup on the first startup. \nOnce that's finished it'll spin up a provider that returns Fisher's Irises dataset. \nTry it out from the web GUI, or from an atlas in another notebook. Remember to get the atlas again once it's finished starting up. \n\nThis uses the built-in `PandasProvider` class to make a provider object, adds it to a `ProviderManager` and then starts it.\n\n```python\nimport lumipy.provider as lp\n\np = lp.PandasProvider(\n 'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv',\n 'iris'\n)\nlp.ProviderManager(p).run()\n```\n\nThis will also default to the active domain if none is specified as an argument to the provider manager. \n\nYou can run globally in the domain so other users can query your provider by setting `user='global'` and `whitelist_me=True` in\nthe `ProviderManager` constructor. \n\nThe setup consists of getting the dlls for the dotnet app (provider factory) and getting the pem files to run in the domain. \nTo run the setup on its own run the `lp.setup()` function. This takes the same arguments as `get_client` and `get_atlas`.\n\n### Building Providers\n\nThe following example will simulate a set of coin flips. It has two columns `Label` and `Result`, and one \nparameter `Probability` with a default value of 0.5. \n\nIts name and column/param content are specified in `__init__`. The simulation of the coin flips\nhappens inside `get_data` where we draw numbers from a binomial distribution with the given probability and n = 1.\n\nWe also have a check for the probability value. If it's out of range an error will be thrown in python and reported back\nin the progress log and query status. \n\nFinally, the provider object is instantiated and given to a provider manager. The provider manager is then started up with\nthe `run()` method. \n\n```python\nimport lumipy.provider as lp\nfrom pandas import DataFrame\nfrom typing import Union, Iterator\nimport numpy as np\n\n\nclass CoinFlips(lp.BaseProvider):\n\n def __init__(self):\n\n columns = [\n lp.ColumnMeta('Label', lp.DType.Text),\n lp.ColumnMeta('Result', lp.DType.Int),\n ]\n\n params = [lp.ParamMeta('Probability', lp.DType.Double, default_value=0.5)]\n\n super().__init__('test.coin.flips', columns, params)\n\n def get_data(self, context) -> Union[DataFrame, Iterator[DataFrame]]:\n\n # If no limit is given, default to 100 rows.\n limit = context.limit()\n if limit is None:\n limit = 100\n\n # Get param value from params dict. If it's out of bounds throw an error. \n p = context.get('Probability')\n if not 0 <= p <= 1:\n raise ValueError(f'Probability must be between 0 and 1. Was {p}.')\n\n # Generate the coin flips and return. \n return DataFrame({'Label':f'Flip {i}', 'Result': np.random.binomial(1, p)} for i in range(limit))\n\n\ncoin_flips = CoinFlips()\n\nlp.ProviderManager(coin_flips).run()\n```\n\n# CLI\nLumipy also contains a command line interface (CLI) app with five different functions. \nYou can view help for the CLI and each of the actions using `--help`. Try this to start with\n```shell\n $ lumipy --help\n```\n\n## Config\nThis lets you configure your domains and PATs. You can show, add, set, delete and deactivate domains.\nTo see all options and args run the following\n```shell\n $ lumipy config --help\n```\n### Config Examples\n`set` Set a domain as the active one.\n```shell\n $ lumipy config set --domain=my-domain\n```\n\n`add` Add a domain and PAT to your config.\n```shell\n $ lumipy config add --domain=my-domain --token=<my token>\n (--overwrite)\n```\n\n`show` Show a censored view of the config contents.\n```shell\n $ lumipy config show\n```\n\n`delete` Delete a domain from the config.\n```shell\n $ lumipy config delete --domain=my-domain\n```\n\n`deactivate` Deactivate the config so no domain is used by default.\n```shell\n $ lumipy config deactivate\n```\n\n## Run\nThis lets you run python providers. You can run prebuilt named sets, CSV files, python files containing provider objects, \nor even a directory containing CSVs and py files. \n\n### Run Examples\n\n`.py` File\n```shell\n $ lumipy run path/to/my_providers.py\n```\n`.csv` File\n```shell\n $ lumipy run path/to/my_data.csv \n```\nBuilt-in Set\n```shell\n $ lumipy run demo\n```\nDirectory\n```shell\n $ lumipy run path/to/dir\n```\n\n\n## Query\nThis command runs a SQL query, gets the result back, shows it on screen and then saves it as a CSV.\n\n### Query Examples\nRun a query (saves as CSV to a temp directory).\n```shell\n $ lumipy query --sql=\"select ^ from lusid.instrument limit 5\"\n```\nRun a query to a defined location.\n```shell\n $ lumipy query --sql=\"select ^ from lusid.instrument limit 5\" --save-to=/path/to/output.csv\n```\n\n## Setup\nThis lets you run the provider infrastructure setup on your machine. \n\n### Setup Examples\nRun the py providers setup. This will redownload the certs and get the latest dlls, overwriting any that are already there.\n```shell\n $ lumipy setup --domain=my-domain\n```\n\n## Test\nThis lets you run the Lumipy test suites.\n\n### Test Examples\nYou can run `unit` tests, `integration` tests, `provider` tests, or `everything`.\n```shell\n $ lumipy test unit\n```\n\n## Windows Setup\n\nTo use LumiPy and run local providers it is recommended that you use an admin `powershell` terminal.\n\nInstall (or update) LumiPy using your `powerhsell` terminal.\n\n\n### LumiPy (V2 Finbourne SDK)\n```shell\n $ pip install lumipy --upgrade\n```\n\nVerify that your install was succesful.\n\n```shell\n $ lumipy --help\n```\n\nSetup your config with a personal access token (PAT).\n\n```shell\n $ lumipy config add --domain=my-domain --token=my-pat-token\n```\n\nEnsure you can run local providers. To run these providers globally add `--user==global` and `--whitelist-me` to the command below.\n\n```shell\n $ lumipy run demo\n```\n\n### Testing Local Changes on Windows\n\nTo test your local `dve-lumipy` changes on Windows add `dve-lumipy` to your python path (inside your environment variables).\n\n## Authenticating with the SDK (Lumipy)\nExample using the `lumipy.client.get_client()` method:\n```python\nfrom lumipy.client import get_client\nclient = get_client()\n```\n\n### Recommended Method\n\nAuthenticate by setting up the PAT token via the CLI or directly in Python (see the Configure section above).\n\n### Secrets File\nInitialize `get_client` using a secrets file:\n```python\nclient = get_client(api_secrets_file=\"secrets_file_path/secrets.json\")\n```\nFile structure should be:\n```json\n{\n \"api\": {\n \"luminesceUrl\": \"https://fbn-ci.lusid.com/honeycomb/\",\n \"clientId\": \"clientId\",\n \"clientSecret\": \"clientSecret\",\n \"appName\": \"appName\",\n \"certificateFilename\": \"test_certificate.pem\",\n \"accessToken\": \"personal access token\"\n },\n \"proxy\": {\n \"address\": \"http://myproxy.com:8080\",\n \"username\": \"proxyuser\",\n \"password\": \"proxypass\"\n }\n}\n```\n\n### Keyword Arguments\nInitialize `get_client` with keyword arguments:\n```python\nclient = get_client(username=\"myusername\", ...)\n```\nRelevant keyword arguments include:\n- token_url\n- api_url\n- username\n- password\n- client_id\n- client_secret\n- app_name\n- certificate_filename\n- proxy_address\n- proxy_username\n- proxy_password\n- access_token\n\n### Environment Variables\nThe following environment variables can also be set:\n- FBN_TOKEN_URL\n- FBN_LUMINESCE_API_URL\n- FBN_USERNAME\n- FBN_PASSWORD\n- FBN_CLIENT_ID\n- FBN_CLIENT_SECRET\n- FBN_APP_NAME\n- FBN_CLIENT_CERTIFICATE\n- FBN_PROXY_ADDRESS\n- FBN_PROXY_USERNAME\n- FBN_PROXY_PASSWORD\n- FBN_ACCESS_TOKEN\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Python library for Luminesce",
"version": "1.0.289",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5bbc54d749c8b5ffed3caf0164e56bc29f4a5b5195af48d5a364eb3d692b2224",
"md5": "7ca173f2cd60d7fbcb57d2b977c5040f",
"sha256": "9b36bd9cc2c44430a1819054a0b701db2c1533e03c61a9b454bef2f274dd13ce"
},
"downloads": -1,
"filename": "dve_lumipy_testing-1.0.289-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7ca173f2cd60d7fbcb57d2b977c5040f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 806134,
"upload_time": "2024-12-04T22:01:49",
"upload_time_iso_8601": "2024-12-04T22:01:49.605363Z",
"url": "https://files.pythonhosted.org/packages/5b/bc/54d749c8b5ffed3caf0164e56bc29f4a5b5195af48d5a364eb3d692b2224/dve_lumipy_testing-1.0.289-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-04 22:01:49",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "dve-lumipy-testing"
}
