# apache-airflow-providers-sktvane
- AIDP 가 제공하는 자원들에 접근하는 용도
- `NES`
- `BigQuery`
- `Vault`
- 기타 공용 목적의 코드
## PyPI
- https://pypi.org/project/apache-airflow-providers-sktvane
## Environments
### Local
* `VAULT_TOKEN` 은 [관련 문서](https://www.notion.so/ai-data-engineering/Public-3ec3cd8a4e444aa38a6f02fe57e3b6bd?pvs=4#c0b08e2b147f4e0383c67cedadbb0bdc) 에서 확인
```shell
export VAULT_ADDR=https://vault-public.sktai.io
export VAULT_TOKEN={{VAULT_TOKEN}}
export AIRFLOW__CORE__DAGS_FOLDER=.
```
## Deployment
* `main` 브랜치에 `push` 이벤트 발생 시 배포, 부득이하게 로컬 환경에서 배포할 경우 아래 명령 수행
```shell
# build
$ python setup.py sdist bdist_wheel
# upload
$ twine upload dist/*
# remove
$ rm -rf build dist apache_airflow_providers_sktvane.egg-info
```
## Components
###### Operators
- `airflow.providers.sktvane.operators.nes.NesOperator` : AIDP 의 `NES` 사용
```python
from airflow.providers.sktvane.operators.nes import NesOperator
...
NesOperator(
task_id="jupyter_daily_count",
input_nb="https://github.com/sktaiflow/notebooks/blob/master/statistics/jupyter_daily_count.ipynb",
parameters={"current_date": "{{ ds }}", "channel": "#aim-statistics"},
)
```
###### Sensors
- `airflow.providers.sktvane.sensors.gcp.BigqueryPartitionSensor` : AIDP 의 `BigQuery` 파티션 체크
```python
from airflow.providers.sktvane.sensors.gcp import BigqueryPartitionSensor
...
BigqueryPartitionSensor(
task_id=f"{table}_partition_sensor",
dataset_id="wind_tmt",
table_id=table,
partition="dt = '{{ds}}'",
)
```
###### Macros
- `airflow.providers.sktvane.macros.slack.send_fail_message` : AIDP 정의 포맷으로 `Slack` 에러 메시지 발송
```python
from airflow.providers.sktvane.macros.slack import send_fail_message
...
def send_aidp_fail_message(slack_email: str) -> None:
send_fail_message(
slack_channel="#aidp-airflow-monitoring",
slack_username=f"Airflow-AlarmBot-{env}",
slack_email=slack_email,
)
```
- `airflow.providers.sktvane.macros.gcp.bigquery_client` : AIDP 의 `BigQuery` 사용
```python
from airflow.providers.sktvane.macros.gcp import bigquery_client
...
def bq_query_to_bq(query, dest_table_name, **kwarg):
bq_client = bigquery_client()
job = bq_client.query(query)
job.result()
```
- `airflow.providers.sktvane.macros.vault.get_secrets` : AIDP 의 `Vault` 사용
```python
from airflow.providers.sktvane.macros.vault import get_secrets
...
def get_hive_conn():
from pyhive import hive
hiveserver2 = get_secrets(path="ye/hiveserver2")
host = hiveserver2["ip"]
port = hiveserver2["port"]
user = hiveserver2["user"]
conn = hive.connect(host, port=port, username=user)
return conn
```
- `airflow.providers.sktvane.macros.date.ds_nodash_plus_days` : AIDP 에서 제공하는 `date` 유틸리티
```python
from airflow.providers.sktvane.macros.date import ds_nodash_plus_days
...
def ds_nodash_tomorrow(ds):
ds_nodash_plus_days(ds, 1)
```
- `airflow.providers.sktvane.macros.date.ds_nodash_minus_days` : `ds_nodash_plus_days` 와 동일
- `airflow.providers.sktvane.macros.date.ym_nodash_add_month` : `ds_nodash_plus_days` 와 동일
- `airflow.providers.sktvane.macros.date.first_day_of_this_month` : `ds_nodash_plus_days` 와 동일
- `airflow.providers.sktvane.macros.date.last_day_of_this_month` : `ds_nodash_plus_days` 와 동일
- `airflow.providers.sktvane.macros.date.get_latest_loaded_dt` : `ds_nodash_plus_days` 와 동일
Raw data
{
"_id": null,
"home_page": "https://github.com/sktaiflow/sktvane-airflow-providers",
"name": "apache-airflow-providers-sktvane",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "airflow, sktvane",
"author": "aidp",
"author_email": "aidp@sktai.io",
"download_url": "https://files.pythonhosted.org/packages/d8/5c/95b755ef7f1b68fcda0cbb3899ad31c7d794656f4af00010a328bfb06765/apache-airflow-providers-sktvane-1.2.10.tar.gz",
"platform": null,
"description": "# apache-airflow-providers-sktvane\n\n- AIDP \uac00 \uc81c\uacf5\ud558\ub294 \uc790\uc6d0\ub4e4\uc5d0 \uc811\uadfc\ud558\ub294 \uc6a9\ub3c4\n - `NES`\n - `BigQuery`\n - `Vault`\n- \uae30\ud0c0 \uacf5\uc6a9 \ubaa9\uc801\uc758 \ucf54\ub4dc\n\n## PyPI\n\n- https://pypi.org/project/apache-airflow-providers-sktvane\n\n## Environments\n\n### Local\n\n* `VAULT_TOKEN` \uc740 [\uad00\ub828 \ubb38\uc11c](https://www.notion.so/ai-data-engineering/Public-3ec3cd8a4e444aa38a6f02fe57e3b6bd?pvs=4#c0b08e2b147f4e0383c67cedadbb0bdc) \uc5d0\uc11c \ud655\uc778\n ```shell\n export VAULT_ADDR=https://vault-public.sktai.io\n export VAULT_TOKEN={{VAULT_TOKEN}}\n export AIRFLOW__CORE__DAGS_FOLDER=.\n ```\n\n## Deployment\n\n* `main` \ube0c\ub79c\uce58\uc5d0 `push` \uc774\ubca4\ud2b8 \ubc1c\uc0dd \uc2dc \ubc30\ud3ec, \ubd80\ub4dd\uc774\ud558\uac8c \ub85c\uceec \ud658\uacbd\uc5d0\uc11c \ubc30\ud3ec\ud560 \uacbd\uc6b0 \uc544\ub798 \uba85\ub839 \uc218\ud589\n ```shell\n # build\n $ python setup.py sdist bdist_wheel\n # upload\n $ twine upload dist/*\n # remove\n $ rm -rf build dist apache_airflow_providers_sktvane.egg-info \n ```\n\n## Components\n\n###### Operators\n\n- `airflow.providers.sktvane.operators.nes.NesOperator` : AIDP \uc758 `NES` \uc0ac\uc6a9\n ```python\n from airflow.providers.sktvane.operators.nes import NesOperator\n \n ...\n \n NesOperator(\n task_id=\"jupyter_daily_count\",\n input_nb=\"https://github.com/sktaiflow/notebooks/blob/master/statistics/jupyter_daily_count.ipynb\",\n parameters={\"current_date\": \"{{ ds }}\", \"channel\": \"#aim-statistics\"},\n )\n ```\n \n\n###### Sensors\n\n- `airflow.providers.sktvane.sensors.gcp.BigqueryPartitionSensor` : AIDP \uc758 `BigQuery` \ud30c\ud2f0\uc158 \uccb4\ud06c\n\n ```python\n from airflow.providers.sktvane.sensors.gcp import BigqueryPartitionSensor\n \n ...\n \n BigqueryPartitionSensor(\n task_id=f\"{table}_partition_sensor\",\n dataset_id=\"wind_tmt\",\n table_id=table,\n partition=\"dt = '{{ds}}'\",\n )\n ``` \n\n###### Macros\n\n- `airflow.providers.sktvane.macros.slack.send_fail_message` : AIDP \uc815\uc758 \ud3ec\ub9f7\uc73c\ub85c `Slack` \uc5d0\ub7ec \uba54\uc2dc\uc9c0 \ubc1c\uc1a1\n ```python\n from airflow.providers.sktvane.macros.slack import send_fail_message\n \n ...\n \n def send_aidp_fail_message(slack_email: str) -> None:\n send_fail_message(\n slack_channel=\"#aidp-airflow-monitoring\",\n slack_username=f\"Airflow-AlarmBot-{env}\",\n slack_email=slack_email,\n )\n ```\n \n- `airflow.providers.sktvane.macros.gcp.bigquery_client` : AIDP \uc758 `BigQuery` \uc0ac\uc6a9\n ```python\n from airflow.providers.sktvane.macros.gcp import bigquery_client\n \n ...\n \n def bq_query_to_bq(query, dest_table_name, **kwarg):\n bq_client = bigquery_client()\n job = bq_client.query(query)\n job.result()\n ```\n \n- `airflow.providers.sktvane.macros.vault.get_secrets` : AIDP \uc758 `Vault` \uc0ac\uc6a9\n ```python\n from airflow.providers.sktvane.macros.vault import get_secrets\n \n ...\n \n def get_hive_conn():\n from pyhive import hive\n \n hiveserver2 = get_secrets(path=\"ye/hiveserver2\")\n host = hiveserver2[\"ip\"]\n port = hiveserver2[\"port\"]\n user = hiveserver2[\"user\"]\n conn = hive.connect(host, port=port, username=user)\n return conn\n ```\n \n- `airflow.providers.sktvane.macros.date.ds_nodash_plus_days` : AIDP \uc5d0\uc11c \uc81c\uacf5\ud558\ub294 `date` \uc720\ud2f8\ub9ac\ud2f0\n ```python\n from airflow.providers.sktvane.macros.date import ds_nodash_plus_days\n \n ...\n \n def ds_nodash_tomorrow(ds):\n ds_nodash_plus_days(ds, 1)\n ```\n- `airflow.providers.sktvane.macros.date.ds_nodash_minus_days` : `ds_nodash_plus_days` \uc640 \ub3d9\uc77c\n- `airflow.providers.sktvane.macros.date.ym_nodash_add_month` : `ds_nodash_plus_days` \uc640 \ub3d9\uc77c\n- `airflow.providers.sktvane.macros.date.first_day_of_this_month` : `ds_nodash_plus_days` \uc640 \ub3d9\uc77c\n- `airflow.providers.sktvane.macros.date.last_day_of_this_month` : `ds_nodash_plus_days` \uc640 \ub3d9\uc77c\n- `airflow.providers.sktvane.macros.date.get_latest_loaded_dt` : `ds_nodash_plus_days` \uc640 \ub3d9\uc77c\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Provider for Apache Airflow. Implements apache-airflow-providers-sktvane package by skt",
"version": "1.2.10",
"project_urls": {
"Download": "https://github.com/sktaiflow/sktvane-airflow-providers",
"Homepage": "https://github.com/sktaiflow/sktvane-airflow-providers"
},
"split_keywords": [
"airflow",
" sktvane"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d85c95b755ef7f1b68fcda0cbb3899ad31c7d794656f4af00010a328bfb06765",
"md5": "e77950daa6e0acda53b8fccebda0e266",
"sha256": "9cce8f38cad139d6088f1feb31ed8f8e8867717d419f388b8a95a556330602d5"
},
"downloads": -1,
"filename": "apache-airflow-providers-sktvane-1.2.10.tar.gz",
"has_sig": false,
"md5_digest": "e77950daa6e0acda53b8fccebda0e266",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 10005,
"upload_time": "2024-08-07T04:30:27",
"upload_time_iso_8601": "2024-08-07T04:30:27.953758Z",
"url": "https://files.pythonhosted.org/packages/d8/5c/95b755ef7f1b68fcda0cbb3899ad31c7d794656f4af00010a328bfb06765/apache-airflow-providers-sktvane-1.2.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-07 04:30:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sktaiflow",
"github_project": "sktvane-airflow-providers",
"github_not_found": true,
"lcname": "apache-airflow-providers-sktvane"
}
JSON
