# Apache Flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Learn more about Flink at [https://flink.apache.org/](https://flink.apache.org/)
## Python Packaging
PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads,
such as real-time data processing pipelines, large-scale exploratory data analysis, Machine Learning (ML)
pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas,
then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem.
Depending on the level of abstraction you need, there are two different APIs that can be used in PyFlink: PyFlink Table API and PyFlink DataStream API.
The PyFlink Table API allows you to write powerful relational queries in a way that is similar to
using SQL or working with tabular data in Python. You can find more information about it via the tutorial
[https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/)
The PyFlink DataStream API gives you lower-level control over the core building blocks of Flink,
state and time, to build more complex stream processing use cases.
Tutorial can be found at [https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/)
You can find more information via the documentation at [https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/)
The auto-generated Python docs can be found at [https://nightlies.apache.org/flink/flink-docs-stable/api/python/](https://nightlies.apache.org/flink/flink-docs-stable/api/python/)
## Python Requirements
Apache Flink Python API depends on Py4J (currently version 0.10.9.7), CloudPickle (currently version 2.2.0), python-dateutil (currently version >=2.8.0,<3), Apache Beam (currently version >=2.43.0,<2.49.0).
## Development Notices
### Protobuf Code Generation
Protocol buffer is used in file `flink_fn_execution_pb2.py` and `flink_fn_execution_pb2.pyi`, the file is generated from `flink-fn-execution.proto`. Whenever `flink-fn-execution.proto` is updated, please re-generate `flink_fn_execution_pb2.py` and `flink_fn_execution_pb2.pyi` by executing:
```
python pyflink/gen_protos.py
```
PyFlink depends on the following libraries to execute the above script:
1. grpcio-tools (>=1.29.0,<=1.50.0)
2. setuptools (>=37.0.0)
3. pip (>=20.3)
### Running Test Cases
Currently, we use conda and tox to verify the compatibility of the Flink Python API for multiple versions of Python and will integrate some useful plugins with tox, such as flake8.
We can enter the directory where this README.md file is located and run test cases by executing
```
./dev/lint-python.sh
```
To use your system conda environment, you can set `FLINK_CONDA_HOME` variable:
```shell
export FLINK_CONDA_HOME=$(dirname $(dirname $CONDA_EXE))
```
Create a virtual environment:
```shell
conda create -n pyflink_38 python=3.8
```
Then you can activate your environment and run tests, for example:
```shell
conda activate pyflink_38
pip install -r ./dev/dev-requirements.txt
./dev/lint-python.sh
```
Raw data
{
"_id": null,
"home_page": "https://flink.apache.org",
"name": "apache-flink",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Apache Software Foundation",
"author_email": "dev@flink.apache.org",
"download_url": "https://files.pythonhosted.org/packages/37/85/830c518b8a80095c371d3af5143c90d51a21958f1d0c4095ce799445bcfe/apache-flink-1.20.0.tar.gz",
"platform": null,
"description": "# Apache Flink\n\nApache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.\n\nLearn more about Flink at [https://flink.apache.org/](https://flink.apache.org/)\n\n## Python Packaging\n\nPyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads,\nsuch as real-time data processing pipelines, large-scale exploratory data analysis, Machine Learning (ML)\npipelines and ETL processes. If you\u2019re already familiar with Python and libraries such as Pandas,\nthen PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem.\nDepending on the level of abstraction you need, there are two different APIs that can be used in PyFlink: PyFlink Table API and PyFlink DataStream API.\n\nThe PyFlink Table API allows you to write powerful relational queries in a way that is similar to\nusing SQL or working with tabular data in Python. You can find more information about it via the tutorial\n[https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/)\n\nThe PyFlink DataStream API gives you lower-level control over the core building blocks of Flink,\nstate and time, to build more complex stream processing use cases.\nTutorial can be found at [https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/datastream_tutorial/)\n\nYou can find more information via the documentation at [https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/)\n\nThe auto-generated Python docs can be found at [https://nightlies.apache.org/flink/flink-docs-stable/api/python/](https://nightlies.apache.org/flink/flink-docs-stable/api/python/)\n\n## Python Requirements\n\nApache Flink Python API depends on Py4J (currently version 0.10.9.7), CloudPickle (currently version 2.2.0), python-dateutil (currently version >=2.8.0,<3), Apache Beam (currently version >=2.43.0,<2.49.0).\n\n## Development Notices\n\n### Protobuf Code Generation\n\nProtocol buffer is used in file `flink_fn_execution_pb2.py` and `flink_fn_execution_pb2.pyi`, the file is generated from `flink-fn-execution.proto`. Whenever `flink-fn-execution.proto` is updated, please re-generate `flink_fn_execution_pb2.py` and `flink_fn_execution_pb2.pyi` by executing:\n\n```\npython pyflink/gen_protos.py\n```\n\nPyFlink depends on the following libraries to execute the above script:\n1. grpcio-tools (>=1.29.0,<=1.50.0)\n2. setuptools (>=37.0.0)\n3. pip (>=20.3)\n\n### Running Test Cases \n\nCurrently, we use conda and tox to verify the compatibility of the Flink Python API for multiple versions of Python and will integrate some useful plugins with tox, such as flake8.\nWe can enter the directory where this README.md file is located and run test cases by executing\n\n```\n./dev/lint-python.sh\n```\n\nTo use your system conda environment, you can set `FLINK_CONDA_HOME` variable:\n\n```shell\nexport FLINK_CONDA_HOME=$(dirname $(dirname $CONDA_EXE))\n```\n\nCreate a virtual environment:\n```shell\nconda create -n pyflink_38 python=3.8\n```\n\nThen you can activate your environment and run tests, for example:\n\n```shell\nconda activate pyflink_38\npip install -r ./dev/dev-requirements.txt\n./dev/lint-python.sh\n```\n",
"bugtrack_url": null,
"license": "https://www.apache.org/licenses/LICENSE-2.0",
"summary": "Apache Flink Python API",
"version": "1.20.0",
"project_urls": {
"Homepage": "https://flink.apache.org"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "328fc6a2061ba5acd81a7d99b2ec6ea2b26708f9b4a73edb06523c11e92c64ec",
"md5": "15f786fef79359ea2b6c13bfa8fc8c94",
"sha256": "68839dc7cbec33f76ec2241e0035aa50642bfe618b0db0c1d49ce9aee216e4ac"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp310-cp310-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "15f786fef79359ea2b6c13bfa8fc8c94",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 2554655,
"upload_time": "2024-08-01T04:14:17",
"upload_time_iso_8601": "2024-08-01T04:14:17.020788Z",
"url": "https://files.pythonhosted.org/packages/32/8f/c6a2061ba5acd81a7d99b2ec6ea2b26708f9b4a73edb06523c11e92c64ec/apache_flink-1.20.0-cp310-cp310-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "906fe2d6e23b52a9fde20660a1d5f8489348d9f7576c05bd5fc02912d65d4cc7",
"md5": "caa0e4efd385df4e4fa2780ea34026f1",
"sha256": "c8c926b6d869dea556cc11d975a7b9c2c8e2489db90cc7035620669f698df62d"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "caa0e4efd385df4e4fa2780ea34026f1",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 2467151,
"upload_time": "2024-08-01T04:14:36",
"upload_time_iso_8601": "2024-08-01T04:14:36.085407Z",
"url": "https://files.pythonhosted.org/packages/90/6f/e2d6e23b52a9fde20660a1d5f8489348d9f7576c05bd5fc02912d65d4cc7/apache_flink-1.20.0-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2b0b01cc78ca3afaab9b683860f6b371d4b4e7d741012377165b902ed0853428",
"md5": "fb7b67c1711b875a8dd76f42894e34a6",
"sha256": "2e5bcbd841f4c3a979001f05ccae16ab6e8845e254ae99cfbcef6b3f00700572"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp310-cp310-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "fb7b67c1711b875a8dd76f42894e34a6",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 2635698,
"upload_time": "2024-08-01T04:18:46",
"upload_time_iso_8601": "2024-08-01T04:18:46.599296Z",
"url": "https://files.pythonhosted.org/packages/2b/0b/01cc78ca3afaab9b683860f6b371d4b4e7d741012377165b902ed0853428/apache_flink-1.20.0-cp310-cp310-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7284c55be6b247d953dab7b50d9639294965469174c01ef8b46a59a7a1b0d50b",
"md5": "9d6764a2731ccd5a5c58773c4759d1fb",
"sha256": "05bd6fee6292d5f4d84bda6ebedf954ca434787cdb14035609a56c2e57850606"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp311-cp311-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "9d6764a2731ccd5a5c58773c4759d1fb",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 2565134,
"upload_time": "2024-08-01T04:14:57",
"upload_time_iso_8601": "2024-08-01T04:14:57.301400Z",
"url": "https://files.pythonhosted.org/packages/72/84/c55be6b247d953dab7b50d9639294965469174c01ef8b46a59a7a1b0d50b/apache_flink-1.20.0-cp311-cp311-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c37d569ae9990c9824b17e477e541e4aa816cbb2e3e691d9e070d61979f36c2c",
"md5": "61170bfceadd78f7cf4722cadc7342f3",
"sha256": "3d013c0ffcfa5cf4491c233d1cee392ae4c7d9262851e01249bad0d6cb2d232b"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "61170bfceadd78f7cf4722cadc7342f3",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 2474999,
"upload_time": "2024-08-01T04:15:11",
"upload_time_iso_8601": "2024-08-01T04:15:11.752329Z",
"url": "https://files.pythonhosted.org/packages/c3/7d/569ae9990c9824b17e477e541e4aa816cbb2e3e691d9e070d61979f36c2c/apache_flink-1.20.0-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0e8352fe4520f102d2743527c39fd7e5cdd40ef6c97481eea34d82ce95754780",
"md5": "8a8ddad426c6144aa8f16eede9901b8f",
"sha256": "cb1bbe4a0a50d0bccc9c452695c4d10b4f1540667b5a17bff6963173a2f5261e"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp311-cp311-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "8a8ddad426c6144aa8f16eede9901b8f",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 2636527,
"upload_time": "2024-08-01T04:15:22",
"upload_time_iso_8601": "2024-08-01T04:15:22.676800Z",
"url": "https://files.pythonhosted.org/packages/0e/83/52fe4520f102d2743527c39fd7e5cdd40ef6c97481eea34d82ce95754780/apache_flink-1.20.0-cp311-cp311-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c90ad796015005f28afb73933f09c199021a4c843c2d8ce8aadf989595514ccd",
"md5": "40a911ac9bdfa1c3bf34319e62c907b0",
"sha256": "93a376478a3abe74dd7b11ea83342bf691b85f56d100d33c59239692751ca9a8"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "40a911ac9bdfa1c3bf34319e62c907b0",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 2559448,
"upload_time": "2024-08-01T04:15:33",
"upload_time_iso_8601": "2024-08-01T04:15:33.827192Z",
"url": "https://files.pythonhosted.org/packages/c9/0a/d796015005f28afb73933f09c199021a4c843c2d8ce8aadf989595514ccd/apache_flink-1.20.0-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3f1593c8d763d5bb29a29318ea87b9924f31478130d373551caebed088d670b8",
"md5": "a0b30c5fb1ec73fcc65aa5796f4fa32f",
"sha256": "85813618e70d8cfb2b9a5f25dca5c53a86af99afebe19cbcec10001618c07e0e"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp38-cp38-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "a0b30c5fb1ec73fcc65aa5796f4fa32f",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 2471872,
"upload_time": "2024-08-01T04:15:43",
"upload_time_iso_8601": "2024-08-01T04:15:43.052604Z",
"url": "https://files.pythonhosted.org/packages/3f/15/93c8d763d5bb29a29318ea87b9924f31478130d373551caebed088d670b8/apache_flink-1.20.0-cp38-cp38-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7fa3ad50270f0f9b7738922a170c47ec18061224a5afa0864e8749d27b1d5501",
"md5": "ec955261302c80a17c14a94cf1138975",
"sha256": "e678a9ecbfa3c30bc02b93e49ee823254614114ca5920b0270dfe741affcdc3d"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp38-cp38-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "ec955261302c80a17c14a94cf1138975",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 6757093,
"upload_time": "2024-08-01T04:15:55",
"upload_time_iso_8601": "2024-08-01T04:15:55.014836Z",
"url": "https://files.pythonhosted.org/packages/7f/a3/ad50270f0f9b7738922a170c47ec18061224a5afa0864e8749d27b1d5501/apache_flink-1.20.0-cp38-cp38-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ed438c23e7a4fda2da98b60efdf4e3d99a01aff81ce584169c3f4fcdfd410eac",
"md5": "79151130445e15d1d6f42204540dff8d",
"sha256": "c22beb6b8488fa86a45581cb6f62ae8b5dff256b7ba1bc8c91bb0878dcd71a74"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "79151130445e15d1d6f42204540dff8d",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 2559009,
"upload_time": "2024-08-01T04:16:13",
"upload_time_iso_8601": "2024-08-01T04:16:13.065702Z",
"url": "https://files.pythonhosted.org/packages/ed/43/8c23e7a4fda2da98b60efdf4e3d99a01aff81ce584169c3f4fcdfd410eac/apache_flink-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ca6449402a48a187df2996212d6a453db2a06d8163d6567960df0b3459832f55",
"md5": "d39974fd92b7ba0a75e5c4739fab0d0a",
"sha256": "23bafeb1b78d39b62fa95b738dbdccff889a57f5fadf3b59a873a0cb4c8f1292"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp39-cp39-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "d39974fd92b7ba0a75e5c4739fab0d0a",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 2470998,
"upload_time": "2024-08-01T04:16:22",
"upload_time_iso_8601": "2024-08-01T04:16:22.380324Z",
"url": "https://files.pythonhosted.org/packages/ca/64/49402a48a187df2996212d6a453db2a06d8163d6567960df0b3459832f55/apache_flink-1.20.0-cp39-cp39-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "19b8439d89a23d8fdb88b0d28dc34a40db7c6796641d9229d669008e797eae9c",
"md5": "f37ac08eb80117973de68cff73a75f0c",
"sha256": "c8074e88ce0dbc5b0e5e76705750e4e630760c37cc7bcd455046e73d8fdfe593"
},
"downloads": -1,
"filename": "apache_flink-1.20.0-cp39-cp39-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "f37ac08eb80117973de68cff73a75f0c",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 2638879,
"upload_time": "2024-08-01T04:16:30",
"upload_time_iso_8601": "2024-08-01T04:16:30.472590Z",
"url": "https://files.pythonhosted.org/packages/19/b8/439d89a23d8fdb88b0d28dc34a40db7c6796641d9229d669008e797eae9c/apache_flink-1.20.0-cp39-cp39-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3785830c518b8a80095c371d3af5143c90d51a21958f1d0c4095ce799445bcfe",
"md5": "7404819bfa3f76e4c553575b41bcc4da",
"sha256": "2f93ff09b911c484fda4c4cf0f175a609d73017ba63d7cb21bd59e53c56fd64b"
},
"downloads": -1,
"filename": "apache-flink-1.20.0.tar.gz",
"has_sig": false,
"md5_digest": "7404819bfa3f76e4c553575b41bcc4da",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1519192,
"upload_time": "2024-08-01T04:21:28",
"upload_time_iso_8601": "2024-08-01T04:21:28.621535Z",
"url": "https://files.pythonhosted.org/packages/37/85/830c518b8a80095c371d3af5143c90d51a21958f1d0c4095ce799445bcfe/apache-flink-1.20.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-01 04:21:28",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "apache-flink"
}
JSON