[](https://github.com/iloveitaly/funcy-pipe/releases) [](https://pepy.tech/project/funcy-pipe) [](https://pypi.org/project/funcy-pipe)  [](https://opensource.org/licenses/MIT)
# Funcy with pipeline-based operators
If [Funcy](https://github.com/Suor/funcy) and [Pipe](https://github.com/JulienPalard/Pipe) had a baby. Deal with data transformation in python in a sane way.
I love Ruby. It's a great language and one of the things they got right was pipelined data transformation. Elixir got this
even more right with the explicit pipeline operator `|>`.
However, Python is the way of the future. As I worked more with Python, it was driving me nuts that the
data transformation options were not chainable.
This project fixes this pet peeve.
## Installation
```shell
pip install funcy-pipe
```
Or, if you are using poetry:
```shell
poetry add funcy-pipe
```
## Examples
Extract a couple key values from a sql alchemy model:
```python
import funcy_pipe as fp
entities_from_sql_alchemy
| fp.lmap(lambda r: r.to_dict())
| fp.lmap(lambda r: r | fp.omit(["id", "created_at", "updated_at"]))
| fp.to_list
```
Or, you can be more fancy and use [whatever](https://github.com/Suor/whatever) and `pmap`:
```python
import funcy_pipe as f
import whatever as _
entities_from_sql_alchemy
| fp.lmap(_.to_dict)
| fp.pmap(fp.omit(["id", "created_at", "updated_at"]))
| fp.to_list
```
Create a map from an array of objects, ensuring the key is always an `int`:
```python
section_map = api.get_sections() | fp.group_by(f.compose(int, that.id))
```
Grab the ID of a specific user:
```python
filter_user_id = (
collaborator_map().values()
| fp.where(email=target_user)
| fp.pluck("id")
| fp.first()
)
```
Get distinct values from a list (in this case, github events):
```python
events | fp.pluck("type") | fp.distinct() | fp.to_list()
```
What if the objects are not dicts?
```python
filter_user_id = (
collaborator_map().values()
| fp.where_attr(email=target_user)
| fp.pluck_attr("id")
| fp.first()
)
```
How about creating a dict where each value is sorted:
```python
data
# each element is a dict of city information, let's group by state
| fp.group_by(itemgetter("state_name"))
# now let's sort each value by population, which is stored as a string
| fp.walk_values(
f.partial(sorted, reverse=True, key=lambda c: int(c["population"])),
)
```
A more complicated example ([lifted from this project](https://github.com/iloveitaly/todoist-digest/blob/2f893709da2cf3a0f715125053af705fc3adbc4c/run.py#L151-L166)):
```python
comments = (
# tasks are pulled from the todoist api
tasks
# get all comments for each relevant task
| fp.lmap(lambda task: api.get_comments(task_id=task.id))
# each task's comments are returned as an array, let's flatten this
| fp.flatten()
# dates are returned as strings, let's convert them to datetime objects
| fp.lmap(enrich_date)
# no date filter is applied by default, we don't want all comments
| fp.lfilter(lambda comment: comment["posted_at_date"] > last_synced_date)
# comments do not come with who created the comment by default, we need to hit a separate API to add this to the comment
| fp.lmap(enrich_comment)
# only select the comments posted by our target user
| fp.lfilter(lambda comment: comment["posted_by_user_id"] == filter_user_id)
# there is no `sort` in the funcy library, so we reexport the sort built-in so it's pipe-able
| fp.sort(key="posted_at_date")
# create a dictionary of task_id => [comments]
| fp.group_by(lambda comment: comment["task_id"])
)
```
## Extras
* to_list
* log
* bp. run `breakpoint()` on the input value
* sort
* exactly_one. Throw an error if the input is not exactly one element
* reduce
* pmap. Pass each element of a sequence into a pipe'd function
## Extensions
There [are some functions](funcy_pipe/funcy_extensions.py) which are not yet merged upstream into funcy, and may never be. You can patch `funcy` to add them using:
```python
import funcy_pipe
funcy_pipe.patch()
```
## Coming From Ruby?
* uniq => distinct
* detect => `where(some="Condition") | first` or `where_attr(some="Condition") | first`
### Module Alias
Create a module alias for `funcy-pipe` to make things clean (`import *` always irks me):
```python
# fp.py
from funcy_pipe import *
# code py
import fp
```
# Inspiration
* Elixir's pipe operator. `array |> map(fn) |> filter(fn)`
* Ruby's enumerable library. `array.map(&:fn).filter(&:fn)`
* https://pypi.org/project/funcy-chain
* https://github.com/JulienPalard/Pipe
# TODO
- [ ] tests
- [ ] docs for additional utils
- [ ] fix typing threading
Raw data
{
"_id": null,
"home_page": "https://github.com/iloveitaly/funcy-pipe",
"name": "funcy-pipe",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "python, functional-programming, pipe, funcy, data-manipulation",
"author": "Michael Bianco",
"author_email": "mike@mikebian.co",
"download_url": "https://files.pythonhosted.org/packages/73/ea/25fc6dcbdba0e4077871b182330b95833a9c11fb2c3d94cc614d3c716493/funcy_pipe-0.11.0.tar.gz",
"platform": null,
"description": "[](https://github.com/iloveitaly/funcy-pipe/releases) [](https://pepy.tech/project/funcy-pipe) [](https://pypi.org/project/funcy-pipe)  [](https://opensource.org/licenses/MIT)\n\n# Funcy with pipeline-based operators\n\nIf [Funcy](https://github.com/Suor/funcy) and [Pipe](https://github.com/JulienPalard/Pipe) had a baby. Deal with data transformation in python in a sane way.\n\nI love Ruby. It's a great language and one of the things they got right was pipelined data transformation. Elixir got this\neven more right with the explicit pipeline operator `|>`.\n\nHowever, Python is the way of the future. As I worked more with Python, it was driving me nuts that the\ndata transformation options were not chainable.\n\nThis project fixes this pet peeve.\n\n## Installation\n\n```shell\npip install funcy-pipe\n```\n\nOr, if you are using poetry:\n\n```shell\npoetry add funcy-pipe\n```\n\n## Examples\n\nExtract a couple key values from a sql alchemy model:\n\n```python\nimport funcy_pipe as fp\n\nentities_from_sql_alchemy\n | fp.lmap(lambda r: r.to_dict())\n | fp.lmap(lambda r: r | fp.omit([\"id\", \"created_at\", \"updated_at\"]))\n | fp.to_list\n```\n\nOr, you can be more fancy and use [whatever](https://github.com/Suor/whatever) and `pmap`:\n\n```python\nimport funcy_pipe as f\nimport whatever as _\n\nentities_from_sql_alchemy\n | fp.lmap(_.to_dict)\n | fp.pmap(fp.omit([\"id\", \"created_at\", \"updated_at\"]))\n | fp.to_list\n```\n\nCreate a map from an array of objects, ensuring the key is always an `int`:\n\n```python\nsection_map = api.get_sections() | fp.group_by(f.compose(int, that.id))\n```\n\nGrab the ID of a specific user:\n\n```python\nfilter_user_id = (\n collaborator_map().values()\n | fp.where(email=target_user)\n | fp.pluck(\"id\")\n | fp.first()\n)\n```\n\nGet distinct values from a list (in this case, github events):\n\n```python\nevents | fp.pluck(\"type\") | fp.distinct() | fp.to_list()\n```\n\nWhat if the objects are not dicts?\n\n```python\nfilter_user_id = (\n collaborator_map().values()\n | fp.where_attr(email=target_user)\n | fp.pluck_attr(\"id\")\n | fp.first()\n)\n```\n\nHow about creating a dict where each value is sorted:\n\n```python\ndata\n # each element is a dict of city information, let's group by state\n | fp.group_by(itemgetter(\"state_name\"))\n # now let's sort each value by population, which is stored as a string\n | fp.walk_values(\n f.partial(sorted, reverse=True, key=lambda c: int(c[\"population\"])),\n )\n```\n\nA more complicated example ([lifted from this project](https://github.com/iloveitaly/todoist-digest/blob/2f893709da2cf3a0f715125053af705fc3adbc4c/run.py#L151-L166)):\n\n```python\ncomments = (\n # tasks are pulled from the todoist api\n tasks\n # get all comments for each relevant task\n | fp.lmap(lambda task: api.get_comments(task_id=task.id))\n # each task's comments are returned as an array, let's flatten this\n | fp.flatten()\n # dates are returned as strings, let's convert them to datetime objects\n | fp.lmap(enrich_date)\n # no date filter is applied by default, we don't want all comments\n | fp.lfilter(lambda comment: comment[\"posted_at_date\"] > last_synced_date)\n # comments do not come with who created the comment by default, we need to hit a separate API to add this to the comment\n | fp.lmap(enrich_comment)\n # only select the comments posted by our target user\n | fp.lfilter(lambda comment: comment[\"posted_by_user_id\"] == filter_user_id)\n # there is no `sort` in the funcy library, so we reexport the sort built-in so it's pipe-able\n | fp.sort(key=\"posted_at_date\")\n # create a dictionary of task_id => [comments]\n | fp.group_by(lambda comment: comment[\"task_id\"])\n)\n```\n\n## Extras\n\n* to_list\n* log\n* bp. run `breakpoint()` on the input value\n* sort\n* exactly_one. Throw an error if the input is not exactly one element\n* reduce\n* pmap. Pass each element of a sequence into a pipe'd function\n\n## Extensions\n\nThere [are some functions](funcy_pipe/funcy_extensions.py) which are not yet merged upstream into funcy, and may never be. You can patch `funcy` to add them using:\n\n```python\nimport funcy_pipe\nfuncy_pipe.patch()\n```\n\n## Coming From Ruby?\n\n* uniq => distinct\n* detect => `where(some=\"Condition\") | first` or `where_attr(some=\"Condition\") | first`\n\n### Module Alias\n\nCreate a module alias for `funcy-pipe` to make things clean (`import *` always irks me):\n\n```python\n# fp.py\nfrom funcy_pipe import *\n\n# code py\nimport fp\n```\n\n# Inspiration\n\n* Elixir's pipe operator. `array |> map(fn) |> filter(fn)`\n* Ruby's enumerable library. `array.map(&:fn).filter(&:fn)`\n* https://pypi.org/project/funcy-chain\n* https://github.com/JulienPalard/Pipe\n\n# TODO\n\n- [ ] tests\n- [ ] docs for additional utils\n- [ ] fix typing threading\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "If Funcy and Pipe had a baby. Decorates all Funcy methods with Pipe superpowers.",
"version": "0.11.0",
"project_urls": {
"Homepage": "https://github.com/iloveitaly/funcy-pipe"
},
"split_keywords": [
"python",
" functional-programming",
" pipe",
" funcy",
" data-manipulation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "34b771574cb0b3f8d1f8a386dbc9c240f0a23852390292d8fe6b8d4b741809f3",
"md5": "26cf917e8c95676a2cdf60e8351b268d",
"sha256": "3ad64a39ea9299e52edaba929803422bd80acc10e04ce1746f4fe549d54f461e"
},
"downloads": -1,
"filename": "funcy_pipe-0.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "26cf917e8c95676a2cdf60e8351b268d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 6749,
"upload_time": "2024-03-26T21:30:19",
"upload_time_iso_8601": "2024-03-26T21:30:19.985733Z",
"url": "https://files.pythonhosted.org/packages/34/b7/71574cb0b3f8d1f8a386dbc9c240f0a23852390292d8fe6b8d4b741809f3/funcy_pipe-0.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "73ea25fc6dcbdba0e4077871b182330b95833a9c11fb2c3d94cc614d3c716493",
"md5": "33933001c7f64209adaf1011da398a9c",
"sha256": "8dcedb05b13f4502e88ec75a794da1c4682ee3c6aab59c9a423cf42dc3b94805"
},
"downloads": -1,
"filename": "funcy_pipe-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "33933001c7f64209adaf1011da398a9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 5812,
"upload_time": "2024-03-26T21:30:25",
"upload_time_iso_8601": "2024-03-26T21:30:25.723987Z",
"url": "https://files.pythonhosted.org/packages/73/ea/25fc6dcbdba0e4077871b182330b95833a9c11fb2c3d94cc614d3c716493/funcy_pipe-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 21:30:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "iloveitaly",
"github_project": "funcy-pipe",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "funcy-pipe"
}
JSON
