Install: `pip install lined`
[Documentation](https://otosense.github.io/lined/)
# lined
Building simple pipelines, simply.
And lightly too! No dependencies. All with pure builtin python.
A really simple example:
```pydocstring
>>> from lined import Line
>>> p = Line(sum, str)
>>> p([2, 3])
'5'
```
A still quite simple example:
```pydocstring
>>> def first(a, b=1):
... return a * b
>>>
>>> def last(c) -> float:
... return c + 10
>>>
>>> f = Line(first, last)
>>>
>>> assert f(2) == 12
>>> assert f(2, 10) == 30
```
Let's check out the signature of f:
```pydocstring
>>> from inspect import signature
>>>
>>> assert str(signature(f)) == '(a, b=1) -> float'
>>> assert signature(f).parameters == signature(first).parameters
>>> assert signature(f).return_annotation == signature(last).return_annotation == float
```
Border case: One function only
```pydocstring
>>> same_as_first = Line(first)
>>> assert same_as_first(42) == first(42)
```
# More?
## string and dot digraph representations
Line's string representation (`__repr__`) and how it deals with callables that don't have a `__name__` (hint: it makes one up):
```python
from lined.base import Line
from functools import partial
pipe = Line(sum, np.log, str, print, pipeline_name='some_name')
pipe
```
```
Line(sum, log, str, print, unnamed_func_001, pipeline_name='some_name')
```
If you have [graphviz](https://pypi.org/project/graphviz/) installed, you can also do this:
```python
pipe.dot_digraph()
```
And if you don't, but have some other [dot language](https://www.graphviz.org/doc/info/lang.html) interpreter, you can just get the body (and fiddle with it):
```python
print('\n'.join(pipe.dot_digraph_body()))
```
```
rankdir="LR"
sum [shape="box"]
log [shape="box"]
str [shape="box"]
print [shape="box"]
unnamed_func_001 [shape="box"]
sum -> log
log -> str
str -> print
print -> unnamed_func_001
```
Optionally, a pipeline can have an `input_name` and/or an `output_name`.
These will be used in the string representation and the dot digraph.
```python
pipe = Line(sum, np.log, str, print, input_name='x', output_name='y')
str(pipe)
```
```
"Line(sum, log, str, print, pipeline_name='some_name')"
```
```python
pipe.dot_digraph()
```
# Tools
## iterize and iterate
```python
from lined import Line
pipe = Line(lambda x: x * 2,
lambda x: f"hello {x}")
pipe(1)
```
'hello 2'
But what if you wanted to use the pipeline on a "stream" of data. The following wouldn't work:
```python
try:
pipe(iter([1,2,3]))
except TypeError as e:
print(f"{type(e).__name__}: {e}")
```
TypeError: unsupported operand type(s) for *: 'list_iterator' and 'int'
Remember that error: You'll surely encounter it at some point.
The solution to it is (often): `iterize`, which transforms a function that is meant to be applied to a single object, into a function that is meant to be applied to an array, or any iterable of such objects.
(You might be familiar (if you use `numpy` for example) with the related concept of "vectorization", or [array programming](https://en.wikipedia.org/wiki/Array_programming).)
```python
from lined import Line, iterize
from typing import Iterable
pipe = Line(iterize(lambda x: x * 2),
iterize(lambda x: f"hello {x}"))
iterable = pipe([1, 2, 3])
assert isinstance(iterable, Iterable) # see that the result is an iterable
list(iterable) # consume the iterable and gather it's items
```
['hello 2', 'hello 4', 'hello 6']
Instead of just computing the string, say that the last step actually printed the string (called a "callback" function whose result was less important than it's effect -- like storing something, etc.).
```python
from lined import Line, iterize, iterate
pipe = Line(iterize(lambda x: x * 2),
iterize(lambda x: print(f"hello {x}")),
)
for _ in pipe([1, 2, 3]):
pass
```
hello 2
hello 4
hello 6
It could be a bit awkward to have to "consume" the iterable to have it take effect.
Just doing a
```python
pipe([1, 2, 3])
```
to get those prints seems like a more natural way.
This is where you can use `iterate`. It basically "launches" that consuming loop for you.
```python
from lined import Line, iterize, iterate
pipe = Line(iterize(lambda x: x * 2),
iterize(lambda x: print(f"hello {x}")),
iterate
)
pipe([1, 2, 3])
```
hello 2
hello 4
hello 6
# Ramblings
## Decorating
Toddlers write lines of code.
Grown-ups write functions. Plenty of them.
Why break lines of code into small functions? Where to start...
- It's called modularity, and that's good
- You can reuse functions (and no, copy/paste isn't D.R.Y. --
and if you don't know what D.R.Y. is,
[grow up](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself!)).
- Because [7+-2](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two),
a.k.a [chunking](https://en.wikipedia.org/wiki/Chunking_(psychology)) or Miller's Law.
- You can [decorate](https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators)
functions, not lines of code.
`lined` sets you up to take advantage of these goodies.
Note this line (currently 117) of lined/base.py , in the init of Line:
self.funcs = tuple(map(fnode, self.funcs))
That is, every function is cast to with `fnode`.
`fnode` is:
def fnode(func, name=None):
return Fnode(func, name)
and `Fnode` is just a class that "transparently" wraps the function.
This is so that we can then use `Fnode` to do all kinds of things to the function
(without actually touching the function itself).
@dataclass
class Fnode:
func: Callable
__name__: Optional[str] = None
def __post_init__(self):
wraps(self.func)(self)
self.__name__ = self.__name__ or func_name(self.func)
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
Raw data
{
"_id": null,
"home_page": "https://github.com/otosense/lined",
"name": "lined",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "streams,pipelines",
"author": "OtoSense",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/98/e3/4c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf/lined-0.1.24.tar.gz",
"platform": "any",
"description": "Install: `pip install lined`\n\n[Documentation](https://otosense.github.io/lined/)\n\n# lined\n\nBuilding simple pipelines, simply.\n\nAnd lightly too! No dependencies. All with pure builtin python.\n\nA really simple example:\n\n```pydocstring\n>>> from lined import Line\n>>> p = Line(sum, str)\n>>> p([2, 3])\n'5'\n```\n\nA still quite simple example:\n\n```pydocstring\n>>> def first(a, b=1):\n... return a * b\n>>>\n>>> def last(c) -> float:\n... return c + 10\n>>>\n>>> f = Line(first, last)\n>>>\n>>> assert f(2) == 12\n>>> assert f(2, 10) == 30\n```\n\nLet's check out the signature of f:\n\n```pydocstring\n>>> from inspect import signature\n>>>\n>>> assert str(signature(f)) == '(a, b=1) -> float'\n>>> assert signature(f).parameters == signature(first).parameters\n>>> assert signature(f).return_annotation == signature(last).return_annotation == float\n```\n\nBorder case: One function only\n\n```pydocstring\n>>> same_as_first = Line(first)\n>>> assert same_as_first(42) == first(42)\n```\n\n\n\n# More?\n\n## string and dot digraph representations\n\nLine's string representation (`__repr__`) and how it deals with callables that don't have a `__name__` (hint: it makes one up):\n\n```python\nfrom lined.base import Line\nfrom functools import partial\n\npipe = Line(sum, np.log, str, print, pipeline_name='some_name')\npipe\n```\n```\nLine(sum, log, str, print, unnamed_func_001, pipeline_name='some_name')\n```\n\nIf you have [graphviz](https://pypi.org/project/graphviz/) installed, you can also do this:\n```python\npipe.dot_digraph()\n```\n\n\nAnd if you don't, but have some other [dot language](https://www.graphviz.org/doc/info/lang.html) interpreter, you can just get the body (and fiddle with it):\n\n```python\nprint('\\n'.join(pipe.dot_digraph_body()))\n```\n```\nrankdir=\"LR\"\nsum [shape=\"box\"]\nlog [shape=\"box\"]\nstr [shape=\"box\"]\nprint [shape=\"box\"]\nunnamed_func_001 [shape=\"box\"]\nsum -> log\nlog -> str\nstr -> print\nprint -> unnamed_func_001\n```\n\nOptionally, a pipeline can have an `input_name` and/or an `output_name`. \nThese will be used in the string representation and the dot digraph.\n\n```python\npipe = Line(sum, np.log, str, print, input_name='x', output_name='y')\nstr(pipe)\n```\n```\n\"Line(sum, log, str, print, pipeline_name='some_name')\"\n```\n\n```python\npipe.dot_digraph()\n```\n\n\n\n\n# Tools\n\n\n## iterize and iterate\n\n\n```python\nfrom lined import Line\n\npipe = Line(lambda x: x * 2, \n lambda x: f\"hello {x}\")\npipe(1)\n```\n\n\n\n\n 'hello 2'\n\n\n\nBut what if you wanted to use the pipeline on a \"stream\" of data. The following wouldn't work:\n\n\n```python\ntry:\n pipe(iter([1,2,3]))\nexcept TypeError as e:\n print(f\"{type(e).__name__}: {e}\")\n```\n\n TypeError: unsupported operand type(s) for *: 'list_iterator' and 'int'\n\n\nRemember that error: You'll surely encounter it at some point. \n\nThe solution to it is (often): `iterize`, which transforms a function that is meant to be applied to a single object, into a function that is meant to be applied to an array, or any iterable of such objects. \n(You might be familiar (if you use `numpy` for example) with the related concept of \"vectorization\", or [array programming](https://en.wikipedia.org/wiki/Array_programming).)\n\n\n\n```python\nfrom lined import Line, iterize\nfrom typing import Iterable\n\npipe = Line(iterize(lambda x: x * 2), \n iterize(lambda x: f\"hello {x}\"))\niterable = pipe([1, 2, 3])\nassert isinstance(iterable, Iterable) # see that the result is an iterable\nlist(iterable) # consume the iterable and gather it's items\n```\n\n\n\n\n ['hello 2', 'hello 4', 'hello 6']\n\n\n\nInstead of just computing the string, say that the last step actually printed the string (called a \"callback\" function whose result was less important than it's effect -- like storing something, etc.).\n\n\n```python\nfrom lined import Line, iterize, iterate\n\npipe = Line(iterize(lambda x: x * 2), \n iterize(lambda x: print(f\"hello {x}\")),\n )\n\nfor _ in pipe([1, 2, 3]):\n pass\n```\n\n hello 2\n hello 4\n hello 6\n\n\nIt could be a bit awkward to have to \"consume\" the iterable to have it take effect. \n\nJust doing a \n```python\npipe([1, 2, 3])\n```\nto get those prints seems like a more natural way. \n\nThis is where you can use `iterate`. It basically \"launches\" that consuming loop for you.\n\n\n```python\nfrom lined import Line, iterize, iterate\n\npipe = Line(iterize(lambda x: x * 2), \n iterize(lambda x: print(f\"hello {x}\")),\n iterate\n )\n\npipe([1, 2, 3])\n```\n\n hello 2\n hello 4\n hello 6\n\n\n# Ramblings\n\n## Decorating\n\nToddlers write lines of code. \nGrown-ups write functions. Plenty of them. \n\nWhy break lines of code into small functions? Where to start...\n- It's called modularity, and that's good\n- You can reuse functions (and no, copy/paste isn't D.R.Y. -- \nand if you don't know what D.R.Y. is, \n[grow up](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself!)).\n- Because [7+-2](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two), \na.k.a [chunking](https://en.wikipedia.org/wiki/Chunking_(psychology)) or Miller's Law.\n- You can [decorate](https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators)\nfunctions, not lines of code.\n\n`lined` sets you up to take advantage of these goodies. \n\nNote this line (currently 117) of lined/base.py , in the init of Line:\n\n self.funcs = tuple(map(fnode, self.funcs))\n\nThat is, every function is cast to with `fnode`.\n\n`fnode` is:\n\n def fnode(func, name=None):\n return Fnode(func, name)\n \nand `Fnode` is just a class that \"transparently\" wraps the function. \nThis is so that we can then use `Fnode` to do all kinds of things to the function \n(without actually touching the function itself).\n\n @dataclass\n class Fnode:\n func: Callable\n __name__: Optional[str] = None\n\n def __post_init__(self):\n wraps(self.func)(self)\n self.__name__ = self.__name__ or func_name(self.func)\n\n def __call__(self, *args, **kwargs):\n return self.func(*args, **kwargs)",
"bugtrack_url": null,
"license": "apache-2.0",
"summary": "Building simple pipelines simply.",
"version": "0.1.24",
"split_keywords": [
"streams",
"pipelines"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "98e34c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf",
"md5": "b7529ae6e59a1d4645431173b11d4fb1",
"sha256": "23ae461ff71d803fda5932d75e90f1ce9c317493d021098fff1cafd3790c27dd"
},
"downloads": -1,
"filename": "lined-0.1.24.tar.gz",
"has_sig": false,
"md5_digest": "b7529ae6e59a1d4645431173b11d4fb1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 34321,
"upload_time": "2023-04-04T23:27:01",
"upload_time_iso_8601": "2023-04-04T23:27:01.705006Z",
"url": "https://files.pythonhosted.org/packages/98/e3/4c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf/lined-0.1.24.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-04 23:27:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "otosense",
"github_project": "lined",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "lined"
}
JSON
