lined


Namelined JSON
Version 0.1.24 PyPI version JSON
download
home_pagehttps://github.com/otosense/lined
SummaryBuilding simple pipelines simply.
upload_time2023-04-04 23:27:01
maintainer
docs_urlNone
authorOtoSense
requires_python
licenseapache-2.0
keywords streams pipelines
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            Install: `pip install lined`

[Documentation](https://otosense.github.io/lined/)

# lined

Building simple pipelines, simply.

And lightly too! No dependencies. All with pure builtin python.

A really simple example:

```pydocstring
>>> from lined import Line
>>> p = Line(sum, str)
>>> p([2, 3])
'5'
```

A still quite simple example:

```pydocstring
>>> def first(a, b=1):
...     return a * b
>>>
>>> def last(c) -> float:
...     return c + 10
>>>
>>> f = Line(first, last)
>>>
>>> assert f(2) == 12
>>> assert f(2, 10) == 30
```

Let's check out the signature of f:

```pydocstring
>>> from inspect import signature
>>>
>>> assert str(signature(f)) == '(a, b=1) -> float'
>>> assert signature(f).parameters == signature(first).parameters
>>> assert signature(f).return_annotation == signature(last).return_annotation == float
```

Border case: One function only

```pydocstring
>>> same_as_first = Line(first)
>>> assert same_as_first(42) == first(42)
```



# More?

## string and dot digraph representations

Line's string representation (`__repr__`) and how it deals with callables that don't have a `__name__` (hint: it makes one up):

```python
from lined.base import Line
from functools import partial

pipe = Line(sum, np.log, str, print, pipeline_name='some_name')
pipe
```
```
Line(sum, log, str, print, unnamed_func_001, pipeline_name='some_name')
```

If you have [graphviz](https://pypi.org/project/graphviz/) installed, you can also do this:
```python
pipe.dot_digraph()
```
![image](https://user-images.githubusercontent.com/1906276/107063948-d23b0680-678f-11eb-88ce-1c0638175569.png)

And if you don't, but have some other [dot language](https://www.graphviz.org/doc/info/lang.html) interpreter, you can just get the body (and fiddle with it):

```python
print('\n'.join(pipe.dot_digraph_body()))
```
```
rankdir="LR"
sum [shape="box"]
log [shape="box"]
str [shape="box"]
print [shape="box"]
unnamed_func_001 [shape="box"]
sum -> log
log -> str
str -> print
print -> unnamed_func_001
```

Optionally, a pipeline can have an `input_name` and/or an `output_name`. 
These will be used in the string representation and the dot digraph.

```python
pipe = Line(sum, np.log, str, print, input_name='x', output_name='y')
str(pipe)
```
```
"Line(sum, log, str, print, pipeline_name='some_name')"
```

```python
pipe.dot_digraph()
```
![image](https://user-images.githubusercontent.com/1906276/107064180-175f3880-6790-11eb-87e0-5b75840a6f73.png)



# Tools


## iterize and iterate


```python
from lined import Line

pipe = Line(lambda x: x * 2, 
            lambda x: f"hello {x}")
pipe(1)
```




    'hello 2'



But what if you wanted to use the pipeline on a "stream" of data. The following wouldn't work:


```python
try:
    pipe(iter([1,2,3]))
except TypeError as e:
    print(f"{type(e).__name__}: {e}")
```

    TypeError: unsupported operand type(s) for *: 'list_iterator' and 'int'


Remember that error: You'll surely encounter it at some point. 

The solution to it is (often): `iterize`, which transforms a function that is meant to be applied to a single object, into a function that is meant to be applied to an array, or any iterable of such objects. 
(You might be familiar (if you use `numpy` for example) with the related concept of "vectorization", or [array programming](https://en.wikipedia.org/wiki/Array_programming).)



```python
from lined import Line, iterize
from typing import Iterable

pipe = Line(iterize(lambda x: x * 2), 
            iterize(lambda x: f"hello {x}"))
iterable = pipe([1, 2, 3])
assert isinstance(iterable, Iterable)  # see that the result is an iterable
list(iterable)  # consume the iterable and gather it's items
```




    ['hello 2', 'hello 4', 'hello 6']



Instead of just computing the string, say that the last step actually printed the string (called a "callback" function whose result was less important than it's effect -- like storing something, etc.).


```python
from lined import Line, iterize, iterate

pipe = Line(iterize(lambda x: x * 2), 
            iterize(lambda x: print(f"hello {x}")),
            )

for _ in pipe([1, 2, 3]):
    pass
```

    hello 2
    hello 4
    hello 6


It could be a bit awkward to have to "consume" the iterable to have it take effect. 

Just doing a 
```python
pipe([1, 2, 3])
```
to get those prints seems like a more natural way. 

This is where you can use `iterate`. It basically "launches" that consuming loop for you.


```python
from lined import Line, iterize, iterate

pipe = Line(iterize(lambda x: x * 2), 
            iterize(lambda x: print(f"hello {x}")),
            iterate
            )

pipe([1, 2, 3])
```

    hello 2
    hello 4
    hello 6


# Ramblings

## Decorating

Toddlers write lines of code. 
Grown-ups write functions. Plenty of them. 

Why break lines of code into small functions? Where to start...
- It's called modularity, and that's good
- You can reuse functions (and no, copy/paste isn't D.R.Y. -- 
and if you don't know what D.R.Y. is, 
[grow up](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself!)).
- Because [7+-2](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two), 
a.k.a [chunking](https://en.wikipedia.org/wiki/Chunking_(psychology)) or Miller's Law.
- You can [decorate](https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators)
functions, not lines of code.

`lined` sets you up to take advantage of these goodies. 

Note this line (currently 117) of lined/base.py , in the init of Line:

    self.funcs = tuple(map(fnode, self.funcs))

That is, every function is cast to with `fnode`.

`fnode` is:

    def fnode(func, name=None):
        return Fnode(func, name)
        
and `Fnode` is just a class that "transparently" wraps the function. 
This is so that we can then use `Fnode` to do all kinds of things to the function 
(without actually touching the function itself).

    @dataclass
    class Fnode:
        func: Callable
        __name__: Optional[str] = None

    def __post_init__(self):
        wraps(self.func)(self)
        self.__name__ = self.__name__ or func_name(self.func)

    def __call__(self, *args, **kwargs):
        return self.func(*args, **kwargs)
            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/otosense/lined",
    "name": "lined",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "streams,pipelines",
    "author": "OtoSense",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/98/e3/4c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf/lined-0.1.24.tar.gz",
    "platform": "any",
    "description": "Install: `pip install lined`\n\n[Documentation](https://otosense.github.io/lined/)\n\n# lined\n\nBuilding simple pipelines, simply.\n\nAnd lightly too! No dependencies. All with pure builtin python.\n\nA really simple example:\n\n```pydocstring\n>>> from lined import Line\n>>> p = Line(sum, str)\n>>> p([2, 3])\n'5'\n```\n\nA still quite simple example:\n\n```pydocstring\n>>> def first(a, b=1):\n...     return a * b\n>>>\n>>> def last(c) -> float:\n...     return c + 10\n>>>\n>>> f = Line(first, last)\n>>>\n>>> assert f(2) == 12\n>>> assert f(2, 10) == 30\n```\n\nLet's check out the signature of f:\n\n```pydocstring\n>>> from inspect import signature\n>>>\n>>> assert str(signature(f)) == '(a, b=1) -> float'\n>>> assert signature(f).parameters == signature(first).parameters\n>>> assert signature(f).return_annotation == signature(last).return_annotation == float\n```\n\nBorder case: One function only\n\n```pydocstring\n>>> same_as_first = Line(first)\n>>> assert same_as_first(42) == first(42)\n```\n\n\n\n# More?\n\n## string and dot digraph representations\n\nLine's string representation (`__repr__`) and how it deals with callables that don't have a `__name__` (hint: it makes one up):\n\n```python\nfrom lined.base import Line\nfrom functools import partial\n\npipe = Line(sum, np.log, str, print, pipeline_name='some_name')\npipe\n```\n```\nLine(sum, log, str, print, unnamed_func_001, pipeline_name='some_name')\n```\n\nIf you have [graphviz](https://pypi.org/project/graphviz/) installed, you can also do this:\n```python\npipe.dot_digraph()\n```\n![image](https://user-images.githubusercontent.com/1906276/107063948-d23b0680-678f-11eb-88ce-1c0638175569.png)\n\nAnd if you don't, but have some other [dot language](https://www.graphviz.org/doc/info/lang.html) interpreter, you can just get the body (and fiddle with it):\n\n```python\nprint('\\n'.join(pipe.dot_digraph_body()))\n```\n```\nrankdir=\"LR\"\nsum [shape=\"box\"]\nlog [shape=\"box\"]\nstr [shape=\"box\"]\nprint [shape=\"box\"]\nunnamed_func_001 [shape=\"box\"]\nsum -> log\nlog -> str\nstr -> print\nprint -> unnamed_func_001\n```\n\nOptionally, a pipeline can have an `input_name` and/or an `output_name`. \nThese will be used in the string representation and the dot digraph.\n\n```python\npipe = Line(sum, np.log, str, print, input_name='x', output_name='y')\nstr(pipe)\n```\n```\n\"Line(sum, log, str, print, pipeline_name='some_name')\"\n```\n\n```python\npipe.dot_digraph()\n```\n![image](https://user-images.githubusercontent.com/1906276/107064180-175f3880-6790-11eb-87e0-5b75840a6f73.png)\n\n\n\n# Tools\n\n\n## iterize and iterate\n\n\n```python\nfrom lined import Line\n\npipe = Line(lambda x: x * 2, \n            lambda x: f\"hello {x}\")\npipe(1)\n```\n\n\n\n\n    'hello 2'\n\n\n\nBut what if you wanted to use the pipeline on a \"stream\" of data. The following wouldn't work:\n\n\n```python\ntry:\n    pipe(iter([1,2,3]))\nexcept TypeError as e:\n    print(f\"{type(e).__name__}: {e}\")\n```\n\n    TypeError: unsupported operand type(s) for *: 'list_iterator' and 'int'\n\n\nRemember that error: You'll surely encounter it at some point. \n\nThe solution to it is (often): `iterize`, which transforms a function that is meant to be applied to a single object, into a function that is meant to be applied to an array, or any iterable of such objects. \n(You might be familiar (if you use `numpy` for example) with the related concept of \"vectorization\", or [array programming](https://en.wikipedia.org/wiki/Array_programming).)\n\n\n\n```python\nfrom lined import Line, iterize\nfrom typing import Iterable\n\npipe = Line(iterize(lambda x: x * 2), \n            iterize(lambda x: f\"hello {x}\"))\niterable = pipe([1, 2, 3])\nassert isinstance(iterable, Iterable)  # see that the result is an iterable\nlist(iterable)  # consume the iterable and gather it's items\n```\n\n\n\n\n    ['hello 2', 'hello 4', 'hello 6']\n\n\n\nInstead of just computing the string, say that the last step actually printed the string (called a \"callback\" function whose result was less important than it's effect -- like storing something, etc.).\n\n\n```python\nfrom lined import Line, iterize, iterate\n\npipe = Line(iterize(lambda x: x * 2), \n            iterize(lambda x: print(f\"hello {x}\")),\n            )\n\nfor _ in pipe([1, 2, 3]):\n    pass\n```\n\n    hello 2\n    hello 4\n    hello 6\n\n\nIt could be a bit awkward to have to \"consume\" the iterable to have it take effect. \n\nJust doing a \n```python\npipe([1, 2, 3])\n```\nto get those prints seems like a more natural way. \n\nThis is where you can use `iterate`. It basically \"launches\" that consuming loop for you.\n\n\n```python\nfrom lined import Line, iterize, iterate\n\npipe = Line(iterize(lambda x: x * 2), \n            iterize(lambda x: print(f\"hello {x}\")),\n            iterate\n            )\n\npipe([1, 2, 3])\n```\n\n    hello 2\n    hello 4\n    hello 6\n\n\n# Ramblings\n\n## Decorating\n\nToddlers write lines of code. \nGrown-ups write functions. Plenty of them. \n\nWhy break lines of code into small functions? Where to start...\n- It's called modularity, and that's good\n- You can reuse functions (and no, copy/paste isn't D.R.Y. -- \nand if you don't know what D.R.Y. is, \n[grow up](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself!)).\n- Because [7+-2](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two), \na.k.a [chunking](https://en.wikipedia.org/wiki/Chunking_(psychology)) or Miller's Law.\n- You can [decorate](https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators)\nfunctions, not lines of code.\n\n`lined` sets you up to take advantage of these goodies. \n\nNote this line (currently 117) of lined/base.py , in the init of Line:\n\n    self.funcs = tuple(map(fnode, self.funcs))\n\nThat is, every function is cast to with `fnode`.\n\n`fnode` is:\n\n    def fnode(func, name=None):\n        return Fnode(func, name)\n        \nand `Fnode` is just a class that \"transparently\" wraps the function. \nThis is so that we can then use `Fnode` to do all kinds of things to the function \n(without actually touching the function itself).\n\n    @dataclass\n    class Fnode:\n        func: Callable\n        __name__: Optional[str] = None\n\n    def __post_init__(self):\n        wraps(self.func)(self)\n        self.__name__ = self.__name__ or func_name(self.func)\n\n    def __call__(self, *args, **kwargs):\n        return self.func(*args, **kwargs)",
    "bugtrack_url": null,
    "license": "apache-2.0",
    "summary": "Building simple pipelines simply.",
    "version": "0.1.24",
    "split_keywords": [
        "streams",
        "pipelines"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "98e34c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf",
                "md5": "b7529ae6e59a1d4645431173b11d4fb1",
                "sha256": "23ae461ff71d803fda5932d75e90f1ce9c317493d021098fff1cafd3790c27dd"
            },
            "downloads": -1,
            "filename": "lined-0.1.24.tar.gz",
            "has_sig": false,
            "md5_digest": "b7529ae6e59a1d4645431173b11d4fb1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 34321,
            "upload_time": "2023-04-04T23:27:01",
            "upload_time_iso_8601": "2023-04-04T23:27:01.705006Z",
            "url": "https://files.pythonhosted.org/packages/98/e3/4c6deec0ed6e6abff9f498f2dcd756ba5388563e804204143c1c38d15edf/lined-0.1.24.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-04-04 23:27:01",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "otosense",
    "github_project": "lined",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "lined"
}
        
Elapsed time: 0.05194s