| Name | pieshell JSON |
| Version |
0.2.12
JSON |
| download |
| home_page | https://github.com/redhog/pieshell |
| Summary | Pieshell is a Python shell environment that combines the expressiveness of shell pipelines with the power of python iterators. It can be used both as an interactive shell and as an ordinary python module replacing e.g. subprocess.Popen |
| upload_time | 2023-11-06 15:29:03 |
| maintainer | |
| docs_url | None |
| author | Egil Moeller |
| requires_python | |
| license | GPL |
| keywords |
python
shell
pipelines
suprocess
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# About
Pieshell is a Python shell environment that combines the
expressiveness of shell pipelines with the power of python iterators.
It can be used in two major ways:
* As an interactive shell replacing e.g. bash
* As an ordinary python module replacing e.g. subprocess.Popen
# Table of contents
* [Installation](#installation)
* [As a shell](#as-a-shell)
* [Executing basic commands](#executing-basic-commands)
* [Full syntax for command lines](#full-syntax-for-command-lines)
* [Redirects](#redirects)
* [Interfacing between python functions and shell
commands](#interfacing-between-python-functions-and-shell-commands)
* [Environment variables](#environment-variables)
* [Argument expansion](#argument-expansion)
* [Processes](#processes)
* [Process browsing](#process-browsing)
* [Job control](#job-control)
* [Error handling](#error-handling)
* [Bashsource](#bashsource)
* [Asynchronous IO](#asynchronous-io)
* [As a python module](#as-a-python-module)
* [Environment variables](#environment-variables-1)
* [Argument expansion](#argument-expansion-1)
* [Pysh modules](#pysh-modules)
* [Pysh functions](#pysh-functions)
* [Eval and exec](#eval-and-exec)
* [Configuration](#configuration)
* [Changing the prompt](#changing-the-prompt)
* [Builtins](#builtins)
* [External tools](#external-tools)
* [Tips and tricks](#tips-and-tricks)
* [Unit tests](#unit-tests)
* [Copyright](#copyright)
# Installation
$ pip install pieshell
# As a shell
## Executing basic commands
To start pieshell in interactive mode, just run the command pieshell:
$ pieshell
The interactive pieshell environment supports all normal python syntax.
140:/home/redhog/Projects/beta/pieshell >>> print 3+4
7
In addition, you can run programs just like in any shell by writing
their names
140:/home/redhog/Projects/beta/pieshell >>> ls
build deps dist LICENSE.txt pieshell pieshell.egg-info README.md
setup.py
Parameters to programs however have to be given as proper python strings
within parenthesis, like a python function call
140:/home/redhog/Projects/beta/pieshell >>> ls("-a")
. .. build deps dist .git .gitignore LICENSE.txt pieshell
pieshell.egg-info .#README.md README.md setup.py
Piping the standard output of one command to the standard input of
another works just like in bash
140:/home/redhog/Projects/beta/pieshell >>> ls("-a") | grep("-e", ".py")
setup.py
Changing directory is done using the command cd:
140:/home/redhog/Projects/beta/pieshell >>> cd("..")
140:/home/redhog/Projects/beta >>>
## Full syntax for command lines
To execute commands that require a path, for example ones in the
current directory, or commands with a dot in their names
140:/home/redhog/Projects/beta/pieshell >>> _("./setup.py", "--help")
Common commands: (see '--help-commands' for more)
...
The underscore represents the virtual root command that has no
parameters, not even a command name. In general, there are two
equivalent syntaxes for parameters: as function parameter strings, and
as attribute names. The two syntaxes can be mixed freely. All of the
following are equivalent:
_("foo", "bar", "fie")
_.foo("bar", "fie")
_.foo.bar.fie()
foo.bar.fie()
foo.bar.fie
Example usage:
git.diff("-U")
In addition to these two generic syntaxes, there are two more
specialized syntaxes for options:
The function call syntax also supports named parameters, which are
converted into "--name=value" pairs. Note that the order can not be
guaranteed as named parameters are sent around as dictionaries inside
python:
git.diff(unified=4)
Short options, like `-U` above do not actually need quotes, and can be
specified inside or outside the function call syntax:
git.diff(-U, -w)
git.diff-U-w
Notes for programs with wierd parameter syntax, like `find`: `find`
does not use the standard double minus (`--`) before long options, and
takes the option value as a separate argument, rather than separating
the name and value with `=` like most programs. The above special
syntax for short options actually cover this use case too:
find(".", -name, "*.py")
## Redirects
Standard out and standard in of a pipeline can be redirected to a file
by piping to or from a string (the filename). As a special case, None
is a short hand for "/dev/null"
140:/home/redhog/Projects/beta/pieshell >>> ls | "foo"
140:/home/redhog/Projects/beta/pieshell >>> "foo" | cat
bar
build
deps
dist
foo
LICENSE.txt
pieshell
pieshell.egg-info
README.md
setup.py
140:/home/redhog/Projects/beta/pieshell >>> ls | None
Redirects can also be made with a more explicit syntax that allows
redirecting other file descriptors than stdin and stdout:
139:/home/redhog/Projects/beta/pieshell >>> cat |
Redirect("stdin", "foo") | Redirect("stdout", "bar")
The constructor for redirect takes the following arguments:
Redirect(fd, source, flag=None, mode=0777)
`fd` can be either an int, or one of `"stdin"`, `"stdout"` and `"stderr"`.
`source` is either a string filename, an int file descriptor or one of
the special values `PIPE`, `TMP` and `STRING`. `PIPE` create a new
pipe which you are required to connect up manually. `TMP` generates a
temporary named file. The filename is available in
`RunningPipeline.processes[].output_files[filedescriptornr].path`.
`STRING` generates a temporary named file, but reads and deletes it
when the pipeline exists, putting its content in
`RunningPipeline.processes[].output_content[filedescriptornr]`.
`flag` and `mode` have the same semantics as for `os.open()`. Flags do
not have to be given for `stdin`, `stdout` and `stderr` / fd `0`, `1`
and `2` and defaults to `os.O_RDONLY` or `os.O_RDONLY | os.O_CREAT`.
## Interfacing between python functions and shell commands
Shell commands are first class python objects, and their input and
output can be interacted with easily from python in the form of
iterators. Iterating over a shell command iterates over the lines of
its standard out
140:/home/redhog/Projects/beta/pieshell >>> list(ls("-a"))
['.', '..', 'build', 'deps', 'dist', '.git', '.gitignore',
'LICENSE.txt', 'pieshell', 'pieshell.egg-info', '.#README.md',
'README.md', 'setup.py']
140:/home/redhog/Projects/beta/pieshell >>> for x in ls("-a"):
... if x.endswith('.py'):
... print x
...
setup.py
Piping an iterator into a shell command, sends its items as lines to
the standard in of the shell command
140:/home/redhog/Projects/beta/pieshell >>> list(["foo", "bar.py", "fie.py"] |
grep("-e", ".py"))
['bar.py', 'fie.py']
140:/home/redhog/Projects/beta/pieshell >>> def foo():
... yield "hello"
... yield "world"
140:/home/redhog/Projects/beta/pieshell >>> foo() | cat
hello
world
In addtion, iterators and pipelines may be used as arguments to
commands and will be seen by the command as a filename, which when
opened and read from will produce the output of that iterator as
lines, or the standard output of the pipeline.
140:/home/redhog/Projects/beta/pieshell >>> list(cat(["foo", "bar"]))
['foo', 'bar']
140:/home/redhog/Projects/beta/pieshell >>> list(cat(["foo", "bar"] | cat))
['foo', 'bar']
## Environment variables
Environment variables are available directly in the shell as
variables, together with any local python variables. In addition, they
are available in the dictionary exports.
140:/home/redhog/Projects/beta/pieshell >>> LANG
'en_US.UTF-8'
Assigning to the name of an already exported environment variable
updates the value of that variable.
140:/home/redhog/Projects/beta/pieshell >>> LANG = "sv_SE.UTF-8"
140:/home/redhog/Projects/beta/pieshell >>> exports["LANG"]
'sv_SE.UTF-8'
Assigning to a variable name not already used as an environment
variable creates a local python variable.
140:/home/redhog/Projects/beta/pieshell >>> foo = "hello"
140:/home/redhog/Projects/beta/pieshell >>> "foo" in exports
False
140:/home/redhog/Projects/beta/pieshell >>> foo
'hello'
To export a new variable, you have to assign it in the exports
dictionary.
140:/home/redhog/Projects/beta/pieshell >>> exports["bar"] = "world"
140:/home/redhog/Projects/beta/pieshell >>> bar
'world'
## Argument expansion
All parameter strings in commands are subject to expansion unless
wrapped in a call to R(), e.g. R("my * string * here").
* "~" and "~username" are expanded using os.path.expanduser()
* Variable expansion is done using the python % operator on python
variables as well as environment variables.
* Pattern matching is done using glob.glob()
## Processes
A running pipeline is represented by a RunningPipeline instance. This
object is returned by the Pipeline.run() and
Pipeline.run_interactive() methods. In interactive shell mode the
RunningPipeline instance for the last executed pipeline is available
in the last_pipeline variable.
A RunningPipeline instance can be used to extract events and statuses
of the processes involved in the pipeline:
* RunningPipeline.processes is a list of RunningItem instances, each
representing an external process or a python function.
* RunningPipeline.failed_processes is a list of RunningItem instances
for those processes in the pipeline that have failed (returned a
non-zero exit status).
* RunningPipeline.pipeline is a (deep) copy of the original pipeline
object, with additional run status added, e.g. links to processes,
exit status etc.
* RunningPipeline.wait() waits for all processes in the pipeline to
terminate.
As a convenience, RunningItem:s can be retrieved by the name of their
corresponding binary as properties of the RunningPipeline. E.g.
p = ls | grep(-e, ".py")
p.grep
A RunningItem instance represents an external process or a python
function:
* RunningItem.cmd points to the part of the
RunningPipeline.pipeline structure that gave rise to this process.
* RunningItem.is_running is True if the process is still
running.
* RunningItem.is_failed is True if the process has failed somehow
(process with non-zero exit status, function threw an exception).
* RunningItem.output_content contains a dictionary of the output of
any STRING redirection for the process with the file descriptors as
keys.
* RunningProcess.iohandler.last_event contains a dictionary of the
members of the last event from the process. The members have the
same names and meaning as the members of the signalfd_siginfo
struct, see "man signalfd" for details.
## Process browsing
This functionality is only available if `psutil` is installed.
The class `PstreeProcess` represents any process not directly started
by Pieshell.
`RunningPipeline`, `RunningItem`, and `PstreeProcess` all exposes a
unified interface where child processes, parent processes, group
leaders and session leaders are available as properties. Child
porcesses uses the names of their binaries as property names. Parent
processes, session and group leaders use `PARENT`, `SESS` and `GROUP`
respectively.
`RunningItem.details`, and `PstreeProcess.details` contains a
`psutil.Process()` instance, and most of the members and methods of
that instance are directly available on the `RunningItem` or
`PstreeProcess` object.
To access a `PstreeProcess` object for the current shell and for the
init process (pid 1), there are two global variables `INIT` and
`CURRENT`.
The currently logged in users and their login session processes can be
accessed using the global `USERS` variable with a similar semantic for
properties.
All these objects support tab completion to let you easily naviage the
tree of running porcesses on your system.
redhog@glittertind:~/pieshell[master] >>> USERS.redhog.localhost.mate_panel.mate_terminal.bash.bin_bash.bash.pid_10171.python3_9
_('/home/redhog/pieshell/env/bin/python3.9', '/home/redhog/pieshell/env/bin/pieshell') as 26341
redhog@glittertind:~/pieshell[master] >>> CURRENT
_('/home/redhog/pieshell/env/bin/python3.9', '/home/redhog/pieshell/env/bin/pieshell') as 26341
## Job control
A pipeline can be started in the background by appending `&True`, or
`&None` to do the same and also redirect stdout to /dev/null.
`last_pipeline` can be used to access the backgrounded pipeline.
A running pipeline can be stopped by hitting `CTRL-Z`. A stopped
pipeline can be restarted in the background with any of
bg
bg(last_pipeline)
last_pipeline.restart()
or in the foreground with
fg
fg(last_pipeline)
last_pipeline.wait()
## Error handling
When a pipeline fails, e.g. by one of the involved processes exiting
with a non-zero status, RunningPipeline.wait() and
Pipeline.run_interactive() will throw a PipelineFailed exception after
all processes have exited.
* PipelineFailed.pipeline holds a reference to the RunningPipeline
instance that generated the exception.
If a pipeline is interrupted with CTRL-C, a PipelineInterrupted is
raised.
* PipelineInterrupted.pipeline holds a reference to the
RunningPipeline instance.
If you want to catch errors in a script, you can use normal Python
exception handling:
try:
except PipelineFailed as e:
e.pipeline.failed_processes[0].pipeline
## Bashsource
Bash provides the command `source` to run the content of a bash script inside the current shell,
effectively letting an external script update the environment variables of the running shell.
This functionality is often used for setting up local development environments, like `virtualenv`.
Pieshell provides a builtin to emulate this functionality, with bash scripts:
>>> bashsource("myscript.sh")
will run `myscript.sh` in a bash shell followed by `declare -x`. It parses the output of
`declare -x` and updates `exports` accordingly. As a special case
>>> bashsource()
will do the same, but without running any script first, esentially
just using whatever variables are set up by your `.bashrc` or
`.profile`.
## Asynchronous IO
All constructs described above to use iterators, can equally well be
used as / with async iterators. Pipelines can be awaited, which runs
them and waits for them to exit. RunningPipelines can be awaited,
which just waits for them to exit.
# As a python module
>>> from pieshell import *
All functionality available in the interactive shell is available when
using pieshell as an ordinary python module. However, a slighly more
cumbersome syntax is required.
In particular, shell commands can not be run just by writing their
names. Instead, they have to be accessed as members of the "env"
object:
>>> list(env.ls("-a") | env.grep("-e", "io"))
["iterio.py", "iterio.pyc"]
Commands are also not run with standard out to the screen when simply
printed using the repr() function but must instead be used as
iterators as is done above using the list() function.
A pipeline can be run in the background (with input/output to the screen) using
>>>> status = ~(env.ls("-a") | env.grep("-e", "io"))
or in the forground (waiting until it exists)
>>>> status = +(env.ls("-a") | env.grep("-e", "io"))
The env object holds the current working directory, which can be changed with
>>> env.cd("..")
You can also create multiple environments and use them
siumultaneously, even within the same pipeline
>>> env2 = env()
>>> env2.cd("somedir")
## Environment variables
Environment variables are available as a dictionary in env._exports.
## Argument expansion
Variable expansion is only done on environment variables, as there is
no way for pieshell to find out about the right scope to do variable
lookups in in any given situation.
## Pysh modules
In addition to being able to use pieshell code in ordinary python
modules using this slightly more verbose syntax, pieshell supports
importing modules named modulename.pysh rather than modulename.py.
Pysh modules support the full syntax of the interactive pieshell
console. Pysh modules can be imported using the standard import syntax
as soon as pieshell itself has been imported, and from the interactive
pieshell.
Be aware that you won't get a `NameError` for misspelled variable
names in pyshmodules, but instead a
`pieshell.pipeline.running.PipelineFailed` error, or if not used in a
way that will cause it to run, just a pipeline object stored somewhere
it shouldn't.
## Pysh functions
Sometimes you don't want to extract a bunch of functions to a separate
pysh module, but you'd still like to be able to use the simpler syntax
in them. That's still possible usig the `@pyshfunction` decorator:
>>> @pieshell.env.pyshfunction
>>> def foo():
... return list(echo("hello"))
You can use any environment for this, not only the default one like in
the example above.
## Eval and exec
By giving an instance of `pieshell.environ.EnvScope` with an environment
set to interactive as `globals` argument to either `eval` or `exec`, the
full syntax of Pieshell can be used in the string to evaluate/execute.
>>> scope = pieshell.environ.EnvScope(env=pieshell.env(interactive=True))
>>> eval("ls()", scope)
build deps dist LICENSE.txt pieshell pieshell.egg-info README.md
setup.py
# Configuration
When running pieshell in interactive mode it executes
`~/.config/pieshell` at startup if it exists. This file can be used to
configure the interactive environment the same way `~/.bashrc` can be
used to configure the bash shell. For example it can be used to load
python modules, execute shell pipelines or set environment variables.
An example config file is supplied in
`pieshell/resources/default_config.pysh`.
## Changing the prompt
The prompt that Pieshell uses is generated by the method
`pieshell.Environment.__str__`. This method can be overridden by
assigning a new function to it:
def prompt(self):
return "Pieshell> "
Be careful if you run pipelines or other code inside the prompt that
might call repr() as that might lead to an infinite loop of prompt
printing. To solve this, use a thread local variable as a flag to
catch and break any recursion.
# Builtins
While pieshell lets you pipe to and from ordinary python functions,
they don't offer the same syntax and tab-completion as external
commands (e.g. 'myfunction.arg1.arg2(name=value)'), they can't modify
the environment or do fancy redirects. Builtin commands provide all of
this, at the cost of a slightly clumsier syntax:
class MyMagicBuiltin(pieshell.Builtin):
"""More magic to the people
"""
name = "magic"
def _run(self, redirects, sess, indentation = ""):
# redirects is an instance of pieshell.Redirects
#
# sess is an opaque data structure that must be passed to
# any call to _run() you do yourself from this method (or
# any function it calls).
#
# indentation is a string containing only whitespace, to
# be prepended to any debug printing lines you print.
#
# Returns a list of instances of some pieshell.RunningItem
# subclass
self._cmd = self._env.find(
".", "-name", "%s.txt" % self._arg[1]) | self._env.tac
return self._cmd._run(redirects, sess, indentation)
# Optional for tab completion
def __dir__(self):
return ["light", "dark"]
Builtins need to be registered to be available in the shell. This can
be done either from python with
pipeline.BuiltinRegistry.register(MyMagicBuiltin)
or from setup.py in any package:
setup(
entry_points={
'pieshell.builtin': [
"magic = mypackage:MyMagicBuiltin",
]
}
)
# External tools
A short list of tools that might be usefull together with this project:
* [psutil](http://pythonhosted.org/psutil) - python api for getting ps / top style information about a process
* [ReRedirect](https://github.com/jerome-pouiller/reredirect) - redirect io for an already running process
* [Reptyr](https://github.com/nelhage/reptyr) - move a running process to a new controlling terminal
* [Deptyr](https://github.com/antifuchs/deptyr) - forward output for a new process to another controlling terminal
# Tips and tricks
The [pandas reader for a table of fixed-width formatted
lines](https://pandas.pydata.org/docs/reference/api/pandas.read_fwf.html#pandas.read_fwf)
does a pretty good job at parsing the output from several standard
command line tools such as ls(-l):
>>> pd.read_fwf(io.StringIO(str(ls(-l))))
total 328 Unnamed: 1 Unnamed: 2 Unnamed: 3 Unnamed: 4 Unnamed: 5 Unnamed: 6 Unnamed: 7 Unnamed: 8
0 drwxr-xr-x 6 redhog redhog 4096 okt. 16 21:45 build
1 -rw-rw-r-- 1 redhog redhog 25 jan. 11 2021 _config.yml
2 drwxrwxr-x 2 redhog redhog 4096 okt. 25 21:30 dist
3
>>> pd.read_fwf(io.StringIO(str(docker.images)))
REPOSITORY TAG IMAGE ID CREATED SIZE
0 python 3 a8405b7e74cf 7 months ago 921MB
1 ubuntu 22.04 08d22c0ceb15 7 months ago 77.8MB
2 ubuntu latest d13c942271d6 21 months ago 72.8MB
# Unit tests
To run the unit- and integration tests
pip install nose2
nose2 -s tests
# Copyright
Pieshell copyright 2016 Egil Möller <egil.moller@piratpartiet.se>
Pieshell is free software: you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this program. If not, see
<http://www.gnu.org/licenses/>.
Raw data
{
"_id": null,
"home_page": "https://github.com/redhog/pieshell",
"name": "pieshell",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "Python shell pipelines suprocess",
"author": "Egil Moeller",
"author_email": "egil.moller@piratpartiet.se",
"download_url": "https://files.pythonhosted.org/packages/45/97/2f1ae9488e26471285fb72bcdab1f642bf732e5e02a9200d51f980e1c603/pieshell-0.2.12.tar.gz",
"platform": null,
"description": "\n\n \n\n\n\n# About\n\nPieshell is a Python shell environment that combines the\nexpressiveness of shell pipelines with the power of python iterators.\n\nIt can be used in two major ways:\n\n* As an interactive shell replacing e.g. bash\n* As an ordinary python module replacing e.g. subprocess.Popen\n\n# Table of contents\n\n* [Installation](#installation)\n* [As a shell](#as-a-shell)\n * [Executing basic commands](#executing-basic-commands)\n * [Full syntax for command lines](#full-syntax-for-command-lines)\n * [Redirects](#redirects)\n * [Interfacing between python functions and shell\n commands](#interfacing-between-python-functions-and-shell-commands)\n * [Environment variables](#environment-variables)\n * [Argument expansion](#argument-expansion)\n * [Processes](#processes)\n * [Process browsing](#process-browsing)\n * [Job control](#job-control)\n * [Error handling](#error-handling)\n * [Bashsource](#bashsource)\n * [Asynchronous IO](#asynchronous-io)\n* [As a python module](#as-a-python-module)\n * [Environment variables](#environment-variables-1)\n * [Argument expansion](#argument-expansion-1)\n * [Pysh modules](#pysh-modules)\n * [Pysh functions](#pysh-functions)\n * [Eval and exec](#eval-and-exec)\n* [Configuration](#configuration)\n * [Changing the prompt](#changing-the-prompt)\n* [Builtins](#builtins)\n* [External tools](#external-tools)\n* [Tips and tricks](#tips-and-tricks)\n* [Unit tests](#unit-tests)\n* [Copyright](#copyright)\n\n\n# Installation\n\n $ pip install pieshell\n\n\n# As a shell\n\n## Executing basic commands\n\nTo start pieshell in interactive mode, just run the command pieshell:\n\n $ pieshell\n\nThe interactive pieshell environment supports all normal python syntax.\n\n 140:/home/redhog/Projects/beta/pieshell >>> print 3+4\n 7\n\nIn addition, you can run programs just like in any shell by writing\ntheir names\n\n 140:/home/redhog/Projects/beta/pieshell >>> ls\n build deps dist LICENSE.txt pieshell pieshell.egg-info README.md\n setup.py\n\nParameters to programs however have to be given as proper python strings\nwithin parenthesis, like a python function call\n\n 140:/home/redhog/Projects/beta/pieshell >>> ls(\"-a\")\n . .. build deps dist .git .gitignore LICENSE.txt pieshell\n pieshell.egg-info .#README.md README.md setup.py\n\nPiping the standard output of one command to the standard input of\nanother works just like in bash\n\n 140:/home/redhog/Projects/beta/pieshell >>> ls(\"-a\") | grep(\"-e\", \".py\")\n setup.py\n\nChanging directory is done using the command cd:\n\n 140:/home/redhog/Projects/beta/pieshell >>> cd(\"..\")\n 140:/home/redhog/Projects/beta >>> \n\n## Full syntax for command lines\n\nTo execute commands that require a path, for example ones in the\ncurrent directory, or commands with a dot in their names\n\n 140:/home/redhog/Projects/beta/pieshell >>> _(\"./setup.py\", \"--help\")\n Common commands: (see '--help-commands' for more)\n ...\n\nThe underscore represents the virtual root command that has no\nparameters, not even a command name. In general, there are two\nequivalent syntaxes for parameters: as function parameter strings, and\nas attribute names. The two syntaxes can be mixed freely. All of the\nfollowing are equivalent:\n\n _(\"foo\", \"bar\", \"fie\")\n _.foo(\"bar\", \"fie\")\n _.foo.bar.fie()\n foo.bar.fie()\n foo.bar.fie\n\nExample usage:\n\n git.diff(\"-U\")\n\nIn addition to these two generic syntaxes, there are two more\nspecialized syntaxes for options:\n\nThe function call syntax also supports named parameters, which are\nconverted into \"--name=value\" pairs. Note that the order can not be\nguaranteed as named parameters are sent around as dictionaries inside\npython:\n\n git.diff(unified=4)\n\nShort options, like `-U` above do not actually need quotes, and can be\nspecified inside or outside the function call syntax:\n\n git.diff(-U, -w)\n git.diff-U-w\n\nNotes for programs with wierd parameter syntax, like `find`: `find`\ndoes not use the standard double minus (`--`) before long options, and\ntakes the option value as a separate argument, rather than separating\nthe name and value with `=` like most programs. The above special\nsyntax for short options actually cover this use case too:\n\n find(\".\", -name, \"*.py\")\n\n## Redirects\n\nStandard out and standard in of a pipeline can be redirected to a file\nby piping to or from a string (the filename). As a special case, None\nis a short hand for \"/dev/null\"\n\n 140:/home/redhog/Projects/beta/pieshell >>> ls | \"foo\"\n\n 140:/home/redhog/Projects/beta/pieshell >>> \"foo\" | cat\n bar\n build\n deps\n dist\n foo\n LICENSE.txt\n pieshell\n pieshell.egg-info\n README.md\n setup.py\n\n 140:/home/redhog/Projects/beta/pieshell >>> ls | None\n\nRedirects can also be made with a more explicit syntax that allows\nredirecting other file descriptors than stdin and stdout:\n\n 139:/home/redhog/Projects/beta/pieshell >>> cat |\n Redirect(\"stdin\", \"foo\") | Redirect(\"stdout\", \"bar\")\n\nThe constructor for redirect takes the following arguments:\n\n Redirect(fd, source, flag=None, mode=0777)\n\n`fd` can be either an int, or one of `\"stdin\"`, `\"stdout\"` and `\"stderr\"`.\n\n`source` is either a string filename, an int file descriptor or one of\nthe special values `PIPE`, `TMP` and `STRING`. `PIPE` create a new\npipe which you are required to connect up manually. `TMP` generates a\ntemporary named file. The filename is available in\n`RunningPipeline.processes[].output_files[filedescriptornr].path`.\n`STRING` generates a temporary named file, but reads and deletes it\nwhen the pipeline exists, putting its content in\n`RunningPipeline.processes[].output_content[filedescriptornr]`.\n\n`flag` and `mode` have the same semantics as for `os.open()`. Flags do\nnot have to be given for `stdin`, `stdout` and `stderr` / fd `0`, `1`\nand `2` and defaults to `os.O_RDONLY` or `os.O_RDONLY | os.O_CREAT`.\n\n## Interfacing between python functions and shell commands\n\nShell commands are first class python objects, and their input and\noutput can be interacted with easily from python in the form of\niterators. Iterating over a shell command iterates over the lines of\nits standard out\n\n 140:/home/redhog/Projects/beta/pieshell >>> list(ls(\"-a\"))\n ['.', '..', 'build', 'deps', 'dist', '.git', '.gitignore',\n 'LICENSE.txt', 'pieshell', 'pieshell.egg-info', '.#README.md',\n 'README.md', 'setup.py']\n 140:/home/redhog/Projects/beta/pieshell >>> for x in ls(\"-a\"):\n ... if x.endswith('.py'):\n ... print x\n ... \n setup.py\n\nPiping an iterator into a shell command, sends its items as lines to\nthe standard in of the shell command\n\n 140:/home/redhog/Projects/beta/pieshell >>> list([\"foo\", \"bar.py\", \"fie.py\"] |\n grep(\"-e\", \".py\"))\n ['bar.py', 'fie.py']\n 140:/home/redhog/Projects/beta/pieshell >>> def foo():\n ... yield \"hello\"\n ... yield \"world\"\n 140:/home/redhog/Projects/beta/pieshell >>> foo() | cat\n hello\n world\n\nIn addtion, iterators and pipelines may be used as arguments to\ncommands and will be seen by the command as a filename, which when\nopened and read from will produce the output of that iterator as\nlines, or the standard output of the pipeline.\n\n 140:/home/redhog/Projects/beta/pieshell >>> list(cat([\"foo\", \"bar\"]))\n ['foo', 'bar']\n 140:/home/redhog/Projects/beta/pieshell >>> list(cat([\"foo\", \"bar\"] | cat))\n ['foo', 'bar']\n\n## Environment variables\n\nEnvironment variables are available directly in the shell as\nvariables, together with any local python variables. In addition, they\nare available in the dictionary exports.\n\n 140:/home/redhog/Projects/beta/pieshell >>> LANG\n 'en_US.UTF-8'\n\nAssigning to the name of an already exported environment variable\nupdates the value of that variable.\n\n 140:/home/redhog/Projects/beta/pieshell >>> LANG = \"sv_SE.UTF-8\"\n 140:/home/redhog/Projects/beta/pieshell >>> exports[\"LANG\"]\n 'sv_SE.UTF-8'\n\nAssigning to a variable name not already used as an environment\nvariable creates a local python variable.\n\n 140:/home/redhog/Projects/beta/pieshell >>> foo = \"hello\"\n 140:/home/redhog/Projects/beta/pieshell >>> \"foo\" in exports\n False\n 140:/home/redhog/Projects/beta/pieshell >>> foo\n 'hello'\n\nTo export a new variable, you have to assign it in the exports\ndictionary.\n\n 140:/home/redhog/Projects/beta/pieshell >>> exports[\"bar\"] = \"world\"\n 140:/home/redhog/Projects/beta/pieshell >>> bar\n 'world'\n\n## Argument expansion\n\nAll parameter strings in commands are subject to expansion unless\nwrapped in a call to R(), e.g. R(\"my * string * here\").\n\n * \"~\" and \"~username\" are expanded using os.path.expanduser()\n\n * Variable expansion is done using the python % operator on python\n variables as well as environment variables.\n\n * Pattern matching is done using glob.glob()\n\n## Processes\n\nA running pipeline is represented by a RunningPipeline instance. This\nobject is returned by the Pipeline.run() and\nPipeline.run_interactive() methods. In interactive shell mode the\nRunningPipeline instance for the last executed pipeline is available\nin the last_pipeline variable.\n\nA RunningPipeline instance can be used to extract events and statuses\nof the processes involved in the pipeline:\n\n* RunningPipeline.processes is a list of RunningItem instances, each\n representing an external process or a python function.\n\n* RunningPipeline.failed_processes is a list of RunningItem instances\n for those processes in the pipeline that have failed (returned a\n non-zero exit status).\n\n* RunningPipeline.pipeline is a (deep) copy of the original pipeline\n object, with additional run status added, e.g. links to processes,\n exit status etc.\n\n* RunningPipeline.wait() waits for all processes in the pipeline to\n terminate.\n\nAs a convenience, RunningItem:s can be retrieved by the name of their\ncorresponding binary as properties of the RunningPipeline. E.g.\n\n p = ls | grep(-e, \".py\")\n p.grep\n\nA RunningItem instance represents an external process or a python\nfunction:\n\n* RunningItem.cmd points to the part of the\n RunningPipeline.pipeline structure that gave rise to this process.\n\n* RunningItem.is_running is True if the process is still\n running.\n\n* RunningItem.is_failed is True if the process has failed somehow\n (process with non-zero exit status, function threw an exception).\n\n* RunningItem.output_content contains a dictionary of the output of\n any STRING redirection for the process with the file descriptors as\n keys.\n\n* RunningProcess.iohandler.last_event contains a dictionary of the\n members of the last event from the process. The members have the\n same names and meaning as the members of the signalfd_siginfo\n struct, see \"man signalfd\" for details.\n\n## Process browsing\n\nThis functionality is only available if `psutil` is installed.\n\nThe class `PstreeProcess` represents any process not directly started\nby Pieshell.\n\n`RunningPipeline`, `RunningItem`, and `PstreeProcess` all exposes a\nunified interface where child processes, parent processes, group\nleaders and session leaders are available as properties. Child\nporcesses uses the names of their binaries as property names. Parent\nprocesses, session and group leaders use `PARENT`, `SESS` and `GROUP`\nrespectively.\n\n`RunningItem.details`, and `PstreeProcess.details` contains a\n`psutil.Process()` instance, and most of the members and methods of\nthat instance are directly available on the `RunningItem` or\n`PstreeProcess` object.\n\nTo access a `PstreeProcess` object for the current shell and for the\ninit process (pid 1), there are two global variables `INIT` and\n`CURRENT`.\n\nThe currently logged in users and their login session processes can be\naccessed using the global `USERS` variable with a similar semantic for\nproperties.\n\nAll these objects support tab completion to let you easily naviage the\ntree of running porcesses on your system.\n\n redhog@glittertind:~/pieshell[master] >>> USERS.redhog.localhost.mate_panel.mate_terminal.bash.bin_bash.bash.pid_10171.python3_9\n _('/home/redhog/pieshell/env/bin/python3.9', '/home/redhog/pieshell/env/bin/pieshell') as 26341\n \n redhog@glittertind:~/pieshell[master] >>> CURRENT\n _('/home/redhog/pieshell/env/bin/python3.9', '/home/redhog/pieshell/env/bin/pieshell') as 26341\n\n\n## Job control\n\nA pipeline can be started in the background by appending `&True`, or\n`&None` to do the same and also redirect stdout to /dev/null.\n\n`last_pipeline` can be used to access the backgrounded pipeline.\n\nA running pipeline can be stopped by hitting `CTRL-Z`. A stopped\npipeline can be restarted in the background with any of\n\n bg\n bg(last_pipeline)\n last_pipeline.restart()\n\nor in the foreground with\n\n fg\n fg(last_pipeline)\n last_pipeline.wait()\n\n## Error handling\n\nWhen a pipeline fails, e.g. by one of the involved processes exiting\nwith a non-zero status, RunningPipeline.wait() and\nPipeline.run_interactive() will throw a PipelineFailed exception after\nall processes have exited.\n\n* PipelineFailed.pipeline holds a reference to the RunningPipeline\n instance that generated the exception.\n\nIf a pipeline is interrupted with CTRL-C, a PipelineInterrupted is\nraised.\n\n* PipelineInterrupted.pipeline holds a reference to the\n RunningPipeline instance.\n\nIf you want to catch errors in a script, you can use normal Python\nexception handling:\n\n try:\n except PipelineFailed as e:\n e.pipeline.failed_processes[0].pipeline\n\n## Bashsource\n\nBash provides the command `source` to run the content of a bash script inside the current shell,\neffectively letting an external script update the environment variables of the running shell.\nThis functionality is often used for setting up local development environments, like `virtualenv`.\n\nPieshell provides a builtin to emulate this functionality, with bash scripts:\n\n >>> bashsource(\"myscript.sh\")\n\nwill run `myscript.sh` in a bash shell followed by `declare -x`. It parses the output of\n`declare -x` and updates `exports` accordingly. As a special case\n\n >>> bashsource()\n\nwill do the same, but without running any script first, esentially\njust using whatever variables are set up by your `.bashrc` or\n`.profile`.\n\n## Asynchronous IO\n\nAll constructs described above to use iterators, can equally well be\nused as / with async iterators. Pipelines can be awaited, which runs\nthem and waits for them to exit. RunningPipelines can be awaited,\nwhich just waits for them to exit.\n\n# As a python module\n\n >>> from pieshell import *\n\nAll functionality available in the interactive shell is available when\nusing pieshell as an ordinary python module. However, a slighly more\ncumbersome syntax is required.\n\nIn particular, shell commands can not be run just by writing their\nnames. Instead, they have to be accessed as members of the \"env\"\nobject:\n\n >>> list(env.ls(\"-a\") | env.grep(\"-e\", \"io\"))\n [\"iterio.py\", \"iterio.pyc\"]\n\nCommands are also not run with standard out to the screen when simply\nprinted using the repr() function but must instead be used as\niterators as is done above using the list() function.\n\nA pipeline can be run in the background (with input/output to the screen) using\n\n >>>> status = ~(env.ls(\"-a\") | env.grep(\"-e\", \"io\"))\n\nor in the forground (waiting until it exists)\n\n >>>> status = +(env.ls(\"-a\") | env.grep(\"-e\", \"io\"))\n\n\nThe env object holds the current working directory, which can be changed with\n\n >>> env.cd(\"..\") \n\nYou can also create multiple environments and use them\nsiumultaneously, even within the same pipeline\n\n >>> env2 = env()\n >>> env2.cd(\"somedir\")\n\n## Environment variables\n\nEnvironment variables are available as a dictionary in env._exports.\n\n## Argument expansion\n\nVariable expansion is only done on environment variables, as there is\nno way for pieshell to find out about the right scope to do variable\nlookups in in any given situation.\n\n## Pysh modules\n\nIn addition to being able to use pieshell code in ordinary python\nmodules using this slightly more verbose syntax, pieshell supports\nimporting modules named modulename.pysh rather than modulename.py.\nPysh modules support the full syntax of the interactive pieshell\nconsole. Pysh modules can be imported using the standard import syntax\nas soon as pieshell itself has been imported, and from the interactive\npieshell.\n\nBe aware that you won't get a `NameError` for misspelled variable\nnames in pyshmodules, but instead a\n`pieshell.pipeline.running.PipelineFailed` error, or if not used in a\nway that will cause it to run, just a pipeline object stored somewhere\nit shouldn't.\n\n## Pysh functions\n\nSometimes you don't want to extract a bunch of functions to a separate\npysh module, but you'd still like to be able to use the simpler syntax\nin them. That's still possible usig the `@pyshfunction` decorator:\n\n >>> @pieshell.env.pyshfunction\n >>> def foo():\n ... return list(echo(\"hello\"))\n\nYou can use any environment for this, not only the default one like in\nthe example above.\n\n## Eval and exec\n\nBy giving an instance of `pieshell.environ.EnvScope` with an environment\nset to interactive as `globals` argument to either `eval` or `exec`, the\nfull syntax of Pieshell can be used in the string to evaluate/execute.\n\n >>> scope = pieshell.environ.EnvScope(env=pieshell.env(interactive=True))\n >>> eval(\"ls()\", scope)\n build deps dist LICENSE.txt pieshell pieshell.egg-info README.md\n setup.py\n\n# Configuration\n\nWhen running pieshell in interactive mode it executes\n`~/.config/pieshell` at startup if it exists. This file can be used to\nconfigure the interactive environment the same way `~/.bashrc` can be\nused to configure the bash shell. For example it can be used to load\npython modules, execute shell pipelines or set environment variables.\nAn example config file is supplied in\n`pieshell/resources/default_config.pysh`.\n\n## Changing the prompt\n\nThe prompt that Pieshell uses is generated by the method\n`pieshell.Environment.__str__`. This method can be overridden by\nassigning a new function to it:\n\n def prompt(self):\n return \"Pieshell> \"\n\nBe careful if you run pipelines or other code inside the prompt that\nmight call repr() as that might lead to an infinite loop of prompt\nprinting. To solve this, use a thread local variable as a flag to\ncatch and break any recursion.\n\n# Builtins\n\nWhile pieshell lets you pipe to and from ordinary python functions,\nthey don't offer the same syntax and tab-completion as external\ncommands (e.g. 'myfunction.arg1.arg2(name=value)'), they can't modify\nthe environment or do fancy redirects. Builtin commands provide all of\nthis, at the cost of a slightly clumsier syntax:\n\n class MyMagicBuiltin(pieshell.Builtin):\n \"\"\"More magic to the people\n \"\"\"\n name = \"magic\"\n\n def _run(self, redirects, sess, indentation = \"\"):\n # redirects is an instance of pieshell.Redirects\n #\n # sess is an opaque data structure that must be passed to\n # any call to _run() you do yourself from this method (or\n # any function it calls).\n #\n # indentation is a string containing only whitespace, to\n # be prepended to any debug printing lines you print.\n #\n # Returns a list of instances of some pieshell.RunningItem\n # subclass\n\n self._cmd = self._env.find(\n \".\", \"-name\", \"%s.txt\" % self._arg[1]) | self._env.tac\n return self._cmd._run(redirects, sess, indentation)\n\n\n # Optional for tab completion\n def __dir__(self):\n return [\"light\", \"dark\"]\n\nBuiltins need to be registered to be available in the shell. This can\nbe done either from python with\n\n pipeline.BuiltinRegistry.register(MyMagicBuiltin)\n\nor from setup.py in any package:\n \n setup(\n entry_points={\n 'pieshell.builtin': [\n \"magic = mypackage:MyMagicBuiltin\",\n ]\n }\n )\n\n# External tools\n\nA short list of tools that might be usefull together with this project:\n\n* [psutil](http://pythonhosted.org/psutil) - python api for getting ps / top style information about a process\n* [ReRedirect](https://github.com/jerome-pouiller/reredirect) - redirect io for an already running process\n* [Reptyr](https://github.com/nelhage/reptyr) - move a running process to a new controlling terminal\n* [Deptyr](https://github.com/antifuchs/deptyr) - forward output for a new process to another controlling terminal\n\n# Tips and tricks\n\nThe [pandas reader for a table of fixed-width formatted\nlines](https://pandas.pydata.org/docs/reference/api/pandas.read_fwf.html#pandas.read_fwf)\ndoes a pretty good job at parsing the output from several standard\ncommand line tools such as ls(-l):\n\n >>> pd.read_fwf(io.StringIO(str(ls(-l))))\n total 328 Unnamed: 1 Unnamed: 2 Unnamed: 3 Unnamed: 4 Unnamed: 5 Unnamed: 6 Unnamed: 7 Unnamed: 8\n 0 drwxr-xr-x 6 redhog redhog 4096 okt. 16 21:45 build\n 1 -rw-rw-r-- 1 redhog redhog 25 jan. 11 2021 _config.yml\n 2 drwxrwxr-x 2 redhog redhog 4096 okt. 25 21:30 dist\n 3\n\n >>> pd.read_fwf(io.StringIO(str(docker.images)))\n REPOSITORY TAG IMAGE ID CREATED SIZE\n 0 python 3 a8405b7e74cf 7 months ago 921MB\n 1 ubuntu 22.04 08d22c0ceb15 7 months ago 77.8MB\n 2 ubuntu latest d13c942271d6 21 months ago 72.8MB\n\n\n# Unit tests\n\nTo run the unit- and integration tests\n\n pip install nose2\n nose2 -s tests\n\n# Copyright\n\nPieshell copyright 2016 Egil M\u00f6ller <egil.moller@piratpartiet.se>\n\nPieshell is free software: you can redistribute it and/or modify it\nunder the terms of the GNU Lesser General Public License as published\nby the Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU Lesser General Public\nLicense along with this program. If not, see\n<http://www.gnu.org/licenses/>.",
"bugtrack_url": null,
"license": "GPL",
"summary": "Pieshell is a Python shell environment that combines the expressiveness of shell pipelines with the power of python iterators. It can be used both as an interactive shell and as an ordinary python module replacing e.g. subprocess.Popen",
"version": "0.2.12",
"project_urls": {
"Homepage": "https://github.com/redhog/pieshell"
},
"split_keywords": [
"python",
"shell",
"pipelines",
"suprocess"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "45972f1ae9488e26471285fb72bcdab1f642bf732e5e02a9200d51f980e1c603",
"md5": "e46caafca0310abedd05416dcd78f757",
"sha256": "edcd4717459edcc2c90f91dad314c33a6660610234a750776dc7f593946f4dcb"
},
"downloads": -1,
"filename": "pieshell-0.2.12.tar.gz",
"has_sig": false,
"md5_digest": "e46caafca0310abedd05416dcd78f757",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 59017,
"upload_time": "2023-11-06T15:29:03",
"upload_time_iso_8601": "2023-11-06T15:29:03.277944Z",
"url": "https://files.pythonhosted.org/packages/45/97/2f1ae9488e26471285fb72bcdab1f642bf732e5e02a9200d51f980e1c603/pieshell-0.2.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-06 15:29:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "redhog",
"github_project": "pieshell",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pieshell"
}
