# command_runner
# Platform agnostic command execution, timed background jobs with live stdout/stderr output capture, and UAC/sudo elevation
[](https://opensource.org/licenses/BSD-3-Clause)
[](http://isitmaintained.com/project/netinvent/command_runner "Percentage of issues still open")
[](https://codeclimate.com/github/netinvent/command_runner/maintainability)
[](https://codecov.io/gh/netinvent/command_runner)
[](https://github.com/netinvent/command_runner/actions/workflows/linux.yaml)
[](https://github.com/netinvent/command_runner/actions/workflows/windows.yaml)
[](https://github.com/netinvent/command_runner/releases/latest)
command_runner's purpose is to run external commands from python, just like subprocess on which it relies,
while solving various problems a developer may face among:
- Handling of all possible subprocess.popen / subprocess.check_output scenarios / python versions in one handy function without encoding / timeout hassle
- Allow stdout/stderr stream output to be redirected to callback functions / output queues / files so you get to handle output in your application while commands are running
- Callback to optional stop check so we can stop execution from outside command_runner
- Callback with optional process information so we get to control the process from outside command_runner
- Callback once we're finished to easen thread usage
- Optional process priority and io_priority settings
- System agnostic functionality, the developer shouldn't carry the burden of Windows & Linux differences
- Optional Windows UAC elevation module compatible with CPython, PyInstaller & Nuitka
- Optional Linux sudo elevation compatible with CPython, PyInstaller & Nuitka
It is compatible with Python 2.7+, tested up to Python 3.11 (backports some newer Python 3.5 functionality) and is tested on both Linux and Windows.
It is also compatible with PyPy Python implementation.
...and yes, keeping Python 2.7 compatibility has proven to be quite challenging.
## command_runner
command_runner is a replacement package for subprocess.popen and subprocess.check_output
The main promise command_runner can do is to make sure to never have a blocking command, and always get results.
It works as wrapper for subprocess.popen and subprocess.communicate that solves:
- Platform differences
- Handle timeouts even for windows GUI applications that don't return anything to stdout
- Python language version differences
- Handle timeouts even on earlier Python implementations
- Handle encoding even on earlier Python implementations
- Keep the promise to always return an exit code (so we don't have to deal with exit codes and exception logic at the same time)
- Keep the promise to always return the command output regardless of the execution state (even with timeouts, callback interrupts and keyboard interrupts)
- Can show command output on the fly without waiting the end of execution (with `live_output=True` argument)
- Can give command output on the fly to application by using queues or callback functions
- Catch all possible exceptions and log them properly with encoding fixes
- Be compatible, and always return the same result regarless of platform
command_runner also promises to properly kill commands when timeouts are reached, including spawned subprocesses of such commands.
This specific behavior is achieved via psutil module, which is an optional dependency.
### command_runner in a nutshell
Install with `pip install command_runner`
The following example will work regardless of the host OS and the Python version.
```python
from command_runner import command_runner
exit_code, output = command_runner('ping 127.0.0.1', timeout=10)
```
## Guide to command_runner
### Setup
`pip install command_runner` or download the latest git release
### Advanced command_runner usage
#### Special exit codes
In order to keep the promise to always provide an exit_code, spcial exit codes have been added for the case where none is given.
Those exit codes are:
- -250 : command_runner called with incompatible arguments
- -251 : stop_on function returned True
- -252 : KeyboardInterrupt
- -253 : FileNotFoundError, OSError, IOError
- -254 : Timeout
- -255 : Any other uncatched exceptions
This allows you to use the standard exit code logic, without having to deal with various exceptions.
#### Default encoding
command_runner has an `encoding` argument which defaults to `utf-8` for Unixes and `cp437` for Windows platforms.
Using `cp437` ensures that most `cmd.exe` output is encoded properly, including accents and special characters, on most locale systems.
Still you can specify your own encoding for other usages, like Powershell where `unicode_escape` is preferred.
```python
from command_runner import command_runner
command = r'C:\Windows\sysnative\WindowsPowerShell\v1.0\powershell.exe --help'
exit_code, output = command_runner(command, encoding='unicode_escape')
```
Earlier subprocess.popen implementations didn't have an encoding setting so command_runner will deal with encoding for those.
You can also disable command_runner's internal encoding in order to get raw process output (bytes) by passing False boolean.
Example:
```python
from command_runner import command_runner
exit_code, raw_output = command_runner('ping 127.0.0.1', encoding=False)
```
#### On the fly (interactive screen) output
**Note: for live output capture and threading, see stream redirection. If you want to run your application while command_runner gives back command output, the best way to go is queues / callbacks.**
command_runner can output a command output on the fly to stdout, eg show output on screen during execution.
This is helpful when the command is long, and we need to know the output while execution is ongoing.
It is also helpful in order to catch partial command output when timeout is reached or a CTRL+C signal is received.
Example:
```python
from command_runner import command_runner
exit_code, output = command_runner('ping 127.0.0.1', shell=True, live_output=True)
```
Note: using live output relies on stdout pipe polling, which has lightly higher cpu usage.
#### Timeouts
**command_runner has a `timeout` argument which defaults to 3600 seconds.**
This default setting ensures commands will not block the main script execution.
Feel free to lower / higher that setting with `timeout` argument.
Note that a command_runner kills the whole process tree that the command may have generated, even under Windows.
```python
from command_runner import command_runner
exit_code, output = command_runner('ping 127.0.0.1', timeout=30)
```
#### Remarks on processes
Using `shell=True` will spawn a shell which will spawn the desired child process.
Be aware that under MS Windows, no direct process tree is available.
We fixed this by walking processes during runtime. The drawback is that orphaned processes cannot be identified this way.
#### Disabling logs / silencing
`command_runner` has it's own logging system, which will log all sorts of error logs.
If you need to disable it's logging, just run with argument silent.
Be aware that logging.DEBUG log levels won't be silenced, by design.
Example:
```python
from command_runner import command_runner
exit_code, output = command_runner('ping 127.0.0.1', silent=True)
```
If you also need to disable logging.DEBUG level, you can run the following code which will required logging.CRITICAL only messages which `command_runner` never does:
```python
import logging
import command_runner
logging.getLogger('command_runner').setLevel(logging.CRITICAL)
```
#### Capture method
`command_runner` allows two different process output capture methods:
`method='monitor'` which is default:
- A thread is spawned in order to check stop conditions and kill process if needed
- A main loop waits for the process to finish, then uses proc.communicate() to get it's output
- Pros:
- less CPU usage
- less threads
- Cons:
- cannot read partial output on KeyboardInterrupt or stop_on (still works for partial timeout output)
- cannot use queues or callback functions redirectors
- is 0.1 seconds slower than poller method
`method='poller'`:
- A thread is spawned and reads stdout/stderr pipes into output queues
- A poller loop reads from the output queues, checks stop conditions and kills process if needed
- Pros:
- Reads on the fly, allowing interactive commands (is also used with `live_output=True`)
- Allows stdout/stderr output to be written live to callback functions, queues or files (useful when threaded)
- is 0.1 seconds faster than monitor method, is preferred method for fast batch runnings
- Cons:
- lightly higher CPU usage
Example:
```python
from command_runner import command_runner
exit_code, output = command_runner('ping 127.0.0.1', method='poller')
exit_code, output = command_runner('ping 127.0.0.1', method='monitor')
```
#### stdin stream redirection
`command_runner` allows to redirect some stream directly into the subprocess it spawns.
Example code
```python
import sys
from command_runner import command_runner
exit_code, output = command_runner("gzip -d", stdin=sys.stdin.buffer)
print("Uncompressed data", output)
```
The above program, when run with `echo "Hello, World!" | gzip | python myscript.py` will show the uncompressed string `Hello, World!`
You can use whatever file descriptor you want, basic ones being sys.stdin for text input and sys.stdin.buffer for binary input.
#### stdout / stderr stream redirection
command_runner can redirect stdout and/or stderr streams to different outputs:
- subprocess pipes
- /dev/null or NUL
- files
- queues
- callback functions
Unless an output redirector is given for `stderr` argument, stderr will be redirected to `stdout` stream.
Note that both queues and callback function redirectors require `poller` method and will fail if method is not set.
Possible output redirection options are:
- subprocess pipes
By default, stdout writes into a subprocess.PIPE which is read by command_runner and returned as `output` variable.
You may also pass any other subprocess.PIPE int values to `stdout` or `stderr` arguments.
- /dev/null or NUL
If `stdout=False` and/or `stderr=False` argument(s) are given, command output will not be saved.
stdout/stderr streams will be redirected to `/dev/null` or `NUL` depending on platform.
Output will always be `None`. See `split_streams` for more details using multiple outputs.
- files
Giving `stdout` and/or `stderr` arguments a string, `command_runner` will consider the string to be a file path where stream output will be written live.
Examples:
```python
from command_runner import command_runner
exit_code, output = command_runner('dir', stdout=r"C:/tmp/command_result", stderr=r"C:/tmp/command_error", shell=True)
```
```python
from command_runner import command_runner
exit_code, output = command_runner('dir', stdout='/tmp/stdout.log', stderr='/tmp/stderr.log', shell=True)
```
Opening a file with the wrong encoding (especially opening a CP437 encoded file on Windows with UTF-8 coded might endup with UnicodedecodeError.)
- queues
Queue(s) will be filled up by command_runner.
In order to keep your program "live", we'll use the threaded version of command_runner which is basically the same except it returns a future result instead of a tuple.
Note: With all the best will, there's no good way to achieve this under Python 2.7 without using more queues, so the threaded version is only compatible with Python 3.3+.
For Python 2.7, you must create your thread and queue reader yourself (see footnote for a Python 2.7 comaptible example).
Threaded command_runner plus queue example:
```python
import queue
from command_runner import command_runner_threaded
output_queue = queue.Queue()
stream_output = ""
thread_result = command_runner_threaded('ping 127.0.0.1', shell=True, method='poller', stdout=output_queue)
read_queue = True
while read_queue:
try:
line = output_queue.get(timeout=0.1)
except queue.Empty:
pass
else:
if line is None:
read_queue = False
else:
stream_output += line
# ADD YOUR LIVE CODE HERE
# Now we may get exit_code and output since result has become available at this point
exit_code, output = thread_result.result()
```
You might also want to read both stdout and stderr queues. In that case, you can create a read loop just like in the following example.
Here we're reading both queues in one loop, so we need to observe a couple of conditions before stopping the loop, in order to catch all queue output:
```python
import queue
from time import sleep
from command_runner import command_runner_threaded
stdout_queue = queue.Queue()
stderr_queue = queue.Queue()
thread_result = command_runner_threaded('ping 127.0.0.1', method='poller', shell=True, stdout=stdout_queue, stderr=stderr_queue)
read_stdout = read_stderr = True
while read_stdout or read_stderr:
try:
stdout_line = stdout_queue.get(timeout=0.1)
except queue.Empty:
pass
else:
if stdout_line is None:
read_stdout = False
else:
print('STDOUT:', stdout_line)
try:
stderr_line = stderr_queue.get(timeout=0.1)
except queue.Empty:
pass
else:
if stderr_line is None:
read_stderr = False
else:
print('STDERR:', stderr_line)
# ADD YOUR LIVE CODE HERE
exit_code, output = thread_result.result()
assert exit_code == 0, 'We did not succeed in running the thread'
```
- callback functions
The callback function will get one argument, being a str of current stream readings.
It will be executed on every line that comes from streams.
Example:
```python
from command_runner import command_runner
def callback_function(string):
# ADD YOUR CODE HERE
print('CALLBACK GOT:', string)
# Launch command_runner
exit_code, output = command_runner('ping 127.0.0.1', stdout=callback_function, method='poller')
```
#### stop_on
In some situations, you want a command to be aborted on some external triggers.
That's where `stop_on` argument comes in handy.
Just pass a function to `stop_on`, as soon as function result becomes True, execution will halt with exit code -251.
Example:
```python
from command_runner import command_runner
def some_function():
return True if we_must_stop_execution
exit_code, output = command_runner('ping 127.0.0.1', stop_on=some_function)
```
#### Checking intervals
By default, command_runner checks timeouts and outputs every 0.05 seconds.
You can increase/decrease this setting via `check_interval` setting which accepts floats.
Example: `command_runner(cmd, check_interval=0.2)`
Note that lowering `check_interval` will increase CPU usage.
#### Getting current process information
`command_runner` can provide a subprocess.Popen instance of currently run process as external data.
In order to do so, just declare a function and give it as `process_callback` argument.
Example:
```python
from command_runner import command_runner
def show_process_info(process):
print('My process has pid: {}'.format(process.pid))
exit_code, output = command_runner('ping 127.0.0.1', process_callback=show_process_info)
```
#### Split stdout and stderr
By default, `command_runner` returns a tuple like `(exit_code, output)` in which output contains both stdout and stderr stream outputs.
You can alter that behavior by using argument `split_stream=True`.
In that case, `command_runner` will return a tuple like `(exit_code, stdout, stderr)`.
Example:
```python
from command_runner import command_runner
exit_code, stdout, stderr = command_runner('ping 127.0.0.1', split_streams=True)
print('exit code:', exit_code)
print('stdout', stdout)
print('stderr', stderr)
```
#### On-exit Callback
`command_runner` allows to execute a callback function once it has finished it's execution.
This might help building threaded programs where a callback is needed to disable GUI elements for example.
Example:
```python
from command_runner import command_runner
def do_something():
print("We're done running")
exit_code, output = command_runner('ping 127.0.0.1', on_exit=do_something)
```
### Process and IO priority
`command_runner` can set it's subprocess priority to 'low', 'normal' or 'high', which translate to 15, 0, -15 niceness on Linux and BELOW_NORMAL_PRIORITY_CLASS and HIGH_PRIORITY_CLASS in Windows.
On Linux, you may also directly use priority with niceness int values.
You may also set subprocess io priority to 'low', 'normal' or 'high'.
Example:
```python
from command_runner import command_runner
exit_code, output = command_runner('some_intensive_process', priority='low', io_priority='high')
```
#### Other arguments
`command_runner` takes **any** argument that `subprocess.Popen()` would take.
It also uses the following standard arguments:
- command (str/list): The command, doesn't need to be a list, a simple string works
- valid_exit_codes (list): List of exit codes which won't trigger error logs
- timeout (int): seconds before a process tree is killed forcefully, defaults to 3600
- shell (bool): Shall we use the cmd.exe or /usr/bin/env shell for command execution, defaults to False
- encoding (str/bool): Which text encoding the command produces, defaults to cp437 under Windows and utf-8 under Linux
- stdin (sys.stdin/int): Optional stdin file descriptor, sent to the process command_runner spawns
- stdout (str/queue.Queue/function/False/None): Optional path to filename where to dump stdout, or queue where to write stdout, or callback function which is called when stdout has output
- stderr (str/queue.Queue/function/False/None): Optional path to filename where to dump stderr, or queue where to write stderr, or callback function which is called when stderr has output
- no_close_queues (bool): Normally, command_runner sends None to stdout / stderr queues when process is finished. This behavior can be disabled allowing to reuse those queues for other functions wrapping command_runner
- windows_no_window (bool): Shall a command create a console window (MS Windows only), defaults to False
- live_output (bool): Print output to stdout while executing command, defaults to False
- method (str): Accepts 'poller' or 'monitor' stdout capture and timeout monitoring methods
- check interval (float): Defaults to 0.05 seconds, which is the time between stream readings and timeout checks
- stop_on (function): Optional function that when returns True stops command_runner execution
- on_exit (function): Optional function that gets executed when command_runner has finished (callback function)
- process_callback (function): Optional function that will take command_runner spawned process as argument, in order to deal with process info outside of command_runner
- split_streams (bool): Split stdout and stderr into two separate results
- silent (bool): Allows to disable command_runner's internal logs, except for logging.DEBUG levels which for obvious reasons should never be silenced
- priority (str): Allows to set CPU bound process priority (takes 'low', 'normal' or 'high' parameter)
- io_priority (str): Allows to set IO priority for process (takes 'low', 'normal' or 'high' parameter)
- close_fds (bool): Like Popen, defaults to True on Linux and False on Windows
- universal_newlines (bool): Like Popen, defaults to False
- creation_flags (int): Like Popen, defaults to 0
- bufsize (int): Like Popen, defaults to 16384. Line buffering (bufsize=1) is deprecated since Python 3.7
**Note that ALL other subprocess.Popen arguments are supported, since they are directly passed to subprocess.**
## UAC Elevation / sudo elevation
command_runner package allowing privilege elevation.
Becoming an admin is fairly easy with command_runner.elevate
You only have to import the elevate module, and then launch your main function with the elevate function.
### elevation In a nutshell
```python
from command_runner.elevate import elevate
def main():
"""My main function that should be elevated"""
print("Who's the administrator, now ?")
if __name__ == '__main__':
elevate(main)
```
elevate function handles arguments (positional and keyword arguments).
`elevate(main, arg, arg2, kw=somearg)` will call `main(arg, arg2, kw=somearg)`
### Advanced elevate usage
#### is_admin() function
The elevate module has a nifty is_admin() function that returns a boolean according to your current root/administrator privileges.
Usage:
```python
from command_runner.elevate import is_admin
print('Am I an admin ? %s' % is_admin())
```
#### sudo elevation
Initially designed for Windows UAC, command_runner.elevate can also elevate privileges on Linux, using the sudo command.
This is mainly designed for PyInstaller / Nuitka executables, as it's really not safe to allow automatic privilege elevation of a Python interpreter.
Example for a binary in `/usr/local/bin/my_compiled_python_binary`
You'll have to allow this file to be run with sudo without a password prompt.
This can be achieved in `/etc/sudoers` file.
Example for Redhat / Rocky Linux, where adding the following line will allow the elevation process to succeed without password:
```
someuser ALL= NOPASSWD:/usr/local/bin/my_compiled_python_binary
```
## Footnotes
#### command_runner Python 2.7 compatible queue reader
The following example is a Python 2.7 compatible threaded implementation that reads stdout / stderr queue in a thread.
This only exists for compatibility reasons.
```python
import queue
import threading
from command_runner import command_runner
def read_queue(output_queue):
"""
Read the queue as thread
Our problem here is that the thread can live forever if we don't check a global value, which is...well ugly
"""
stream_output = ""
read_queue = True
while read_queue:
try:
line = output_queue.get(timeout=1)
except queue.Empty:
pass
else:
# The queue reading can be stopped once 'None' is received.
if line is None:
read_queue = False
else:
stream_output += line
# ADD YOUR LIVE CODE HERE
# Create a new queue that command_runner will fill up
output_queue = queue.Queue()
# Create a thread of read_queue() in order to read the queue while command_runner executes the command
read_thread = threading.Thread(
target=read_queue, args=(output_queue)
)
read_thread.daemon = True # thread dies with the program
read_thread.start()
# Launch command_runner, which will be blocking. Your live code goes directly into the threaded function
exit_code, output = command_runner('ping 127.0.0.1', stdout=output_queue, method='poller')
```
Raw data
{
"_id": null,
"home_page": "https://github.com/netinvent/command_runner",
"name": "command-runner",
"maintainer": "",
"docs_url": null,
"requires_python": ">=2.7",
"maintainer_email": "",
"keywords": "shell,execution,subprocess,check_output,wrapper,uac,sudo,elevate,privilege",
"author": "NetInvent - Orsiris de Jong",
"author_email": "contact@netinvent.fr",
"download_url": "https://files.pythonhosted.org/packages/fb/38/f2a02537b014876d9d674a2839fdacfbacfd433ba154fe7c4e93562c667b/command_runner-1.6.0.tar.gz",
"platform": null,
"description": "# command_runner\r\n# Platform agnostic command execution, timed background jobs with live stdout/stderr output capture, and UAC/sudo elevation\r\n\r\n[](https://opensource.org/licenses/BSD-3-Clause)\r\n[](http://isitmaintained.com/project/netinvent/command_runner \"Percentage of issues still open\")\r\n[](https://codeclimate.com/github/netinvent/command_runner/maintainability)\r\n[](https://codecov.io/gh/netinvent/command_runner)\r\n[](https://github.com/netinvent/command_runner/actions/workflows/linux.yaml)\r\n[](https://github.com/netinvent/command_runner/actions/workflows/windows.yaml)\r\n[](https://github.com/netinvent/command_runner/releases/latest)\r\n\r\n\r\ncommand_runner's purpose is to run external commands from python, just like subprocess on which it relies, \r\nwhile solving various problems a developer may face among:\r\n - Handling of all possible subprocess.popen / subprocess.check_output scenarios / python versions in one handy function without encoding / timeout hassle\r\n - Allow stdout/stderr stream output to be redirected to callback functions / output queues / files so you get to handle output in your application while commands are running\r\n - Callback to optional stop check so we can stop execution from outside command_runner\r\n - Callback with optional process information so we get to control the process from outside command_runner\r\n - Callback once we're finished to easen thread usage\r\n - Optional process priority and io_priority settings\r\n - System agnostic functionality, the developer shouldn't carry the burden of Windows & Linux differences\r\n - Optional Windows UAC elevation module compatible with CPython, PyInstaller & Nuitka\r\n - Optional Linux sudo elevation compatible with CPython, PyInstaller & Nuitka\r\n\r\nIt is compatible with Python 2.7+, tested up to Python 3.11 (backports some newer Python 3.5 functionality) and is tested on both Linux and Windows.\r\nIt is also compatible with PyPy Python implementation.\r\n...and yes, keeping Python 2.7 compatibility has proven to be quite challenging.\r\n\r\n## command_runner\r\n\r\ncommand_runner is a replacement package for subprocess.popen and subprocess.check_output\r\nThe main promise command_runner can do is to make sure to never have a blocking command, and always get results.\r\n\r\nIt works as wrapper for subprocess.popen and subprocess.communicate that solves:\r\n - Platform differences\r\n - Handle timeouts even for windows GUI applications that don't return anything to stdout\r\n - Python language version differences\r\n - Handle timeouts even on earlier Python implementations\r\n - Handle encoding even on earlier Python implementations\r\n - Keep the promise to always return an exit code (so we don't have to deal with exit codes and exception logic at the same time)\r\n - Keep the promise to always return the command output regardless of the execution state (even with timeouts, callback interrupts and keyboard interrupts)\r\n - Can show command output on the fly without waiting the end of execution (with `live_output=True` argument)\r\n - Can give command output on the fly to application by using queues or callback functions\r\n - Catch all possible exceptions and log them properly with encoding fixes\r\n - Be compatible, and always return the same result regarless of platform\r\n\r\ncommand_runner also promises to properly kill commands when timeouts are reached, including spawned subprocesses of such commands.\r\nThis specific behavior is achieved via psutil module, which is an optional dependency.\r\n\r\n \r\n### command_runner in a nutshell\r\n\r\nInstall with `pip install command_runner`\r\n\r\nThe following example will work regardless of the host OS and the Python version.\r\n\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', timeout=10)\r\n```\r\n\r\n\r\n## Guide to command_runner\r\n\r\n### Setup\r\n\r\n`pip install command_runner` or download the latest git release\r\n\r\n\r\n### Advanced command_runner usage\r\n\r\n\r\n#### Special exit codes\r\n\r\nIn order to keep the promise to always provide an exit_code, spcial exit codes have been added for the case where none is given.\r\nThose exit codes are:\r\n\r\n- -250 : command_runner called with incompatible arguments\r\n- -251 : stop_on function returned True\r\n- -252 : KeyboardInterrupt\r\n- -253 : FileNotFoundError, OSError, IOError\r\n- -254 : Timeout\r\n- -255 : Any other uncatched exceptions\r\n\r\nThis allows you to use the standard exit code logic, without having to deal with various exceptions.\r\n\r\n#### Default encoding\r\n\r\ncommand_runner has an `encoding` argument which defaults to `utf-8` for Unixes and `cp437` for Windows platforms.\r\nUsing `cp437` ensures that most `cmd.exe` output is encoded properly, including accents and special characters, on most locale systems.\r\nStill you can specify your own encoding for other usages, like Powershell where `unicode_escape` is preferred.\r\n\r\n```python\r\nfrom command_runner import command_runner\r\n\r\ncommand = r'C:\\Windows\\sysnative\\WindowsPowerShell\\v1.0\\powershell.exe --help'\r\nexit_code, output = command_runner(command, encoding='unicode_escape')\r\n```\r\n\r\nEarlier subprocess.popen implementations didn't have an encoding setting so command_runner will deal with encoding for those. \r\nYou can also disable command_runner's internal encoding in order to get raw process output (bytes) by passing False boolean.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, raw_output = command_runner('ping 127.0.0.1', encoding=False)\r\n```\r\n\r\n#### On the fly (interactive screen) output\r\n\r\n**Note: for live output capture and threading, see stream redirection. If you want to run your application while command_runner gives back command output, the best way to go is queues / callbacks.**\r\n\r\ncommand_runner can output a command output on the fly to stdout, eg show output on screen during execution.\r\nThis is helpful when the command is long, and we need to know the output while execution is ongoing.\r\nIt is also helpful in order to catch partial command output when timeout is reached or a CTRL+C signal is received.\r\nExample:\r\n\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', shell=True, live_output=True)\r\n```\r\n\r\nNote: using live output relies on stdout pipe polling, which has lightly higher cpu usage.\r\n\r\n#### Timeouts\r\n\r\n**command_runner has a `timeout` argument which defaults to 3600 seconds.**\r\nThis default setting ensures commands will not block the main script execution.\r\nFeel free to lower / higher that setting with `timeout` argument.\r\nNote that a command_runner kills the whole process tree that the command may have generated, even under Windows.\r\n\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', timeout=30)\r\n```\r\n\r\n#### Remarks on processes\r\n\r\nUsing `shell=True` will spawn a shell which will spawn the desired child process.\r\nBe aware that under MS Windows, no direct process tree is available.\r\nWe fixed this by walking processes during runtime. The drawback is that orphaned processes cannot be identified this way.\r\n\r\n\r\n#### Disabling logs / silencing\r\n\r\n`command_runner` has it's own logging system, which will log all sorts of error logs.\r\nIf you need to disable it's logging, just run with argument silent.\r\nBe aware that logging.DEBUG log levels won't be silenced, by design.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', silent=True)\r\n```\r\n\r\nIf you also need to disable logging.DEBUG level, you can run the following code which will required logging.CRITICAL only messages which `command_runner` never does:\r\n\r\n```python\r\nimport logging\r\nimport command_runner\r\n\r\nlogging.getLogger('command_runner').setLevel(logging.CRITICAL)\r\n```\r\n\r\n#### Capture method\r\n\r\n`command_runner` allows two different process output capture methods:\r\n\r\n`method='monitor'` which is default:\r\n - A thread is spawned in order to check stop conditions and kill process if needed\r\n - A main loop waits for the process to finish, then uses proc.communicate() to get it's output\r\n - Pros:\r\n - less CPU usage\r\n - less threads\r\n - Cons:\r\n - cannot read partial output on KeyboardInterrupt or stop_on (still works for partial timeout output)\r\n - cannot use queues or callback functions redirectors\r\n - is 0.1 seconds slower than poller method\r\n \r\n\r\n`method='poller'`:\r\n - A thread is spawned and reads stdout/stderr pipes into output queues\r\n - A poller loop reads from the output queues, checks stop conditions and kills process if needed\r\n - Pros: \r\n - Reads on the fly, allowing interactive commands (is also used with `live_output=True`)\r\n - Allows stdout/stderr output to be written live to callback functions, queues or files (useful when threaded)\r\n - is 0.1 seconds faster than monitor method, is preferred method for fast batch runnings\r\n - Cons:\r\n - lightly higher CPU usage\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', method='poller')\r\nexit_code, output = command_runner('ping 127.0.0.1', method='monitor')\r\n```\r\n\r\n#### stdin stream redirection\r\n\r\n`command_runner` allows to redirect some stream directly into the subprocess it spawns.\r\n\r\nExample code\r\n```python\r\nimport sys\r\nfrom command_runner import command_runner\r\n\r\n\r\nexit_code, output = command_runner(\"gzip -d\", stdin=sys.stdin.buffer)\r\nprint(\"Uncompressed data\", output)\r\n```\r\nThe above program, when run with `echo \"Hello, World!\" | gzip | python myscript.py` will show the uncompressed string `Hello, World!`\r\n\r\nYou can use whatever file descriptor you want, basic ones being sys.stdin for text input and sys.stdin.buffer for binary input.\r\n\r\n#### stdout / stderr stream redirection\r\n\r\ncommand_runner can redirect stdout and/or stderr streams to different outputs:\r\n - subprocess pipes\r\n - /dev/null or NUL\r\n - files\r\n - queues\r\n - callback functions\r\n\r\nUnless an output redirector is given for `stderr` argument, stderr will be redirected to `stdout` stream.\r\nNote that both queues and callback function redirectors require `poller` method and will fail if method is not set.\r\n\r\nPossible output redirection options are:\r\n\r\n- subprocess pipes\r\n\r\nBy default, stdout writes into a subprocess.PIPE which is read by command_runner and returned as `output` variable.\r\nYou may also pass any other subprocess.PIPE int values to `stdout` or `stderr` arguments.\r\n\r\n- /dev/null or NUL\r\n\r\nIf `stdout=False` and/or `stderr=False` argument(s) are given, command output will not be saved.\r\nstdout/stderr streams will be redirected to `/dev/null` or `NUL` depending on platform.\r\n\r\nOutput will always be `None`. See `split_streams` for more details using multiple outputs.\r\n\r\n- files\r\n\r\nGiving `stdout` and/or `stderr` arguments a string, `command_runner` will consider the string to be a file path where stream output will be written live.\r\n\r\nExamples:\r\n```python\r\nfrom command_runner import command_runner\r\nexit_code, output = command_runner('dir', stdout=r\"C:/tmp/command_result\", stderr=r\"C:/tmp/command_error\", shell=True)\r\n```\r\n```python\r\nfrom command_runner import command_runner\r\nexit_code, output = command_runner('dir', stdout='/tmp/stdout.log', stderr='/tmp/stderr.log', shell=True)\r\n```\r\n\r\nOpening a file with the wrong encoding (especially opening a CP437 encoded file on Windows with UTF-8 coded might endup with UnicodedecodeError.)\r\n\r\n- queues\r\n\r\nQueue(s) will be filled up by command_runner.\r\n\r\nIn order to keep your program \"live\", we'll use the threaded version of command_runner which is basically the same except it returns a future result instead of a tuple.\r\n\r\nNote: With all the best will, there's no good way to achieve this under Python 2.7 without using more queues, so the threaded version is only compatible with Python 3.3+.\r\n\r\nFor Python 2.7, you must create your thread and queue reader yourself (see footnote for a Python 2.7 comaptible example).\r\n\r\nThreaded command_runner plus queue example:\r\n\r\n```python\r\nimport queue\r\nfrom command_runner import command_runner_threaded\r\n\r\noutput_queue = queue.Queue()\r\nstream_output = \"\"\r\nthread_result = command_runner_threaded('ping 127.0.0.1', shell=True, method='poller', stdout=output_queue)\r\n\r\nread_queue = True\r\nwhile read_queue:\r\n try:\r\n line = output_queue.get(timeout=0.1)\r\n except queue.Empty:\r\n pass\r\n else:\r\n if line is None:\r\n read_queue = False\r\n else:\r\n stream_output += line\r\n # ADD YOUR LIVE CODE HERE\r\n\r\n# Now we may get exit_code and output since result has become available at this point\r\nexit_code, output = thread_result.result()\r\n```\r\nYou might also want to read both stdout and stderr queues. In that case, you can create a read loop just like in the following example.\r\nHere we're reading both queues in one loop, so we need to observe a couple of conditions before stopping the loop, in order to catch all queue output:\r\n```python\r\nimport queue\r\nfrom time import sleep\r\nfrom command_runner import command_runner_threaded\r\n\r\nstdout_queue = queue.Queue()\r\nstderr_queue = queue.Queue()\r\nthread_result = command_runner_threaded('ping 127.0.0.1', method='poller', shell=True, stdout=stdout_queue, stderr=stderr_queue)\r\n\r\nread_stdout = read_stderr = True\r\nwhile read_stdout or read_stderr:\r\n\r\n try:\r\n stdout_line = stdout_queue.get(timeout=0.1)\r\n except queue.Empty:\r\n pass\r\n else:\r\n if stdout_line is None:\r\n read_stdout = False\r\n else:\r\n print('STDOUT:', stdout_line)\r\n\r\n try:\r\n stderr_line = stderr_queue.get(timeout=0.1)\r\n except queue.Empty:\r\n pass\r\n else:\r\n if stderr_line is None:\r\n read_stderr = False\r\n else:\r\n print('STDERR:', stderr_line)\r\n \r\n # ADD YOUR LIVE CODE HERE\r\n\r\nexit_code, output = thread_result.result()\r\nassert exit_code == 0, 'We did not succeed in running the thread'\r\n\r\n```\r\n\r\n- callback functions\r\n\r\nThe callback function will get one argument, being a str of current stream readings.\r\nIt will be executed on every line that comes from streams.\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\ndef callback_function(string):\r\n # ADD YOUR CODE HERE\r\n print('CALLBACK GOT:', string)\r\n \r\n# Launch command_runner\r\nexit_code, output = command_runner('ping 127.0.0.1', stdout=callback_function, method='poller')\r\n```\r\n\r\n#### stop_on\r\n\r\nIn some situations, you want a command to be aborted on some external triggers.\r\nThat's where `stop_on` argument comes in handy.\r\nJust pass a function to `stop_on`, as soon as function result becomes True, execution will halt with exit code -251.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\ndef some_function():\r\n return True if we_must_stop_execution\r\nexit_code, output = command_runner('ping 127.0.0.1', stop_on=some_function)\r\n```\r\n\r\n#### Checking intervals\r\n\r\nBy default, command_runner checks timeouts and outputs every 0.05 seconds.\r\nYou can increase/decrease this setting via `check_interval` setting which accepts floats.\r\nExample: `command_runner(cmd, check_interval=0.2)`\r\nNote that lowering `check_interval` will increase CPU usage.\r\n\r\n#### Getting current process information\r\n\r\n`command_runner` can provide a subprocess.Popen instance of currently run process as external data.\r\nIn order to do so, just declare a function and give it as `process_callback` argument.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\ndef show_process_info(process):\r\n print('My process has pid: {}'.format(process.pid))\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', process_callback=show_process_info)\r\n```\r\n\r\n#### Split stdout and stderr\r\n\r\nBy default, `command_runner` returns a tuple like `(exit_code, output)` in which output contains both stdout and stderr stream outputs.\r\nYou can alter that behavior by using argument `split_stream=True`.\r\nIn that case, `command_runner` will return a tuple like `(exit_code, stdout, stderr)`.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, stdout, stderr = command_runner('ping 127.0.0.1', split_streams=True)\r\nprint('exit code:', exit_code)\r\nprint('stdout', stdout)\r\nprint('stderr', stderr)\r\n```\r\n\r\n#### On-exit Callback\r\n\r\n`command_runner` allows to execute a callback function once it has finished it's execution.\r\nThis might help building threaded programs where a callback is needed to disable GUI elements for example.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\ndef do_something():\r\n print(\"We're done running\")\r\n\r\nexit_code, output = command_runner('ping 127.0.0.1', on_exit=do_something)\r\n```\r\n\r\n### Process and IO priority\r\n`command_runner` can set it's subprocess priority to 'low', 'normal' or 'high', which translate to 15, 0, -15 niceness on Linux and BELOW_NORMAL_PRIORITY_CLASS and HIGH_PRIORITY_CLASS in Windows.\r\nOn Linux, you may also directly use priority with niceness int values.\r\n\r\nYou may also set subprocess io priority to 'low', 'normal' or 'high'.\r\n\r\nExample:\r\n```python\r\nfrom command_runner import command_runner\r\n\r\nexit_code, output = command_runner('some_intensive_process', priority='low', io_priority='high')\r\n```\r\n\r\n#### Other arguments\r\n\r\n`command_runner` takes **any** argument that `subprocess.Popen()` would take.\r\n\r\nIt also uses the following standard arguments:\r\n - command (str/list): The command, doesn't need to be a list, a simple string works\r\n - valid_exit_codes (list): List of exit codes which won't trigger error logs\r\n - timeout (int): seconds before a process tree is killed forcefully, defaults to 3600\r\n - shell (bool): Shall we use the cmd.exe or /usr/bin/env shell for command execution, defaults to False\r\n - encoding (str/bool): Which text encoding the command produces, defaults to cp437 under Windows and utf-8 under Linux\r\n - stdin (sys.stdin/int): Optional stdin file descriptor, sent to the process command_runner spawns\r\n - stdout (str/queue.Queue/function/False/None): Optional path to filename where to dump stdout, or queue where to write stdout, or callback function which is called when stdout has output\r\n - stderr (str/queue.Queue/function/False/None): Optional path to filename where to dump stderr, or queue where to write stderr, or callback function which is called when stderr has output\r\n - no_close_queues (bool): Normally, command_runner sends None to stdout / stderr queues when process is finished. This behavior can be disabled allowing to reuse those queues for other functions wrapping command_runner\r\n - windows_no_window (bool): Shall a command create a console window (MS Windows only), defaults to False\r\n - live_output (bool): Print output to stdout while executing command, defaults to False\r\n - method (str): Accepts 'poller' or 'monitor' stdout capture and timeout monitoring methods\r\n - check interval (float): Defaults to 0.05 seconds, which is the time between stream readings and timeout checks\r\n - stop_on (function): Optional function that when returns True stops command_runner execution\r\n - on_exit (function): Optional function that gets executed when command_runner has finished (callback function)\r\n - process_callback (function): Optional function that will take command_runner spawned process as argument, in order to deal with process info outside of command_runner\r\n - split_streams (bool): Split stdout and stderr into two separate results\r\n - silent (bool): Allows to disable command_runner's internal logs, except for logging.DEBUG levels which for obvious reasons should never be silenced\r\n - priority (str): Allows to set CPU bound process priority (takes 'low', 'normal' or 'high' parameter)\r\n - io_priority (str): Allows to set IO priority for process (takes 'low', 'normal' or 'high' parameter)\r\n - close_fds (bool): Like Popen, defaults to True on Linux and False on Windows\r\n - universal_newlines (bool): Like Popen, defaults to False\r\n - creation_flags (int): Like Popen, defaults to 0\r\n - bufsize (int): Like Popen, defaults to 16384. Line buffering (bufsize=1) is deprecated since Python 3.7\r\n\r\n**Note that ALL other subprocess.Popen arguments are supported, since they are directly passed to subprocess.**\r\n\r\n\r\n## UAC Elevation / sudo elevation\r\n\r\ncommand_runner package allowing privilege elevation.\r\nBecoming an admin is fairly easy with command_runner.elevate\r\nYou only have to import the elevate module, and then launch your main function with the elevate function.\r\n\r\n### elevation In a nutshell\r\n\r\n```python\r\nfrom command_runner.elevate import elevate\r\n\r\ndef main():\r\n \"\"\"My main function that should be elevated\"\"\"\r\n print(\"Who's the administrator, now ?\")\r\n\r\nif __name__ == '__main__':\r\n elevate(main)\r\n```\r\n\r\nelevate function handles arguments (positional and keyword arguments).\r\n`elevate(main, arg, arg2, kw=somearg)` will call `main(arg, arg2, kw=somearg)`\r\n\r\n### Advanced elevate usage\r\n\r\n#### is_admin() function\r\n\r\nThe elevate module has a nifty is_admin() function that returns a boolean according to your current root/administrator privileges.\r\nUsage:\r\n\r\n```python\r\nfrom command_runner.elevate import is_admin\r\n\r\nprint('Am I an admin ? %s' % is_admin())\r\n```\r\n\r\n#### sudo elevation\r\n\r\nInitially designed for Windows UAC, command_runner.elevate can also elevate privileges on Linux, using the sudo command.\r\nThis is mainly designed for PyInstaller / Nuitka executables, as it's really not safe to allow automatic privilege elevation of a Python interpreter.\r\n\r\nExample for a binary in `/usr/local/bin/my_compiled_python_binary`\r\n\r\nYou'll have to allow this file to be run with sudo without a password prompt.\r\nThis can be achieved in `/etc/sudoers` file.\r\n\r\nExample for Redhat / Rocky Linux, where adding the following line will allow the elevation process to succeed without password:\r\n```\r\nsomeuser ALL= NOPASSWD:/usr/local/bin/my_compiled_python_binary\r\n```\r\n\r\n## Footnotes\r\n\r\n#### command_runner Python 2.7 compatible queue reader\r\n\r\nThe following example is a Python 2.7 compatible threaded implementation that reads stdout / stderr queue in a thread.\r\nThis only exists for compatibility reasons.\r\n\r\n```python\r\nimport queue\r\nimport threading\r\nfrom command_runner import command_runner\r\n\r\ndef read_queue(output_queue):\r\n \"\"\"\r\n Read the queue as thread\r\n Our problem here is that the thread can live forever if we don't check a global value, which is...well ugly\r\n \"\"\"\r\n stream_output = \"\"\r\n read_queue = True\r\n while read_queue:\r\n try:\r\n line = output_queue.get(timeout=1)\r\n except queue.Empty:\r\n pass\r\n else:\r\n # The queue reading can be stopped once 'None' is received.\r\n if line is None:\r\n read_queue = False\r\n else:\r\n stream_output += line\r\n # ADD YOUR LIVE CODE HERE\r\n\r\n\r\n# Create a new queue that command_runner will fill up\r\noutput_queue = queue.Queue()\r\n\r\n# Create a thread of read_queue() in order to read the queue while command_runner executes the command\r\nread_thread = threading.Thread(\r\n target=read_queue, args=(output_queue)\r\n)\r\nread_thread.daemon = True # thread dies with the program\r\nread_thread.start()\r\n\r\n# Launch command_runner, which will be blocking. Your live code goes directly into the threaded function\r\nexit_code, output = command_runner('ping 127.0.0.1', stdout=output_queue, method='poller')\r\n```\r\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "Platform agnostic command and shell execution tool, also allows UAC/sudo privilege elevation",
"version": "1.6.0",
"project_urls": {
"Homepage": "https://github.com/netinvent/command_runner"
},
"split_keywords": [
"shell",
"execution",
"subprocess",
"check_output",
"wrapper",
"uac",
"sudo",
"elevate",
"privilege"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "edf08440eeddd9f9bc59fa07f0ede2a33c46835d4c67fd64fe9e19d9671b12d9",
"md5": "d1abbc94a5aef7844411407b372d7c5a",
"sha256": "59f59293b0ba2f8e862a7558ea616b4b157ac7da294be70363ad9082d91f537e"
},
"downloads": -1,
"filename": "command_runner-1.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d1abbc94a5aef7844411407b372d7c5a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=2.7",
"size": 24974,
"upload_time": "2024-01-09T19:45:53",
"upload_time_iso_8601": "2024-01-09T19:45:53.962516Z",
"url": "https://files.pythonhosted.org/packages/ed/f0/8440eeddd9f9bc59fa07f0ede2a33c46835d4c67fd64fe9e19d9671b12d9/command_runner-1.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fb38f2a02537b014876d9d674a2839fdacfbacfd433ba154fe7c4e93562c667b",
"md5": "e724f6455b1ef58efe4b0689829ed30d",
"sha256": "973b7552186b3ea42b04ab1198f86a86d3887c5942b5e42aa3ab19eab3aeb740"
},
"downloads": -1,
"filename": "command_runner-1.6.0.tar.gz",
"has_sig": false,
"md5_digest": "e724f6455b1ef58efe4b0689829ed30d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.7",
"size": 38917,
"upload_time": "2024-01-09T19:45:58",
"upload_time_iso_8601": "2024-01-09T19:45:58.433948Z",
"url": "https://files.pythonhosted.org/packages/fb/38/f2a02537b014876d9d674a2839fdacfbacfd433ba154fe7c4e93562c667b/command_runner-1.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-09 19:45:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "netinvent",
"github_project": "command_runner",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "command-runner"
}
JSON
