Spur+
=====
.. image:: https://api.travis-ci.com/Parquery/spurplus.svg?branch=master
:target: https://api.travis-ci.com/Parquery/spurplus.svg?branch=master
:alt: Build Status
.. image:: https://coveralls.io/repos/github/Parquery/spurplus/badge.svg?branch=master
:target: https://coveralls.io/github/Parquery/spurplus?branch=master
:alt: Coverage
.. image:: https://readthedocs.org/projects/spurplus/badge/?version=latest
:target: https://spurplus.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://badge.fury.io/py/spurplus.svg
:target: https://pypi.org/project/spurplus/
:alt: PyPi
.. image:: https://img.shields.io/pypi/pyversions/spurplus.svg
:alt: PyPI - Python Version
Spur+ is a library to manage remote machines and perform file operations over SSH.
It builds on top of Spur_ and Paramiko_ libraries. While we already find that Spur_ and Paramiko_ provide most of the
functionality out-of-the-box, we missed certain features:
- typing. Since spur supports both Python 2 and 3, it does not provide any type annotations which makes it harder to use
with type checkers such as mypy.
- pathlib.Path support. We find it easier to manipulate paths using pathlib.Path instead of plain strings. spur+
provides support for both.
- a function for creating directories. spur relies on sftp client. While it is fairly straightforward to get an sftp
client from ``spur.SshShell`` and create a directory, we think that it merits a wrapper function akin to
``pathlib.Path.mkdir()`` provided how often this functionality is needed.
- reading/writing text and binary data in one go. Similarly to creating directories, ``spur.SshShell.open()`` already
provides all the functionality you need to read/write files. However, we found the usage code to be more readable when
written in one line and no extra variables for file descriptors are introduced.
- a function for putting and getting files to/from the remote host, respectively.
- a function to sync a local directory to a remote directory (similar to ``rsync``).
- a function for computing MD5 checksums.
- a function to check if a file exists.
- a more elaborate context manager for a temporary directory which allows for specifying prefix, suffix and
base directory and gives you a pathlib.Path. In contrast, ``spur.temporary_directory()`` gives you only a string with
no knobs.
- an initializer function to repeatedly re-connect on connection failure. We found this function particularly important
when you spin a virtual instance in the cloud and need to wait for it to initialize.
- a wrapper around paramiko's SFTP client (``spurplus.sftp.ReconnectingSFTP``) to automatically reconnect if the SFTP
client experienced a connection failure. While original ``spur.SshShell.open()`` creates a new SFTP client on every
call in order to prevent issues with time-outs, `spurplus.SshShell` is able to re-use the SFTP client over multiple
calls via ``spurplus.sftp.ReconnectingSFTP``.
This can lead up to 10x speed-up (see the benchmark in ``tests/live_test.py``).
.. _Spur: https://github.com/mwilliamson/spur.py
.. _Paramiko: https://github.com/paramiko/paramiko
Usage
=====
.. code-block:: python
import pathlib
import spurplus
# Re-try on connection failure; sftp client and the underlying spur SshShell
# are automatically closed when the shell is closed.
with spurplus.connect_with_retries(
hostname='some-machine.example.com', username='devop') as shell:
p = pathlib.Path('/some/directory')
# Create a directory
shell.mkdir(remote_path=p, parents=True, exist_ok=True)
# Write a file
shell.write_text(remote_path=p/'some-file', text='hello world!')
# Read from a file
text = shell.read_text(remote_path=p/'some-file')
# Change the permissions
shell.chmod(remote_path=p/'some-file', mode=0o444)
# Sync a local directory to a remote.
# Only differing files are uploaded,
# files missing locally are deleted before the transfer and
# the permissions are mirrored from the local.
sync_to_remote(
local_path="/some/local/directory",
remote_path="/some/remote/directory",
delete=spurplus.Delete.BEFORE,
preserve_permissions = True)
# Stat the file
print("The stat of {}: {}".format(p/'some-file', shell.stat(p/'some-file')))
# Use a wrapped SFTP client
sftp = shell.as_sftp()
# Do something with the SFTP
for attr in sftp.listdir_attr(path=p.as_posix()):
do_something(attr.filename, attr.st_size)
Documentation
=============
The documentation is available on `readthedocs <https://spurplus.readthedocs.io/en/latest/>`_.
Installation
============
* Create a virtual environment:
.. code-block:: bash
python3 -m venv venv3
* Activate it:
.. code-block:: bash
source venv3/bin/activate
* Install spur+ with pip:
.. code-block:: bash
pip3 install spurplus
Development
===========
* Check out the repository.
* In the repository root, create the virtual environment:
.. code-block:: bash
python3 -m venv venv3
* Activate the virtual environment:
.. code-block:: bash
source venv3/bin/activate
* Install the development dependencies:
.. code-block:: bash
pip3 install -e .[dev]
* There are live tests for which you need to have a running SSH server. The parameters of the tests
are passed via environment variables:
- ``TEST_SSH_HOSTNAME`` (host name of the SSH server, defaults to "127.0.0.1"),
- ``TEST_SSH_PORT`` (optional, defaults to 22),
- ``TEST_SSH_USERNAME`` (optional, uses paramiko's default),
- ``TEST_SSH_PASSWORD`` (optional, uses private key file if not specified) and
- ``TEST_SSH_PRIVATE_KEY_FILE`` (optional, looks for private key in expected places if not specified).
We use tox for testing and packaging the distribution. Assuming that the above-mentioned environment variables has
been set, the virutal environment has been activated and the development dependencies have been installed, run:
.. code-block:: bash
tox
Pre-commit Checks
-----------------
We provide a set of pre-commit checks that lint and check code for formatting.
Namely, we use:
* `yapf <https://github.com/google/yapf>`_ to check the formatting.
* The style of the docstrings is checked with `pydocstyle <https://github.com/PyCQA/pydocstyle>`_.
* Static type analysis is performed with `mypy <http://mypy-lang.org/>`_.
* Various linter checks are done with `pylint <https://www.pylint.org/>`_.
* Doctests are executed using the Python `doctest module <https://docs.python.org/3.5/library/doctest.html>`_.
Run the pre-commit checks locally from an activated virtual environment with development dependencies:
.. code-block:: bash
./precommit.py
* The pre-commit script can also automatically format the code:
.. code-block:: bash
./precommit.py --overwrite
Versioning
==========
We follow `Semantic Versioning <http://semver.org/spec/v1.0.0.html>`_. The version X.Y.Z indicates:
* X is the major version (backward-incompatible),
* Y is the minor version (backward-compatible), and
* Z is the patch version (backward-compatible bug fix).
Raw data
{
"_id": null,
"home_page": "http://github.com/Parquery/spurplus",
"name": "spurplus",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "ssh sftp spur paramiko execute remote commands modify files",
"author": "Marko Ristin",
"author_email": "marko.ristin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c2/5f/a11bbda84921ffbbc12196cdc21bee2f6754372b8a8ec9b11f9b4f3b4087/spurplus-2.3.5.tar.gz",
"platform": null,
"description": "Spur+\n=====\n\n.. image:: https://api.travis-ci.com/Parquery/spurplus.svg?branch=master\n :target: https://api.travis-ci.com/Parquery/spurplus.svg?branch=master\n :alt: Build Status\n\n.. image:: https://coveralls.io/repos/github/Parquery/spurplus/badge.svg?branch=master\n :target: https://coveralls.io/github/Parquery/spurplus?branch=master\n :alt: Coverage\n\n.. image:: https://readthedocs.org/projects/spurplus/badge/?version=latest\n :target: https://spurplus.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n \n.. image:: https://badge.fury.io/py/spurplus.svg\n :target: https://pypi.org/project/spurplus/\n :alt: PyPi\n\n.. image:: https://img.shields.io/pypi/pyversions/spurplus.svg\n :alt: PyPI - Python Version\n\nSpur+ is a library to manage remote machines and perform file operations over SSH.\n\nIt builds on top of Spur_ and Paramiko_ libraries. While we already find that Spur_ and Paramiko_ provide most of the\nfunctionality out-of-the-box, we missed certain features:\n\n- typing. Since spur supports both Python 2 and 3, it does not provide any type annotations which makes it harder to use\n with type checkers such as mypy.\n\n- pathlib.Path support. We find it easier to manipulate paths using pathlib.Path instead of plain strings. spur+\n provides support for both.\n\n- a function for creating directories. spur relies on sftp client. While it is fairly straightforward to get an sftp\n client from ``spur.SshShell`` and create a directory, we think that it merits a wrapper function akin to\n ``pathlib.Path.mkdir()`` provided how often this functionality is needed.\n\n- reading/writing text and binary data in one go. Similarly to creating directories, ``spur.SshShell.open()`` already\n provides all the functionality you need to read/write files. However, we found the usage code to be more readable when\n written in one line and no extra variables for file descriptors are introduced.\n\n- a function for putting and getting files to/from the remote host, respectively.\n\n- a function to sync a local directory to a remote directory (similar to ``rsync``).\n\n- a function for computing MD5 checksums.\n\n- a function to check if a file exists.\n\n- a more elaborate context manager for a temporary directory which allows for specifying prefix, suffix and\n base directory and gives you a pathlib.Path. In contrast, ``spur.temporary_directory()`` gives you only a string with\n no knobs.\n\n- an initializer function to repeatedly re-connect on connection failure. We found this function particularly important\n when you spin a virtual instance in the cloud and need to wait for it to initialize.\n\n- a wrapper around paramiko's SFTP client (``spurplus.sftp.ReconnectingSFTP``) to automatically reconnect if the SFTP\n client experienced a connection failure. While original ``spur.SshShell.open()`` creates a new SFTP client on every\n call in order to prevent issues with time-outs, `spurplus.SshShell` is able to re-use the SFTP client over multiple\n calls via ``spurplus.sftp.ReconnectingSFTP``.\n\n This can lead up to 10x speed-up (see the benchmark in ``tests/live_test.py``).\n\n.. _Spur: https://github.com/mwilliamson/spur.py\n.. _Paramiko: https://github.com/paramiko/paramiko\n\nUsage\n=====\n.. code-block:: python\n\n import pathlib\n\n import spurplus\n\n # Re-try on connection failure; sftp client and the underlying spur SshShell\n # are automatically closed when the shell is closed.\n with spurplus.connect_with_retries(\n hostname='some-machine.example.com', username='devop') as shell:\n p = pathlib.Path('/some/directory')\n\n # Create a directory\n shell.mkdir(remote_path=p, parents=True, exist_ok=True)\n\n # Write a file\n shell.write_text(remote_path=p/'some-file', text='hello world!')\n\n # Read from a file\n text = shell.read_text(remote_path=p/'some-file')\n\n # Change the permissions\n shell.chmod(remote_path=p/'some-file', mode=0o444)\n\n # Sync a local directory to a remote.\n # Only differing files are uploaded,\n # files missing locally are deleted before the transfer and\n # the permissions are mirrored from the local.\n sync_to_remote(\n local_path=\"/some/local/directory\",\n remote_path=\"/some/remote/directory\",\n delete=spurplus.Delete.BEFORE,\n preserve_permissions = True)\n\n # Stat the file\n print(\"The stat of {}: {}\".format(p/'some-file', shell.stat(p/'some-file')))\n\n # Use a wrapped SFTP client\n sftp = shell.as_sftp()\n # Do something with the SFTP\n for attr in sftp.listdir_attr(path=p.as_posix()):\n do_something(attr.filename, attr.st_size)\n\nDocumentation\n=============\nThe documentation is available on `readthedocs <https://spurplus.readthedocs.io/en/latest/>`_.\n\nInstallation\n============\n\n* Create a virtual environment:\n\n.. code-block:: bash\n\n python3 -m venv venv3\n\n* Activate it:\n\n.. code-block:: bash\n\n source venv3/bin/activate\n\n* Install spur+ with pip:\n\n.. code-block:: bash\n\n pip3 install spurplus\n\nDevelopment\n===========\n\n* Check out the repository.\n\n* In the repository root, create the virtual environment:\n\n.. code-block:: bash\n\n python3 -m venv venv3\n\n* Activate the virtual environment:\n\n.. code-block:: bash\n\n source venv3/bin/activate\n\n* Install the development dependencies:\n\n.. code-block:: bash\n\n pip3 install -e .[dev]\n\n* There are live tests for which you need to have a running SSH server. The parameters of the tests\n are passed via environment variables:\n\n - ``TEST_SSH_HOSTNAME`` (host name of the SSH server, defaults to \"127.0.0.1\"),\n - ``TEST_SSH_PORT`` (optional, defaults to 22),\n - ``TEST_SSH_USERNAME`` (optional, uses paramiko's default),\n - ``TEST_SSH_PASSWORD`` (optional, uses private key file if not specified) and\n - ``TEST_SSH_PRIVATE_KEY_FILE`` (optional, looks for private key in expected places if not specified).\n\nWe use tox for testing and packaging the distribution. Assuming that the above-mentioned environment variables has\nbeen set, the virutal environment has been activated and the development dependencies have been installed, run:\n\n.. code-block:: bash\n\n tox\n\nPre-commit Checks\n-----------------\nWe provide a set of pre-commit checks that lint and check code for formatting.\n\nNamely, we use:\n\n* `yapf <https://github.com/google/yapf>`_ to check the formatting.\n* The style of the docstrings is checked with `pydocstyle <https://github.com/PyCQA/pydocstyle>`_.\n* Static type analysis is performed with `mypy <http://mypy-lang.org/>`_.\n* Various linter checks are done with `pylint <https://www.pylint.org/>`_.\n* Doctests are executed using the Python `doctest module <https://docs.python.org/3.5/library/doctest.html>`_.\n\nRun the pre-commit checks locally from an activated virtual environment with development dependencies:\n\n.. code-block:: bash\n\n ./precommit.py\n\n* The pre-commit script can also automatically format the code:\n\n.. code-block:: bash\n\n ./precommit.py --overwrite\n\n\nVersioning\n==========\nWe follow `Semantic Versioning <http://semver.org/spec/v1.0.0.html>`_. The version X.Y.Z indicates:\n\n* X is the major version (backward-incompatible),\n* Y is the minor version (backward-compatible), and\n* Z is the patch version (backward-compatible bug fix).\n",
"bugtrack_url": null,
"license": "License :: OSI Approved :: MIT License",
"summary": "Manage remote machines and file operations over SSH.",
"version": "2.3.5",
"project_urls": {
"Homepage": "http://github.com/Parquery/spurplus"
},
"split_keywords": [
"ssh",
"sftp",
"spur",
"paramiko",
"execute",
"remote",
"commands",
"modify",
"files"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6cffce6fbdb6ef2d11faa37907babf01a4d1fce88259dac6b34b338ff7def7a3",
"md5": "d6fa7dbabdbc93dbe4ff5181ae0673a7",
"sha256": "bde03d81d439753d89ce84f6330c50c426b334b3bb1ab62e39c2e81d7555a0dd"
},
"downloads": -1,
"filename": "spurplus-2.3.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d6fa7dbabdbc93dbe4ff5181ae0673a7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 19937,
"upload_time": "2024-03-29T09:50:33",
"upload_time_iso_8601": "2024-03-29T09:50:33.583739Z",
"url": "https://files.pythonhosted.org/packages/6c/ff/ce6fbdb6ef2d11faa37907babf01a4d1fce88259dac6b34b338ff7def7a3/spurplus-2.3.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c25fa11bbda84921ffbbc12196cdc21bee2f6754372b8a8ec9b11f9b4f3b4087",
"md5": "b954816076e17063d44cddfa8e5178a2",
"sha256": "07752555e36c63655fdb2c8bd6e64dfb48c535045a8d4a9d0fbdbf1991ca99ca"
},
"downloads": -1,
"filename": "spurplus-2.3.5.tar.gz",
"has_sig": false,
"md5_digest": "b954816076e17063d44cddfa8e5178a2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 28918,
"upload_time": "2024-03-29T09:50:35",
"upload_time_iso_8601": "2024-03-29T09:50:35.031351Z",
"url": "https://files.pythonhosted.org/packages/c2/5f/a11bbda84921ffbbc12196cdc21bee2f6754372b8a8ec9b11f9b4f3b4087/spurplus-2.3.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-29 09:50:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Parquery",
"github_project": "spurplus",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "spurplus"
}
JSON
