mig-swarmspawner


Namemig-swarmspawner JSON
Version 0.0.8 PyPI version JSON
download
home_pagehttps://github.com/rasmunk/SwarmSpawner
Summary MiG-SwarmSpawner enables JupyterHub to spawn jupyter notebooks that can interact with the MiG infrastructure.
upload_time2018-04-03 15:06:15
maintainer
docs_urlNone
authorRasmus Munk
requires_python
licenseBSD
keywords interactive interpreter shell web
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI
coveralls test coverage No coveralls.
            ==============================
SwarmSpawner
==============================

.. image:: https://travis-ci.org/rasmunk/SwarmSpawner.svg?branch=master
    :target: https://travis-ci.org/rasmunk/SwarmSpawner


**MiG SwarmSpawner** enables `JupyterHub <https://github
.com/jupyterhub/jupyterhub>`_ to spawn jupyter notebooks that can interact
with the MiG infrastructure.

More info about Docker Services `here <https://docs.docker.com/engine/reference/commandline/service_create/>`_.


Prerequisites
================

Python version 3.3 and above is required.


Installation
================

.. code-block:: sh

   pip install mig-swarmspawner

Installation from GitHub
============================

.. code-block:: sh

   git clone https://github.com/rasmunk/SwarmSpawner
   cd SwarmSpawner
   python setup.py install

Configuration
================

You can find an example jupyter_config.py inside `examples <examples>`_.

The spawner
================
Docker Engine in Swarm mode and the related services work in a different way compared to Docker containers.

Tell JupyterHub to use SwarmSpawner by adding the following lines to your `jupyterhub_config.py`:

.. code-block:: python

        c.JupyterHub.spawner_class = 'mig.SwarmSpawner'
        c.JupyterHub.hub_ip = '0.0.0.0'
        # This should be the name of the jupyterhub service
        c.SwarmSpawner.jupyterhub_service_name = 'NameOfTheService'

What is ``jupyterhub_service_name``?

Inside a Docker engine in Swarm mode the services use a `name` instead of a `ip` to communicate with each other.
'jupyterhub_service_name' is the name of ther service for the JupyterHub.

Networks
============
It's important to put the JupyterHub service (also the proxy) and the services that are running jupyter notebook inside the same network, otherwise they couldn't reach each other.
SwarmSpawner use the service's name instead of the service's ip, as a consequence JupyterHub and servers should share the same overlay network (network across nodes).

.. code-block:: python

        #list of networks
        c.SwarmSpawner.networks = ["mynetwork"]


Define the services inside jupyterhub_config.py
===============================================
You can define *container_spec*, *resource_spec* and *networks* inside **jupyterhub_config.py**.

Container_spec__
-------------------
__ https://github.com/docker/docker-py/blob/master/docs/user_guides/swarm_services.md


The ``command`` and ``args`` definitions depends on the image that you are using.
I.e the command must be possible to execute in the selected image
The '/usr/local/bin/start-singleuser.sh' is provided by the jupyter
`base-notebook <https://github.com/jupyter/docker-stacks/tree/master/base-notebook>`_
The start-singleuser.sh ``args`` assumes that the launched image is extended from a version of this.

.. code-block:: python

    c.SwarmSpawner.container_spec = {
                  # The command to run inside the service
                  'args' : ['/usr/local/bin/start-singleuser.sh']
          }


**Note:** in a container spec, ``args`` sets the equivalent of CMD in the Dockerfile, ``command`` sets the equivalent of ENTRYPOINT.
The notebook server command should not be the ENTRYPOINT, so generally use ``args``, not ``command``, to specify how to launch the notebook server.

See this `issue <https://github.com/cassinyio/SwarmSpawner/issues/6>`_  for more info.

Dockerimages
---------------------

To define which images are available to the users, a list of dockerimages must be declared
The individual dictionaries also makes it possible to define whether the image should mount any volumes when it is spawned

.. code-block:: python

    # Available docker images the user can spawn
    c.SwarmSpawner.dockerimages = [
        {'image': 'jupyter/base-notebook:30f16d52126f',
         'name': 'Minimal python notebook'},
        {'image': 'nielsbohr/base-notebook:latest',
         'name': 'Image with automatic {replace_me} mount, supports Py2/3 and R,',
         'mounts': mounts}
    ]


To make the user able to select between the available images, the following must be set.
If this is not the case, the user will simply spawn an instance of the default image. i.e. dockerimages[0]

.. code-block:: python

    # Before the user can select which image to spawn,
    # user_options has to be enabled
    c.SwarmSpawner.use_user_options = True

This enables an image select form in the users /hub/home url path when a notebook hasen't been spawned already.


Bind a Host dir
---------------------
With ``'type':'bind'`` you mount a local directory of the host inside the container.

*Remember that source should exist in the node where you are creating the service.*

.. code-block:: python

        notebook_dir = os.environ.get('NOTEBOOK_DIR') or '/home/jovyan/work'
        c.SwarmSpawner.notebook_dir = notebook_dir

.. code-block:: python

        mounts = [{'type' : 'bind',
                'source' : 'MountPointOnTheHost',
                'target' : 'MountPointInsideTheContainer',}]


Volumes
-------
With ``'type':'volume'`` you mount a Docker Volume inside the container.
If the volume doesn't exist it will be created.

.. code-block:: python

        mounts = [{'type' : 'volume',
                'source' : 'NameOfTheVolume',
                'target' : 'MountPointInsideTheContainer',}]


Named path
--------------
For both types, volume and bind, you can specify a ``{username}`` inside the source:

.. code-block:: python

        mounts = [{'type' : 'volume',
                'source' : 'jupyterhub-user-{username}',
                'target' : 'MountPointInsideTheContainer',}]


username will be the hashed version of the username.


Mount an anonymous volume
-------------------------
**This kind of volume will be removed with the service.**

.. code-block:: python

        mounts = [{'type' : 'volume',
                'target' : 'MountPointInsideTheContainer',}]


SSHFS mount
----------------

It is also possible to mount a volume that is an sshfs mount to another host
supports either passing ``{id_rsa}`` or ``{password}`` that should be used to authenticate,
in addition the typical sshfs flags are supported, defaults to port 22

.. code-block:: python


    mounts = [{'type': 'volume',
               'driver_config': 'rasmunk/sshfs:latest',
               'driver_options': {'sshcmd': '{sshcmd}', 'id_rsa': '{id_rsa}',
                                  'big_writes': '', 'allow_other': '',
                                  'reconnect': '', 'port': '2222'},
               'source': 'sshvolume-user-{username}',
               'target': '/home/jovyan/work'
               }]


Resource_spec
---------------

You can also specify some resource for each service

.. code-block:: python

        c.SwarmSpawner.resource_spec = {
                        'cpu_limit' : 1000, # (int) – CPU limit in units of 10^9 CPU shares.
                        'mem_limit' : int(512 * 1e6), # (int) – Memory limit in Bytes.
                        'cpu_reservation' : 1000, # (int) – CPU reservation in units of 10^9 CPU shares.
                        'mem_reservation' : int(512 * 1e6), # (int) – Memory reservation in Bytes
                        }

Using user_options
--------------------

There is the possibility to set parameters using ``user_options``

.. code-block:: python

        # To use user_options in service creation
        c.SwarmSpawner.use_user_options = False


To control the creation of the services you have 2 ways, using **jupyterhub_config.py** or **user_options**.

Remember that at the end you are just using the `Docker Engine API <https://docs.docker.com/engine/api/>`_.

**user_options, if used, will overwrite jupyter_config.py for services.**

If you set ``c.SwarmSpawner.use_user_option = True`` the spawner will use the dict passed through the form or as json body when using the Hub Api.

The spawner expect a dict with these keys:

.. code-block:: python

        user_options = {
                'container_spec' : {
                        # (string or list) command to run in the image.
                        'args' : ['/usr/local/bin/start-singleuser.sh'],
                        # name of the image
                        'Image' : '',
                        'mounts' : mounts,
                        'resource_spec' : {
                                # (int) – CPU limit in units of 10^9 CPU shares.
                                'cpu_limit': int(1 * 1e9),
                                # (int) – Memory limit in Bytes.
                                'mem_limit': int(512 * 1e6),
                                # (int) – CPU reservation in units of 10^9 CPU shares.
                                'cpu_reservation': int(1 * 1e9),
                                # (int) – Memory reservation in bytes
                                'mem_reservation': int(512 * 1e6),
                                },
                        # list of constrains
                        'placement' : [],
                        # list of networks
                        'network' : [],
                        # name of service
                        'name' : ''
                        }
                }


Names of the Jupyter notebook service inside Docker engine in Swarm mode
--------------------------------------------------------------------------

When JupyterHub spawns a new Jupyter notebook server the name of the service will be ``{service_prefix}-{service_owner}-{service_suffix}``

You can change the service_prefix in this way:

Prefix of the service in Docker

.. code-block:: python

        c.SwarmSpawner.service_prefix = "jupyterhub"


``service_owner`` is the hexdigest() of the hashed ``user.name``.

In case of named servers (more than one server for user) ``service_suffix`` is the name of the server, otherwise is always 1.

Downloading images
-------------------
Docker Engine in Swarm mode downloads images automatically from the repository.
Either the image is available on the remote repository or locally, if not you will get an error.

Because before starting the service you have to complete the download of the image is better to have a longer timeout (default is 30 secs)

.. code-block:: python

        c.SwarmSpawner.start_timeout = 60 * 5


You can use all the docker images inside the `Jupyter docker-stacks`_.

.. _Jupyter docker-stacks: https://github.com/jupyter/docker-stacks


Credit
=======
`DockerSpawner <https://github.com/jupyterhub/dockerspawner>`_
`CassinyioSpawner <https://github.com/cassinyio/SwarmSpawner>`_


License
=======
All code is licensed under the terms of the revised BSD license.


            

Raw data

            {
    "maintainer": "", 
    "docs_url": null, 
    "requires_python": "", 
    "maintainer_email": "", 
    "cheesecake_code_kwalitee_id": null, 
    "keywords": "Interactive,Interpreter,Shell,Web", 
    "upload_time": "2018-04-03 15:06:15", 
    "author": "Rasmus Munk", 
    "home_page": "https://github.com/rasmunk/SwarmSpawner", 
    "github_user": "rasmunk", 
    "download_url": "https://pypi.python.org/packages/6e/60/8d173d364ce86b59a2deda87d78061ec1bf9109dadc88e0ef7e3313e4130/mig-swarmspawner-0.0.8.tar.gz", 
    "platform": "", 
    "version": "0.0.8", 
    "cheesecake_documentation_id": null, 
    "description": "==============================\nSwarmSpawner\n==============================\n\n.. image:: https://travis-ci.org/rasmunk/SwarmSpawner.svg?branch=master\n    :target: https://travis-ci.org/rasmunk/SwarmSpawner\n\n\n**MiG SwarmSpawner** enables `JupyterHub <https://github\n.com/jupyterhub/jupyterhub>`_ to spawn jupyter notebooks that can interact\nwith the MiG infrastructure.\n\nMore info about Docker Services `here <https://docs.docker.com/engine/reference/commandline/service_create/>`_.\n\n\nPrerequisites\n================\n\nPython version 3.3 and above is required.\n\n\nInstallation\n================\n\n.. code-block:: sh\n\n   pip install mig-swarmspawner\n\nInstallation from GitHub\n============================\n\n.. code-block:: sh\n\n   git clone https://github.com/rasmunk/SwarmSpawner\n   cd SwarmSpawner\n   python setup.py install\n\nConfiguration\n================\n\nYou can find an example jupyter_config.py inside `examples <examples>`_.\n\nThe spawner\n================\nDocker Engine in Swarm mode and the related services work in a different way compared to Docker containers.\n\nTell JupyterHub to use SwarmSpawner by adding the following lines to your `jupyterhub_config.py`:\n\n.. code-block:: python\n\n        c.JupyterHub.spawner_class = 'mig.SwarmSpawner'\n        c.JupyterHub.hub_ip = '0.0.0.0'\n        # This should be the name of the jupyterhub service\n        c.SwarmSpawner.jupyterhub_service_name = 'NameOfTheService'\n\nWhat is ``jupyterhub_service_name``?\n\nInside a Docker engine in Swarm mode the services use a `name` instead of a `ip` to communicate with each other.\n'jupyterhub_service_name' is the name of ther service for the JupyterHub.\n\nNetworks\n============\nIt's important to put the JupyterHub service (also the proxy) and the services that are running jupyter notebook inside the same network, otherwise they couldn't reach each other.\nSwarmSpawner use the service's name instead of the service's ip, as a consequence JupyterHub and servers should share the same overlay network (network across nodes).\n\n.. code-block:: python\n\n        #list of networks\n        c.SwarmSpawner.networks = [\"mynetwork\"]\n\n\nDefine the services inside jupyterhub_config.py\n===============================================\nYou can define *container_spec*, *resource_spec* and *networks* inside **jupyterhub_config.py**.\n\nContainer_spec__\n-------------------\n__ https://github.com/docker/docker-py/blob/master/docs/user_guides/swarm_services.md\n\n\nThe ``command`` and ``args`` definitions depends on the image that you are using.\nI.e the command must be possible to execute in the selected image\nThe '/usr/local/bin/start-singleuser.sh' is provided by the jupyter\n`base-notebook <https://github.com/jupyter/docker-stacks/tree/master/base-notebook>`_\nThe start-singleuser.sh ``args`` assumes that the launched image is extended from a version of this.\n\n.. code-block:: python\n\n    c.SwarmSpawner.container_spec = {\n                  # The command to run inside the service\n                  'args' : ['/usr/local/bin/start-singleuser.sh']\n          }\n\n\n**Note:** in a container spec, ``args`` sets the equivalent of CMD in the Dockerfile, ``command`` sets the equivalent of ENTRYPOINT.\nThe notebook server command should not be the ENTRYPOINT, so generally use ``args``, not ``command``, to specify how to launch the notebook server.\n\nSee this `issue <https://github.com/cassinyio/SwarmSpawner/issues/6>`_  for more info.\n\nDockerimages\n---------------------\n\nTo define which images are available to the users, a list of dockerimages must be declared\nThe individual dictionaries also makes it possible to define whether the image should mount any volumes when it is spawned\n\n.. code-block:: python\n\n    # Available docker images the user can spawn\n    c.SwarmSpawner.dockerimages = [\n        {'image': 'jupyter/base-notebook:30f16d52126f',\n         'name': 'Minimal python notebook'},\n        {'image': 'nielsbohr/base-notebook:latest',\n         'name': 'Image with automatic {replace_me} mount, supports Py2/3 and R,',\n         'mounts': mounts}\n    ]\n\n\nTo make the user able to select between the available images, the following must be set.\nIf this is not the case, the user will simply spawn an instance of the default image. i.e. dockerimages[0]\n\n.. code-block:: python\n\n    # Before the user can select which image to spawn,\n    # user_options has to be enabled\n    c.SwarmSpawner.use_user_options = True\n\nThis enables an image select form in the users /hub/home url path when a notebook hasen't been spawned already.\n\n\nBind a Host dir\n---------------------\nWith ``'type':'bind'`` you mount a local directory of the host inside the container.\n\n*Remember that source should exist in the node where you are creating the service.*\n\n.. code-block:: python\n\n        notebook_dir = os.environ.get('NOTEBOOK_DIR') or '/home/jovyan/work'\n        c.SwarmSpawner.notebook_dir = notebook_dir\n\n.. code-block:: python\n\n        mounts = [{'type' : 'bind',\n                'source' : 'MountPointOnTheHost',\n                'target' : 'MountPointInsideTheContainer',}]\n\n\nVolumes\n-------\nWith ``'type':'volume'`` you mount a Docker Volume inside the container.\nIf the volume doesn't exist it will be created.\n\n.. code-block:: python\n\n        mounts = [{'type' : 'volume',\n                'source' : 'NameOfTheVolume',\n                'target' : 'MountPointInsideTheContainer',}]\n\n\nNamed path\n--------------\nFor both types, volume and bind, you can specify a ``{username}`` inside the source:\n\n.. code-block:: python\n\n        mounts = [{'type' : 'volume',\n                'source' : 'jupyterhub-user-{username}',\n                'target' : 'MountPointInsideTheContainer',}]\n\n\nusername will be the hashed version of the username.\n\n\nMount an anonymous volume\n-------------------------\n**This kind of volume will be removed with the service.**\n\n.. code-block:: python\n\n        mounts = [{'type' : 'volume',\n                'target' : 'MountPointInsideTheContainer',}]\n\n\nSSHFS mount\n----------------\n\nIt is also possible to mount a volume that is an sshfs mount to another host\nsupports either passing ``{id_rsa}`` or ``{password}`` that should be used to authenticate,\nin addition the typical sshfs flags are supported, defaults to port 22\n\n.. code-block:: python\n\n\n    mounts = [{'type': 'volume',\n               'driver_config': 'rasmunk/sshfs:latest',\n               'driver_options': {'sshcmd': '{sshcmd}', 'id_rsa': '{id_rsa}',\n                                  'big_writes': '', 'allow_other': '',\n                                  'reconnect': '', 'port': '2222'},\n               'source': 'sshvolume-user-{username}',\n               'target': '/home/jovyan/work'\n               }]\n\n\nResource_spec\n---------------\n\nYou can also specify some resource for each service\n\n.. code-block:: python\n\n        c.SwarmSpawner.resource_spec = {\n                        'cpu_limit' : 1000, # (int) \u2013 CPU limit in units of 10^9 CPU shares.\n                        'mem_limit' : int(512 * 1e6), # (int) \u2013 Memory limit in Bytes.\n                        'cpu_reservation' : 1000, # (int) \u2013 CPU reservation in units of 10^9 CPU shares.\n                        'mem_reservation' : int(512 * 1e6), # (int) \u2013 Memory reservation in Bytes\n                        }\n\nUsing user_options\n--------------------\n\nThere is the possibility to set parameters using ``user_options``\n\n.. code-block:: python\n\n        # To use user_options in service creation\n        c.SwarmSpawner.use_user_options = False\n\n\nTo control the creation of the services you have 2 ways, using **jupyterhub_config.py** or **user_options**.\n\nRemember that at the end you are just using the `Docker Engine API <https://docs.docker.com/engine/api/>`_.\n\n**user_options, if used, will overwrite jupyter_config.py for services.**\n\nIf you set ``c.SwarmSpawner.use_user_option = True`` the spawner will use the dict passed through the form or as json body when using the Hub Api.\n\nThe spawner expect a dict with these keys:\n\n.. code-block:: python\n\n        user_options = {\n                'container_spec' : {\n                        # (string or list) command to run in the image.\n                        'args' : ['/usr/local/bin/start-singleuser.sh'],\n                        # name of the image\n                        'Image' : '',\n                        'mounts' : mounts,\n                        'resource_spec' : {\n                                # (int) \u2013 CPU limit in units of 10^9 CPU shares.\n                                'cpu_limit': int(1 * 1e9),\n                                # (int) \u2013 Memory limit in Bytes.\n                                'mem_limit': int(512 * 1e6),\n                                # (int) \u2013 CPU reservation in units of 10^9 CPU shares.\n                                'cpu_reservation': int(1 * 1e9),\n                                # (int) \u2013 Memory reservation in bytes\n                                'mem_reservation': int(512 * 1e6),\n                                },\n                        # list of constrains\n                        'placement' : [],\n                        # list of networks\n                        'network' : [],\n                        # name of service\n                        'name' : ''\n                        }\n                }\n\n\nNames of the Jupyter notebook service inside Docker engine in Swarm mode\n--------------------------------------------------------------------------\n\nWhen JupyterHub spawns a new Jupyter notebook server the name of the service will be ``{service_prefix}-{service_owner}-{service_suffix}``\n\nYou can change the service_prefix in this way:\n\nPrefix of the service in Docker\n\n.. code-block:: python\n\n        c.SwarmSpawner.service_prefix = \"jupyterhub\"\n\n\n``service_owner`` is the hexdigest() of the hashed ``user.name``.\n\nIn case of named servers (more than one server for user) ``service_suffix`` is the name of the server, otherwise is always 1.\n\nDownloading images\n-------------------\nDocker Engine in Swarm mode downloads images automatically from the repository.\nEither the image is available on the remote repository or locally, if not you will get an error.\n\nBecause before starting the service you have to complete the download of the image is better to have a longer timeout (default is 30 secs)\n\n.. code-block:: python\n\n        c.SwarmSpawner.start_timeout = 60 * 5\n\n\nYou can use all the docker images inside the `Jupyter docker-stacks`_.\n\n.. _Jupyter docker-stacks: https://github.com/jupyter/docker-stacks\n\n\nCredit\n=======\n`DockerSpawner <https://github.com/jupyterhub/dockerspawner>`_\n`CassinyioSpawner <https://github.com/cassinyio/SwarmSpawner>`_\n\n\nLicense\n=======\nAll code is licensed under the terms of the revised BSD license.\n\n", 
    "lcname": "mig-swarmspawner", 
    "bugtrack_url": null, 
    "github": true, 
    "coveralls": false, 
    "name": "mig-swarmspawner", 
    "license": "BSD", 
    "travis_ci": true, 
    "github_project": "SwarmSpawner", 
    "summary": " MiG-SwarmSpawner enables JupyterHub to spawn jupyter notebooks that can interact with the MiG infrastructure.", 
    "split_keywords": [
        "interactive", 
        "interpreter", 
        "shell", 
        "web"
    ], 
    "author_email": "rasmus.munk@nbi.ku.dk", 
    "urls": [
        {
            "has_sig": false, 
            "upload_time": "2018-04-03T15:06:15", 
            "comment_text": "", 
            "python_version": "source", 
            "url": "https://pypi.python.org/packages/6e/60/8d173d364ce86b59a2deda87d78061ec1bf9109dadc88e0ef7e3313e4130/mig-swarmspawner-0.0.8.tar.gz", 
            "md5_digest": "9b1f913d846ba2d83cbb690f6a9a23b3", 
            "downloads": 0, 
            "filename": "mig-swarmspawner-0.0.8.tar.gz", 
            "packagetype": "sdist", 
            "path": "6e/60/8d173d364ce86b59a2deda87d78061ec1bf9109dadc88e0ef7e3313e4130/mig-swarmspawner-0.0.8.tar.gz", 
            "digests": {
                "sha256": "aafc61851fb9ce740a91798ab0680092c13b0b7df2f77fcde6e258f7e0301d43", 
                "md5": "9b1f913d846ba2d83cbb690f6a9a23b3"
            }, 
            "sha256_digest": "aafc61851fb9ce740a91798ab0680092c13b0b7df2f77fcde6e258f7e0301d43", 
            "size": 11268
        }, 
        {
            "has_sig": false, 
            "upload_time": "2018-04-03T15:06:14", 
            "comment_text": "", 
            "python_version": "py3", 
            "url": "https://pypi.python.org/packages/0a/5f/1a6265b1cd99829b1a728c546cd30576e978aacba706c70cf70367fc85ae/mig_swarmspawner-0.0.8-py3-none-any.whl", 
            "md5_digest": "ec4fcfd228582e3987f40851df4fe677", 
            "downloads": 0, 
            "filename": "mig_swarmspawner-0.0.8-py3-none-any.whl", 
            "packagetype": "bdist_wheel", 
            "path": "0a/5f/1a6265b1cd99829b1a728c546cd30576e978aacba706c70cf70367fc85ae/mig_swarmspawner-0.0.8-py3-none-any.whl", 
            "digests": {
                "sha256": "82406c028a7a0ace98b95a018d2f03044aa5b48c30c6ae95dcb73b17fa755511", 
                "md5": "ec4fcfd228582e3987f40851df4fe677"
            }, 
            "sha256_digest": "82406c028a7a0ace98b95a018d2f03044aa5b48c30c6ae95dcb73b17fa755511", 
            "size": 10991
        }
    ], 
    "_id": null, 
    "cheesecake_installability_id": null
}