.. note::
For the latest source, discussion, etc, please visit the
`GitHub repository <https://github.com/spotify/luigi>`_
.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png
:alt: Luigi Logo
:align: center
.. image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fspotify%2Fluigi%2Fbadge&label=build&logo=none&%3Fref%3Dmaster&style=flat
:target: https://actions-badge.atrox.dev/spotify/luigi/goto?ref=master
.. image:: https://img.shields.io/codecov/c/github/spotify/luigi/master.svg?style=flat
:target: https://codecov.io/gh/spotify/luigi?branch=master
.. image:: https://img.shields.io/pypi/v/luigi.svg?style=flat
:target: https://pypi.python.org/pypi/luigi
.. image:: https://img.shields.io/pypi/l/luigi.svg?style=flat
:target: https://pypi.python.org/pypi/luigi
.. image:: https://readthedocs.org/projects/luigi/badge/?version=stable
:target: https://luigi.readthedocs.io/en/stable/?badge=stable
:alt: Documentation Status
Luigi is a Python (3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 tested) package that helps you build complex
pipelines of batch jobs. It handles dependency resolution, workflow management,
visualization, handling failures, command line integration, and much more.
Getting Started
---------------
Run ``pip install luigi`` to install the latest stable version from `PyPI
<https://pypi.python.org/pypi/luigi>`_. `Documentation for the latest release
<https://luigi.readthedocs.io/en/stable/>`__ is hosted on readthedocs.
Run ``pip install luigi[toml]`` to install Luigi with `TOML-based configs
<https://luigi.readthedocs.io/en/stable/configuration.html>`__ support.
For the bleeding edge code, ``pip install
git+https://github.com/spotify/luigi.git``. `Bleeding edge documentation
<https://luigi.readthedocs.io/en/latest/>`__ is also available.
Background
----------
The purpose of Luigi is to address all the plumbing typically associated
with long-running batch processes. You want to chain many tasks,
automate them, and failures *will* happen. These tasks can be anything,
but are typically long running things like
`Hadoop <http://hadoop.apache.org/>`_ jobs, dumping data to/from
databases, running machine learning algorithms, or anything else.
There are other software packages that focus on lower level aspects of
data processing, like `Hive <http://hive.apache.org/>`__,
`Pig <http://pig.apache.org/>`_, or
`Cascading <http://www.cascading.org/>`_. Luigi is not a framework to
replace these. Instead it helps you stitch many tasks together, where
each task can be a `Hive query <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
a `Hadoop job in Java <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop_jar.html>`_,
a `Spark job in Scala or Python <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.spark.html>`_,
a Python snippet,
`dumping a table <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.sqla.html>`_
from a database, or anything else. It's easy to build up
long-running pipelines that comprise thousands of tasks and take days or
weeks to complete. Luigi takes care of a lot of the workflow management
so that you can focus on the tasks themselves and their dependencies.
You can build pretty much any task you want, but Luigi also comes with a
*toolbox* of several common task templates that you use. It includes
support for running
`Python mapreduce jobs <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop.html>`_
in Hadoop, as well as
`Hive <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
and `Pig <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`__,
jobs. It also comes with
`file system abstractions for HDFS <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hdfs.html>`_,
and local files that ensures all file system operations are atomic. This
is important because it means your data pipeline will not crash in a
state containing partial data.
Visualiser page
---------------
The Luigi server comes with a web interface too, so you can search and filter
among all your tasks.
.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/visualiser_front_page.png
:alt: Visualiser page
Dependency graph example
------------------------
Just to give you an idea of what Luigi does, this is a screen shot from
something we are running in production. Using Luigi's visualiser, we get
a nice visual overview of the dependency graph of the workflow. Each
node represents a task which has to be run. Green tasks are already
completed whereas yellow tasks are yet to be run. Most of these tasks
are Hadoop jobs, but there are also some things that run locally and
build up data files.
.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/user_recs.png
:alt: Dependency graph
Philosophy
----------
Conceptually, Luigi is similar to `GNU
Make <http://www.gnu.org/software/make/>`_ where you have certain tasks
and these tasks in turn may have dependencies on other tasks. There are
also some similarities to `Oozie <http://oozie.apache.org/>`_
and `Azkaban <https://azkaban.github.io/>`_. One major
difference is that Luigi is not just built specifically for Hadoop, and
it's easy to extend it with other kinds of tasks.
Everything in Luigi is in Python. Instead of XML configuration or
similar external data files, the dependency graph is specified *within
Python*. This makes it easy to build up complex dependency graphs of
tasks, where the dependencies can involve date algebra or recursive
references to other versions of the same task. However, the workflow can
trigger things not in Python, such as running
`Pig scripts <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`_
or `scp'ing files <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.ssh.html>`_.
Who uses Luigi?
---------------
We use Luigi internally at `Spotify <https://www.spotify.com>`_ to run
thousands of tasks every day, organized in complex dependency graphs.
Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
that powers all kinds of stuff including recommendations, toplists, A/B
test analysis, external reports, internal dashboards, etc.
Since Luigi is open source and without any registration walls, the exact number
of Luigi users is unknown. But based on the number of unique contributors, we
expect hundreds of enterprises to use it. Some users have written blog posts
or held presentations about Luigi:
* `Spotify <https://www.spotify.com>`_ `(presentation, 2014) <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`__
* `Foursquare <https://foursquare.com/>`_ `(presentation, 2013) <http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897>`__
* `Mortar Data (Datadog) <https://www.datadoghq.com/>`_ `(documentation / tutorial) <http://help.mortardata.com/technologies/luigi>`__
* `Stripe <https://stripe.com/>`_ `(presentation, 2014) <http://www.slideshare.net/PyData/python-as-part-of-a-production-machine-learning-stack-by-michael-manapat-pydata-sv-2014>`__
* `Buffer <https://buffer.com/>`_ `(blog, 2014) <https://buffer.com/resources/buffers-new-data-architecture/>`__
* `SeatGeek <https://seatgeek.com/>`_ `(blog, 2015) <http://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/>`__
* `Treasure Data <https://www.treasuredata.com/>`_ `(blog, 2015) <http://blog.treasuredata.com/blog/2015/02/25/managing-the-data-pipeline-with-git-luigi/>`__
* `Growth Intelligence <http://growthintel.com/>`_ `(presentation, 2015) <http://www.slideshare.net/growthintel/a-beginners-guide-to-building-data-pipelines-with-luigi>`__
* `AdRoll <https://www.adroll.com/>`_ `(blog, 2015) <http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html>`__
* 17zuoye `(presentation, 2015) <https://speakerdeck.com/mvj3/luiti-an-offline-task-management-framework>`__
* `Custobar <https://www.custobar.com/>`_ `(presentation, 2016) <http://www.slideshare.net/teemukurppa/managing-data-workflows-with-luigi>`__
* `Blendle <https://launch.blendle.com/>`_ `(presentation) <http://www.anneschuth.nl/wp-content/uploads/sea-anneschuth-streamingblendle.pdf#page=126>`__
* `TrustYou <http://www.trustyou.com/>`_ `(presentation, 2015) <https://speakerdeck.com/mfcabrera/pydata-berlin-2015-processing-hotel-reviews-with-python>`__
* `Groupon <https://www.groupon.com/>`_ / `OrderUp <https://orderup.com>`_ `(alternative implementation) <https://github.com/groupon/luigi-warehouse>`__
* `Red Hat - Marketing Operations <https://www.redhat.com>`_ `(blog, 2017) <https://github.com/rh-marketingops/rh-mo-scc-luigi>`__
* `GetNinjas <https://www.getninjas.com.br/>`_ `(blog, 2017) <https://labs.getninjas.com.br/using-luigi-to-create-and-monitor-pipelines-of-batch-jobs-eb8b3cd2a574>`__
* `voyages-sncf.com <https://www.voyages-sncf.com/>`_ `(presentation, 2017) <https://github.com/voyages-sncf-technologies/meetup-afpy-nantes-luigi>`__
* `Open Targets <https://www.opentargets.org/>`_ `(blog, 2017) <https://blog.opentargets.org/using-containers-with-luigi>`__
* `Leipzig University Library <https://ub.uni-leipzig.de>`_ `(presentation, 2016) <https://de.slideshare.net/MartinCzygan/build-your-own-discovery-index-of-scholary-eresources>`__ / `(project) <https://finc.info/de/datenquellen>`__
* `Synetiq <https://synetiq.net/>`_ `(presentation, 2017) <https://www.youtube.com/watch?v=M4xUQXogSfo>`__
* `Glossier <https://www.glossier.com/>`_ `(blog, 2018) <https://medium.com/glossier/how-to-build-a-data-warehouse-what-weve-learned-so-far-at-glossier-6ff1e1783e31>`__
* `Data Revenue <https://www.datarevenue.com/>`_ `(blog, 2018) <https://www.datarevenue.com/en/blog/how-to-scale-your-machine-learning-pipeline>`_
* `Uppsala University <http://pharmb.io>`_ `(tutorial) <http://uppnex.se/twiki/do/view/Courses/EinfraMPS2015/Luigi.html>`_ / `(presentation, 2015) <https://www.youtube.com/watch?v=f26PqSXZdWM>`_ / `(slides, 2015) <https://www.slideshare.net/SamuelLampa/building-workflows-with-spotifys-luigi>`_ / `(poster, 2015) <https://pharmb.io/poster/2015-sciluigi/>`_ / `(paper, 2016) <https://doi.org/10.1186/s13321-016-0179-6>`_ / `(project) <https://github.com/pharmbio/sciluigi>`_
* `GIPHY <https://giphy.com/>`_ `(blog, 2019) <https://engineering.giphy.com/luigi-the-10x-plumber-containerizing-scaling-luigi-in-kubernetes/>`__
* `xtream <https://xtreamers.io/>`__ `(blog, 2019) <https://towardsdatascience.com/lessons-from-a-real-machine-learning-project-part-1-from-jupyter-to-luigi-bdfd0b050ca5>`__
* `CIAN <https://cian.ru/>`__ `(presentation, 2019) <https://www.highload.ru/moscow/2019/abstracts/6030>`__
Some more companies are using Luigi but haven't had a chance yet to write about it:
* `Schibsted <http://www.schibsted.com/>`_
* `enbrite.ly <http://enbrite.ly/>`_
* `Dow Jones / The Wall Street Journal <http://wsj.com>`_
* `Hotels.com <https://hotels.com>`_
* `Newsela <https://newsela.com>`_
* `Squarespace <https://www.squarespace.com/>`_
* `OAO <https://adops.com/>`_
* `Grovo <https://grovo.com/>`_
* `Weebly <https://www.weebly.com/>`_
* `Deloitte <https://www.Deloitte.co.uk/>`_
* `Stacktome <https://stacktome.com/>`_
* `LINX+Neemu+Chaordic <https://www.chaordic.com.br/>`_
* `Foxberry <https://www.foxberry.com/>`_
* `Okko <https://okko.tv/>`_
* `ISVWorld <http://isvworld.com/>`_
* `Big Data <https://bigdata.com.br/>`_
* `Movio <https://movio.co.nz/>`_
* `Bonnier News <https://www.bonniernews.se/>`_
* `Starsky Robotics <https://www.starsky.io/>`_
* `BaseTIS <https://www.basetis.com/>`_
* `Hopper <https://www.hopper.com/>`_
* `VOYAGE GROUP/Zucks <https://zucks.co.jp/en/>`_
* `Textpert <https://www.textpert.ai/>`_
* `Tracktics <https://www.tracktics.com/>`_
* `Whizar <https://www.whizar.com/>`_
* `xtream <https://www.xtreamers.io/>`__
* `Skyscanner <https://www.skyscanner.net/>`_
* `Jodel <https://www.jodel.com/>`_
* `Mekar <https://mekar.id/en/>`_
* `M3 <https://corporate.m3.com/en/>`_
* `Assist Digital <https://www.assistdigital.com/>`_
* `Meltwater <https://www.meltwater.com/>`_
* `DevSamurai <https://www.devsamurai.com/>`_
* `Veridas <https://veridas.com/>`_
We're more than happy to have your company added here. Just send a PR on GitHub.
External links
--------------
* `Mailing List <https://groups.google.com/d/forum/luigi-user/>`_ for discussions and asking questions. (Google Groups)
* `Releases <https://pypi.python.org/pypi/luigi>`_ (PyPI)
* `Source code <https://github.com/spotify/luigi>`_ (GitHub)
* `Hubot Integration <https://github.com/houzz/hubot-luigi>`_ plugin for Slack, Hipchat, etc (GitHub)
Authors
-------
Luigi was built at `Spotify <https://www.spotify.com>`_, mainly by
`Erik Bernhardsson <https://github.com/erikbern>`_ and
`Elias Freider <https://github.com/freider>`_.
`Many other people <https://github.com/spotify/luigi/graphs/contributors>`_
have contributed since open sourcing in late 2012.
`Arash Rouhani <https://github.com/tarrasch>`_ was the chief maintainer from 2015 to 2019, and now
Spotify's Data Team maintains Luigi.
Raw data
{
"_id": null,
"home_page": "https://github.com/spotify/luigi",
"name": "luigi",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "The Luigi Authors",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/47/bf/40a9ea8860a0d6387fe974d10d93b539fb5e477fc590e5d7673d2322b42d/luigi-3.5.2.tar.gz",
"platform": null,
"description": "\n\n\n.. note::\n\n For the latest source, discussion, etc, please visit the\n `GitHub repository <https://github.com/spotify/luigi>`_\n\n\n.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png\n :alt: Luigi Logo\n :align: center\n\n.. image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fspotify%2Fluigi%2Fbadge&label=build&logo=none&%3Fref%3Dmaster&style=flat\n :target: https://actions-badge.atrox.dev/spotify/luigi/goto?ref=master\n\n.. image:: https://img.shields.io/codecov/c/github/spotify/luigi/master.svg?style=flat\n :target: https://codecov.io/gh/spotify/luigi?branch=master\n\n.. image:: https://img.shields.io/pypi/v/luigi.svg?style=flat\n :target: https://pypi.python.org/pypi/luigi\n\n.. image:: https://img.shields.io/pypi/l/luigi.svg?style=flat\n :target: https://pypi.python.org/pypi/luigi\n\n.. image:: https://readthedocs.org/projects/luigi/badge/?version=stable\n :target: https://luigi.readthedocs.io/en/stable/?badge=stable\n :alt: Documentation Status\n\nLuigi is a Python (3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 tested) package that helps you build complex\npipelines of batch jobs. It handles dependency resolution, workflow management,\nvisualization, handling failures, command line integration, and much more.\n\nGetting Started\n---------------\n\nRun ``pip install luigi`` to install the latest stable version from `PyPI\n<https://pypi.python.org/pypi/luigi>`_. `Documentation for the latest release\n<https://luigi.readthedocs.io/en/stable/>`__ is hosted on readthedocs.\n\nRun ``pip install luigi[toml]`` to install Luigi with `TOML-based configs\n<https://luigi.readthedocs.io/en/stable/configuration.html>`__ support.\n\nFor the bleeding edge code, ``pip install\ngit+https://github.com/spotify/luigi.git``. `Bleeding edge documentation\n<https://luigi.readthedocs.io/en/latest/>`__ is also available.\n\nBackground\n----------\n\nThe purpose of Luigi is to address all the plumbing typically associated\nwith long-running batch processes. You want to chain many tasks,\nautomate them, and failures *will* happen. These tasks can be anything,\nbut are typically long running things like\n`Hadoop <http://hadoop.apache.org/>`_ jobs, dumping data to/from\ndatabases, running machine learning algorithms, or anything else.\n\nThere are other software packages that focus on lower level aspects of\ndata processing, like `Hive <http://hive.apache.org/>`__,\n`Pig <http://pig.apache.org/>`_, or\n`Cascading <http://www.cascading.org/>`_. Luigi is not a framework to\nreplace these. Instead it helps you stitch many tasks together, where\neach task can be a `Hive query <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,\na `Hadoop job in Java <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop_jar.html>`_,\na `Spark job in Scala or Python <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.spark.html>`_,\na Python snippet,\n`dumping a table <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.sqla.html>`_\nfrom a database, or anything else. It's easy to build up\nlong-running pipelines that comprise thousands of tasks and take days or\nweeks to complete. Luigi takes care of a lot of the workflow management\nso that you can focus on the tasks themselves and their dependencies.\n\nYou can build pretty much any task you want, but Luigi also comes with a\n*toolbox* of several common task templates that you use. It includes\nsupport for running\n`Python mapreduce jobs <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop.html>`_\nin Hadoop, as well as\n`Hive <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,\nand `Pig <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`__,\njobs. It also comes with\n`file system abstractions for HDFS <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hdfs.html>`_,\nand local files that ensures all file system operations are atomic. This\nis important because it means your data pipeline will not crash in a\nstate containing partial data.\n\nVisualiser page\n---------------\n\nThe Luigi server comes with a web interface too, so you can search and filter\namong all your tasks.\n\n.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/visualiser_front_page.png\n :alt: Visualiser page\n\nDependency graph example\n------------------------\n\nJust to give you an idea of what Luigi does, this is a screen shot from\nsomething we are running in production. Using Luigi's visualiser, we get\na nice visual overview of the dependency graph of the workflow. Each\nnode represents a task which has to be run. Green tasks are already\ncompleted whereas yellow tasks are yet to be run. Most of these tasks\nare Hadoop jobs, but there are also some things that run locally and\nbuild up data files.\n\n.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/user_recs.png\n :alt: Dependency graph\n\nPhilosophy\n----------\n\nConceptually, Luigi is similar to `GNU\nMake <http://www.gnu.org/software/make/>`_ where you have certain tasks\nand these tasks in turn may have dependencies on other tasks. There are\nalso some similarities to `Oozie <http://oozie.apache.org/>`_\nand `Azkaban <https://azkaban.github.io/>`_. One major\ndifference is that Luigi is not just built specifically for Hadoop, and\nit's easy to extend it with other kinds of tasks.\n\nEverything in Luigi is in Python. Instead of XML configuration or\nsimilar external data files, the dependency graph is specified *within\nPython*. This makes it easy to build up complex dependency graphs of\ntasks, where the dependencies can involve date algebra or recursive\nreferences to other versions of the same task. However, the workflow can\ntrigger things not in Python, such as running\n`Pig scripts <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`_\nor `scp'ing files <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.ssh.html>`_.\n\nWho uses Luigi?\n---------------\n\nWe use Luigi internally at `Spotify <https://www.spotify.com>`_ to run\nthousands of tasks every day, organized in complex dependency graphs.\nMost of these tasks are Hadoop jobs. Luigi provides an infrastructure\nthat powers all kinds of stuff including recommendations, toplists, A/B\ntest analysis, external reports, internal dashboards, etc.\n\nSince Luigi is open source and without any registration walls, the exact number\nof Luigi users is unknown. But based on the number of unique contributors, we\nexpect hundreds of enterprises to use it. Some users have written blog posts\nor held presentations about Luigi:\n\n* `Spotify <https://www.spotify.com>`_ `(presentation, 2014) <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`__\n* `Foursquare <https://foursquare.com/>`_ `(presentation, 2013) <http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897>`__\n* `Mortar Data (Datadog) <https://www.datadoghq.com/>`_ `(documentation / tutorial) <http://help.mortardata.com/technologies/luigi>`__\n* `Stripe <https://stripe.com/>`_ `(presentation, 2014) <http://www.slideshare.net/PyData/python-as-part-of-a-production-machine-learning-stack-by-michael-manapat-pydata-sv-2014>`__\n* `Buffer <https://buffer.com/>`_ `(blog, 2014) <https://buffer.com/resources/buffers-new-data-architecture/>`__\n* `SeatGeek <https://seatgeek.com/>`_ `(blog, 2015) <http://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/>`__\n* `Treasure Data <https://www.treasuredata.com/>`_ `(blog, 2015) <http://blog.treasuredata.com/blog/2015/02/25/managing-the-data-pipeline-with-git-luigi/>`__\n* `Growth Intelligence <http://growthintel.com/>`_ `(presentation, 2015) <http://www.slideshare.net/growthintel/a-beginners-guide-to-building-data-pipelines-with-luigi>`__\n* `AdRoll <https://www.adroll.com/>`_ `(blog, 2015) <http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html>`__\n* 17zuoye `(presentation, 2015) <https://speakerdeck.com/mvj3/luiti-an-offline-task-management-framework>`__\n* `Custobar <https://www.custobar.com/>`_ `(presentation, 2016) <http://www.slideshare.net/teemukurppa/managing-data-workflows-with-luigi>`__\n* `Blendle <https://launch.blendle.com/>`_ `(presentation) <http://www.anneschuth.nl/wp-content/uploads/sea-anneschuth-streamingblendle.pdf#page=126>`__\n* `TrustYou <http://www.trustyou.com/>`_ `(presentation, 2015) <https://speakerdeck.com/mfcabrera/pydata-berlin-2015-processing-hotel-reviews-with-python>`__\n* `Groupon <https://www.groupon.com/>`_ / `OrderUp <https://orderup.com>`_ `(alternative implementation) <https://github.com/groupon/luigi-warehouse>`__\n* `Red Hat - Marketing Operations <https://www.redhat.com>`_ `(blog, 2017) <https://github.com/rh-marketingops/rh-mo-scc-luigi>`__\n* `GetNinjas <https://www.getninjas.com.br/>`_ `(blog, 2017) <https://labs.getninjas.com.br/using-luigi-to-create-and-monitor-pipelines-of-batch-jobs-eb8b3cd2a574>`__\n* `voyages-sncf.com <https://www.voyages-sncf.com/>`_ `(presentation, 2017) <https://github.com/voyages-sncf-technologies/meetup-afpy-nantes-luigi>`__\n* `Open Targets <https://www.opentargets.org/>`_ `(blog, 2017) <https://blog.opentargets.org/using-containers-with-luigi>`__\n* `Leipzig University Library <https://ub.uni-leipzig.de>`_ `(presentation, 2016) <https://de.slideshare.net/MartinCzygan/build-your-own-discovery-index-of-scholary-eresources>`__ / `(project) <https://finc.info/de/datenquellen>`__\n* `Synetiq <https://synetiq.net/>`_ `(presentation, 2017) <https://www.youtube.com/watch?v=M4xUQXogSfo>`__\n* `Glossier <https://www.glossier.com/>`_ `(blog, 2018) <https://medium.com/glossier/how-to-build-a-data-warehouse-what-weve-learned-so-far-at-glossier-6ff1e1783e31>`__\n* `Data Revenue <https://www.datarevenue.com/>`_ `(blog, 2018) <https://www.datarevenue.com/en/blog/how-to-scale-your-machine-learning-pipeline>`_\n* `Uppsala University <http://pharmb.io>`_ `(tutorial) <http://uppnex.se/twiki/do/view/Courses/EinfraMPS2015/Luigi.html>`_ / `(presentation, 2015) <https://www.youtube.com/watch?v=f26PqSXZdWM>`_ / `(slides, 2015) <https://www.slideshare.net/SamuelLampa/building-workflows-with-spotifys-luigi>`_ / `(poster, 2015) <https://pharmb.io/poster/2015-sciluigi/>`_ / `(paper, 2016) <https://doi.org/10.1186/s13321-016-0179-6>`_ / `(project) <https://github.com/pharmbio/sciluigi>`_\n* `GIPHY <https://giphy.com/>`_ `(blog, 2019) <https://engineering.giphy.com/luigi-the-10x-plumber-containerizing-scaling-luigi-in-kubernetes/>`__\n* `xtream <https://xtreamers.io/>`__ `(blog, 2019) <https://towardsdatascience.com/lessons-from-a-real-machine-learning-project-part-1-from-jupyter-to-luigi-bdfd0b050ca5>`__\n* `CIAN <https://cian.ru/>`__ `(presentation, 2019) <https://www.highload.ru/moscow/2019/abstracts/6030>`__\n\nSome more companies are using Luigi but haven't had a chance yet to write about it:\n\n* `Schibsted <http://www.schibsted.com/>`_\n* `enbrite.ly <http://enbrite.ly/>`_\n* `Dow Jones / The Wall Street Journal <http://wsj.com>`_\n* `Hotels.com <https://hotels.com>`_\n* `Newsela <https://newsela.com>`_\n* `Squarespace <https://www.squarespace.com/>`_\n* `OAO <https://adops.com/>`_\n* `Grovo <https://grovo.com/>`_\n* `Weebly <https://www.weebly.com/>`_\n* `Deloitte <https://www.Deloitte.co.uk/>`_\n* `Stacktome <https://stacktome.com/>`_\n* `LINX+Neemu+Chaordic <https://www.chaordic.com.br/>`_\n* `Foxberry <https://www.foxberry.com/>`_\n* `Okko <https://okko.tv/>`_\n* `ISVWorld <http://isvworld.com/>`_\n* `Big Data <https://bigdata.com.br/>`_\n* `Movio <https://movio.co.nz/>`_\n* `Bonnier News <https://www.bonniernews.se/>`_\n* `Starsky Robotics <https://www.starsky.io/>`_\n* `BaseTIS <https://www.basetis.com/>`_\n* `Hopper <https://www.hopper.com/>`_\n* `VOYAGE GROUP/Zucks <https://zucks.co.jp/en/>`_\n* `Textpert <https://www.textpert.ai/>`_\n* `Tracktics <https://www.tracktics.com/>`_\n* `Whizar <https://www.whizar.com/>`_\n* `xtream <https://www.xtreamers.io/>`__\n* `Skyscanner <https://www.skyscanner.net/>`_\n* `Jodel <https://www.jodel.com/>`_\n* `Mekar <https://mekar.id/en/>`_\n* `M3 <https://corporate.m3.com/en/>`_\n* `Assist Digital <https://www.assistdigital.com/>`_\n* `Meltwater <https://www.meltwater.com/>`_\n* `DevSamurai <https://www.devsamurai.com/>`_\n* `Veridas <https://veridas.com/>`_\n\nWe're more than happy to have your company added here. Just send a PR on GitHub.\n\nExternal links\n--------------\n\n* `Mailing List <https://groups.google.com/d/forum/luigi-user/>`_ for discussions and asking questions. (Google Groups)\n* `Releases <https://pypi.python.org/pypi/luigi>`_ (PyPI)\n* `Source code <https://github.com/spotify/luigi>`_ (GitHub)\n* `Hubot Integration <https://github.com/houzz/hubot-luigi>`_ plugin for Slack, Hipchat, etc (GitHub)\n\nAuthors\n-------\n\nLuigi was built at `Spotify <https://www.spotify.com>`_, mainly by\n`Erik Bernhardsson <https://github.com/erikbern>`_ and\n`Elias Freider <https://github.com/freider>`_.\n`Many other people <https://github.com/spotify/luigi/graphs/contributors>`_\nhave contributed since open sourcing in late 2012.\n`Arash Rouhani <https://github.com/tarrasch>`_ was the chief maintainer from 2015 to 2019, and now\nSpotify's Data Team maintains Luigi.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Workflow mgmgt + task scheduling + dependency resolution.",
"version": "3.5.2",
"project_urls": {
"Homepage": "https://github.com/spotify/luigi"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "47bf40a9ea8860a0d6387fe974d10d93b539fb5e477fc590e5d7673d2322b42d",
"md5": "9ad1406522eb16f25661068f1fbae511",
"sha256": "d000fe6a6ea77c9376674fe87a045ac00c3fcf7ebe8414655a06630aa9db5111"
},
"downloads": -1,
"filename": "luigi-3.5.2.tar.gz",
"has_sig": false,
"md5_digest": "9ad1406522eb16f25661068f1fbae511",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1226476,
"upload_time": "2024-09-04T11:52:42",
"upload_time_iso_8601": "2024-09-04T11:52:42.471851Z",
"url": "https://files.pythonhosted.org/packages/47/bf/40a9ea8860a0d6387fe974d10d93b539fb5e477fc590e5d7673d2322b42d/luigi-3.5.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-04 11:52:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "spotify",
"github_project": "luigi",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "luigi"
}
JSON
