Luigi

    Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

    • Free • Open Source
    • Linux
    ...
    Luigi info, screenshots & reviewsAlternatives to Luigi

    What is Luigi?

    The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

    There are other software packages that focus on lower level aspects of data processing, like Hive, Pig, or Cascading. Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python a Python snippet, dumping a table from a database, or anything else. It's easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete. Luigi takes care of a lot of the workflow management so that you can focus on the tasks themselves and their dependencies.

    You can build pretty much any task you want, but Luigi also comes with a toolbox of several common task templates that you use. It includes support for running Python mapreduce jobs in Hadoop, as well as Hive, and Pig, jobs. It also comes with file system abstractions for HDFS, and local files that ensures all file system operations are atomic. This is important because it means your data pipeline will not crash in a state containing partial data.
    Show full description ▾
    Luigi

    Summary

    Our users have written 0 comments and reviews about Luigi, and it has gotten 0 likes

    • Developed by Spotify
    • Open Source and Free product.
    • 10 alternatives listed

    Popular alternatives

    View all
    Luigi was added to AlternativeTo by thomasleveil on Dec 6, 2016 and this page was last updated Dec 6, 2016.

    Features Vote on or suggest new features

    This app doesn't have any features at the moment.

    Comments and Reviews Post a comment / review
     all • positive • negative      relevance • date

    No comments or reviews, maybe you want to be first?

    Categories

    No category, edit the app to suggest tags that will place this app in a category

    Tags

    • etl
    Luigi

    Links and Download

     Official Website

    Recent user activities on Luigi