Apache Airflow

Linux

Airflow is a platform to programmaticaly author, schedule and monitor data pipelines.

Workflow Automation

Tool

StackStorm

Self-Hosted

StackStorm is a powerful open-source automation platform that wires together all of your apps, services and workflows. It’s extendable, flexible, and built with love for DevOps and ChatOps.

Workflows & Scheduling cares about your time and money, so it helps you troubleshooting any inconveniences you may experience during the execution or development of your business software by providing a very wide range of monitoring mechanisms accessible through our Web Portals and APIs.

At any time, you can supervise your jobs and its tasks, and get fine details on its duration, results, execution node, and failures, among others. Get the state of your execution nodes themselves: disk space left, CPU usage, memory usage, network usage, disk IO usage, list of processes. Do not forget to supervise your virtual resources: as your infrastructure grows with our cloud connectors, your supervision and control on the new virtual resources will grow too, allowing you to get vendor specific information from the IaaS service provider monitoring API.

The Scheduler Web Portal is dedicated to users running workflows. From there you can submit workflows, monitor their execution and retrieve results. Remote visualization of graphical tasks is also possible directly from the browser. Progress of Workflow execution is graphically visible in the portal.

The Resource Manager Web Portal allows administrators to operate the platform and users to monitor their resources. It is the place where you can connect compute resources and monitor them, and even control dynamic policy-based provisioning of resources.

Aggregate and manage resources from different infrastructures. Resources coming from multiple origins are unified as ProActive Nodes and can be accessed transparently: Desktop machines (Windows, Linux, MacOS X), all kind of stand-alone server machines, cluster nodes managed by common batch-schedulers (Slurm, LSF, SGE) and private or public Clouds resources (OpenStack, CloudStack, VMWare, Windows Azure, Amazon EC2, Numergy, etc.)

Control the resources acquisition behavior according to various policies such as Load based or Time based.

ProActive Workflows & Scheduling

Windows

ProActive Workflows &amp; Scheduling allows you to easily execute all your company jobs and business applications, monitor activity and quickly access job results.

Metaflow

Metaflow is a human-friendly Python library that helps scientists and engineers build and manage real-life data science projects.

Luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Azkaban

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.

Flowsaber

A dataflow-based python workflow framework.

Oozie is a workflow scheduler system to manage Apache Hadoop jobs.

Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions.

Oozie Coordinator jobs are recurrent Oozie Workflow jobs triggered by time (frequency) and data availability.

Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp) as well as system specific jobs (such as Java programs and shell scripts).

Oozie is a scalable, reliable and extensible system.

Apache Oozie Alternatives

Platforms

Features & Tags

License