How Makers Build
Check my stackPlan my stack
Menu

Apache Airflow

Open-source platform for writing, scheduling and monitoring batch data pipelines as Python DAGs.

Visit airflow.apache.org ↗Alternatives
Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

Apache Airflow runs recurring batch work that has to happen in a set order: pull data from an API, load it into a warehouse, rebuild a report, retrain a model. Instead of a pile of cron jobs that fail silently, each pipeline is one definition with a schedule, dependencies between steps, retries and a history of every run.

Pipelines are written in Python as DAGs (directed acyclic graphs) of tasks. A scheduler triggers DAGs on a time schedule, on events, or when an upstream DAG updates an asset, and hands tasks to an executor; a DAG processor parses your DAG files; an API server provides the REST API and the web UI; and a metadata database, usually PostgreSQL or MySQL, stores the state of every DAG and task.

What you get: a UI where you can trigger DAGs, read logs and watch task status, and executors that range from the default LocalExecutor for a single machine to Celery, Kubernetes and ECS for multi-machine setups. Locally, airflow standalone starts everything in one command. If you'd rather not run it, several clouds and vendors offer managed Airflow, among them Astro, Google Cloud Composer and Amazon MWAA.

The limits: Airflow is built for batch workflows with a clear start and end, not streaming or per-request jobs from your app. XComs, the way tasks pass values to each other, are meant only for small amounts of data, so datasets go through your own storage. Running it yourself means operating several components and a database, and the docs say there is no fixed minimum spec for production.

Where it fits

Background Jobs & Cron · Data pipeline orchestratorScheduled batch pipelines — loading data, rebuilding reports, retraining models — written as Python DAGs, with managed versions such as Astro, Google Cloud Composer and Amazon MWAA if you don't want to run it.Trade-off: You run a scheduler, DAG processor, API server and a Postgres or MySQL database, or pay for a managed service; built for batch, not per-request app jobs or streaming, and tasks pass only small values to each other.

Who uses it

No maker's live product on record yet, and 2 open-source projects that declare it in their code.

Declared in code 2How evidence is collected →

We haven't found a maker's live product that uses Apache Airflow yet — only the open-source projects below, which declare it in their code.

Open source: a project that declares Apache Airflow as a dependency in its public code — verifiable, but not necessarily a live product.

Alternatives to Apache Airflow

All alternatives by situation →
Trigger.devLong-running TypeScript tasks — AI calls, media processing — that would time out in a serverless function.Open sourceFree tier23 makers · background jobs & cron
InngestEvent-driven, multi-step workflows where each step retries on its own and your code stays in your existing app.Open sourceFree tier16 makers · background jobs & cron
TemporalLong-running, multi-step workflows — payments, onboarding, AI agent loops — whose state must survive crashes and deploys, in Go, Java, Python, TypeScript and more.Open sourceFree tier35 makers · background jobs & cron
UpstashSimple scheduled calls and delayed HTTP jobs on serverless or edge hosting, via QStash.Free tier2 makers · background jobs & cron
BullMQNode apps on a long-running server that already have Redis — free and fully in your control.Open sourceFree tier1 makers · background jobs & cron
RabbitMQA dedicated message broker for queues and pub/sub between services, in any language, or as the broker behind Celery.Open sourceFree tier3 makers · background jobs & cron
DagsterData pipelines described by the tables, files and models they produce, with lineage between them, schedules, sensors and native dbt integration, in Python.Open sourceFree tier3 makers · background jobs & cron

Questions makers ask about Apache Airflow

What kind of work is Airflow for?

Batch-oriented workflows, time-based or event-triggered, that have a clear start and end and run on a schedule. It is not designed for streaming, and the docs note that if you prefer clicking over coding it may not be the best fit. source ↗

How is it different from a job queue like Celery?

A job queue runs individual tasks your app enqueues; Airflow schedules whole pipelines of dependent tasks and tracks each run. It can even use Celery underneath, as one of its executors alongside the default LocalExecutor, Kubernetes and ECS. source ↗

What do I need to run it myself?

A scheduler, a DAG processor, an API server that serves the UI and REST API, and a metadata database, usually PostgreSQL or MySQL. For small, single-machine production installs the docs say the LocalExecutor is enough; multi-machine setups use a remote executor. source ↗

Is there a hosted version?

Not from the Apache project itself, but the Airflow site lists managed services including Astro, Google Cloud Composer, Amazon Managed Workflows for Apache Airflow and Azure Data Factory Managed Airflow. source ↗

Can tasks pass data to each other?

Through XComs, but the docs say they are only designed for small amounts of data and should not be used to pass large values like dataframes. Larger data goes through your own storage. source ↗

Which Python versions are supported, and how do I try it locally?

Since Airflow 3.2.0 it supports Python 3.10 through 3.14. Running airflow standalone initializes the database, creates a user and starts all components; for production the docs recommend running the components separately. source ↗

Is Apache Airflow free?

Yes — it is free, open-source software.

Is Apache Airflow open source or self-hostable?

Open source, and you can self-host it. source ↗

Can AI coding agents work with Apache Airflow?

No llms.txt, official MCP server or CLI found yet.

Who uses Apache Airflow?

No maker's live product we track yet; 2 open-source projects declare it in their code. source ↗