►
From YouTube: Airflow on Kubernetes: Dynamic Workflows Simplified - Daniel Imberman, Bloomberg & Barni Seetharaman
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Airflow on Kubernetes: Dynamic Workflows Simplified - Daniel Imberman, Bloomberg & Barni Seetharaman, Google
Apache Airflow is an open source workflow orchestration engine that allows users to write Directed Acyclic Graph (DAG)-based workflows using a simple Python library. Airflow offers a wide range of native operators for services ranging from Spark and HBase to Google Cloud Platform (GCP) and Amazon Web Services (AWS). Until recently, the Airflow user experience has been hindered by the need to launch and maintain statically-sized Celery-based Airflow clusters. These clusters were both expensive (over and under-utilization) and complex (multiple points of failure). To address these issues, we developed and published a native Kubernetes Operator and Kubernetes Executor for Apache Airflow. These products allow one-step Airflow deployments, dynamic allocation of Airflow worker pods, full power over run-time environments, and per-task resource management.
To learn more: https://sched.co/GrUO
A
B
A
Out
of
the
way
pipelines
are
hard.
Let's
take
a
classic
example
that
you're
a
data
engineer
and
your
job
is
to
take
raw
data
and
turn
it
into
actionable
data.
So,
like
any
good
data
engineer,
you
build
a
spark
job
and
that
sparked
up
to
well.
You
set
it
to
run
once
a
day
and
then
your
data
scientist
come
to
you
and
say:
hey.
We've
actually
figured
out
a
really
cool
way
of
taking
the
results
of
your
spark
job,
adding
it
a
tensorflow
component
and
building
a
really
cool
NLP
model.
A
We
would
love
to
add
that
to
this
pipeline,
you've
created
to
run
once
a
day,
and
you
say,
of
course,
why?
Wouldn't
you
want
to
help
that
and
then
another
data
science
team
says:
we've
also
figured
out
a
way
to
take
Twitter
firehose
data
and
use
some
scikit-learn
model
to
build
a
different
NLP
model.
That's
especially
good
for
their
customers,
and
why?
A
It
has
a
really
active
open-source
community
and
a
large
number
of
operators
and
hooks
for
things
such
as
h,
TF
s,
spark
bash,
a
lot
of
GCP
functions
and
AWS
functions
and,
as
a
DevOps
engineer,
when
you
actually
look
into
the
airflow
dashboard
you're
able
to
see
all
of
your
pipelines
at
once
and
you're
able
to
see
which
one's
failed
and
you
can
dive
in
see
historical
data
and
check
out
the
failure
and
see
it
running
in
real
time,
see
each
of
the
tasks
and
go
in
and
say.
This
task
failed.
A
Give
me
the
logs
and
show
me
why
this
failed
and
how
I
can
remedy
this
problem.
This
is
what
the
airflow
Python
SDK
actually
looks
like.
It
has
this
concept
of
operators
where
each
operator
is
a
step,
so
this
one
step
you
would
use
to
describe
a
spark,
submit
command
a
bash
command
or
in
a
cloud
function
and
once
you've
defined
all
of
your
operators.
You
can
use
this
function
called
set
upstream
to
set
the
dependencies
and
once
you've
set
the
dependencies.
B
Thanks
to
you
so
I
feel
it's
a
very
complex
system
to
deploy
and
manage
so
I
flows
to
develop
during
the
vmas
pre
container
age.
I
would
say
it
has
multiple
components
like
the
UI
scheduler
worker,
nodes,
broker,
sequel
storage
and
file
storage,
and
it
can
be
deployed
in
many
configurations.
You
can
choose
your
executor,
you
can
change
your
DAC
source
and
also
you
can
change
your
broker
etcetera.
So
all
of
these,
combined
together
makes
our
flow
complex
to
deploy
and
manage
and
also
introduces
multiple
failure
points.
B
Thank
you,
so
one
of
the
common
executives
that
our
flow
users
use
is
the
celery
executor.
It
deploys
a
static
number
of
worker
nodes.
The
problem
with
that
is,
if
you
end
up
deploying
the
less
number
of
worker
nodes
under
provisioning,
the
gear
system
you
end
up,
you
know
most
of
your
DAGs
end
up
in
the
queue
and
you
end
up
missing
your
s.
La's
and
the
other
thing
is,
you
could
end
up
over
positioning
your
worker
nodes,
thereby
wasting
resources,
and
if
you
are
fiscally
and
inclined
your
money.
B
We
looked
at
this
and
we
looked
at
communities,
and
so
this
is
a
great
way
cuban
artists
can
solve
all
of
these
issues
by
container
rising
your
air
flow
components
and
running
it
on
communities.
You
can
reduce
deployment
and
management
complexity,
not
just
that
you
can
also
take
advantage
of
the
dynamic
resource.
Api
that
given
artists,
provides
to
have
better
resource
utilization
and
also
communities
recovers
from
enables
you
to
do
fault
remediation
and
automate
that
process,
so
that,
if
any
of
your
components
file
it
has
recovered
automatically.
B
A
We
wanted
the
ability
to
give
give
air
flow
an
arbitrary
docker
container
and
run
this
container,
monitor
it
and
only
alert
us
about
whether
that
container
passed
or
failed
at
the
command.
It's
attempting
to
do
so.
The
kubernetes
pot
operator
does
exactly
that
with
the
cube
API.
This
has
a
huge
benefit
that
now
you
can
actually
offload
a
lot
of
the
dependencies
from
the
airflow
workers.
So
you
no
longer
have
to
have
a
lot
of
these
libraries
in
the
worker.
You
can
just
deploy
a
docker
container
that
has
those
libraries
to
run
that
task.
A
It
also
has
a
huge
benefit
where
you
can
now
do
very
easy:
roll
backs
and
deployments,
because
you
can
tag
the
images
in
your
decks.
So
now
not
only
are
the
DAGs
item
potent
but
the
actual
tasks
that
they're
running
your
item,
potent
because
it's
running
within
a
docker
container
for
the
issue
of
static
versus
dynamic
allocation,
we
created,
what's
called
the
kubernetes
executor.
A
Pod
configuration
so
the
way
that
the
dynamic
allocation
and
the
in
the
kubernetes
executors
works
is
that
you
only
have
to
actually
release
a
single
pod
containing
the
UI
in
the
scheduler
and
when
you
actually
set
one
of
your
DAGs
to
run
that
scheduler
speaks
to
the
Kubb
api
and
is
able
to
dynamically,
create
create
airflow
pods,
each
of
which
containing
a
single
task
to
run
so
once
it
runs
that
task.
The
the
worker
pod
terminates
and
then
really
releases
those
sort
resources
to
the
cluster.
This
means
at
times
of
high
traffic.
A
You
can
scale
it
out
as
much
as
needed
and
then
once
they're
all
completed
they're
able
to
completely
disappear,
and
you
only
have
to
maintain
a
single
pot.
This
had
the
side
effect
of
being
able
to
allow
us
to
do
task
level
configs.
So
before
with
static
workers,
you
couldn't
really
make
that
many
changes
because
they
were
already
allocated.
But
now
you
can
actually
say
you
know,
I
think
this
task
deserves
this
much
memory.
I
actually
want
to
do
a
sci-fi
task
with
this
with
this
task.
A
A
You
would
actually
have
to
have
the
scheduler
checking
the
status
of
each
of
the
tasks
to
see
if
they
passed
or
failed
or
still
running
so
now,
actually,
because
the
workers
are
speaking
to
the
same
Postgres
back
end
as
the
scheduler.
The
only
thing
that
the
scheduler
needs
to
know
about
is
did
this.
Did
this
pod
fail
and
the
way
it
does
that
is
through
the
kubernetes
watcher
API,
where
the
kubernetes
wash
api
allows
the
allows
the
scheduler
to
read
an
event
log
for
anything
with
a
label
tied
to
that
airflow
instance?
A
A
But
we
also
wanted
to
consider
fault
tolerance
for
when
the
scheduler
goes
down
with
previous
air
flow
configurations,
there
was
a
there
was
a
problem
where
a
lot
of
state
was
being
held
inside
of
local
Python
queues,
and
that
meant
that
if
the
scheduler
went
down
there
was
no
easy
way
to
recreate
this
state.
So
you
might
have
tasks
that
run
twice.
A
You
might
have
passed
and
don't
run
at
all,
and
so
we
were
able
to
take
advantage
of
what's
called
a
resource
version
and
kubernetes,
and
we
by
storing
the
resource
version
in
the
sequel
back-end
whence
when
a
scheduler
goes
down,
it
can
check
the
Postgres
check
the
resource
version
reread
the
event
log
from
that
point
and
then
be
up
to
date.
This
means
that
the
scheduler
can
go
down
for
hours
and
even
up
to
days
and
still
be
able
to
come
back
to
a
point
of
up
to
date.
A
One
other
issue
we
had
to
consider
was
dag
propagation.
You
start
out
with
the
Dagon
schedulers,
but
with
these
workers
coming
up
dynamically,
how
do
we
get
all
these
dag
files
to
the
workers
to
actually
run
your
workloads?
We
have
two
modes
currently
with
a
third
on
the
way
the
two
existing
ones.
Are
they
getting
it
mode
in
the
persistent
volume
mode
and
with
one
point,
10.2
we're
going
to
have
a
pre-baked
mode
where
you
can
actually
pre-baked
your
DAGs
into
the
docker
versions?
A
We
recommend
the
get
in
pre-baked
mode
for
small
to
medium
sized
air
flow
instances,
because
it's
the
least
amount
of
configuration
the
pre-baked
will
be
the
easiest
one.
Because
then,
as
long
as
you
know,
the
version
of
all
the
dogs
are
running.
Everything
will
be
consistent
and
there's
no
need
to
speak
to
any
external
services
at
all,
but
we
do
also
understand
that
certain
companies
have
pretty
large
dag
file
systems
and
to
handle
those
cases.
A
B
Thank
You
Danielle-
that
was
an
amazing
piece
of
work.
The
community's
executors
effectively
modernized
our
flow
and
took
it
to
the
container
world,
but
it
still
didn't
solve
the
problem
of
managing
and
deploying
our
flow.
That's
where
the
operator
comes
in
the
operator
is
used
to
is
a
custom
controller
on
kubernetes
that
extends
the
API
surface
of
kubernetes.
By
introducing
two
new
CR
DS,
that's
the
airflow
base
and
the
airflow
cluster.
Let's
take
a
look
at
the
base.
The
airflow
base.
B
B
In
this
example
of
the
spec,
the
user
requests
for
my
sequel
instance
and
a
storage
he
uses
cubes
ETL
to
post
it
to
the
API
server.
The
airflow
controller
monitors
the
CR
DS
and
then
creates
a
stateful
sets.
Pod
Services,
config
map
secrets,
etc
needed
for
deploying
this
company
and
in
let's
take
a
look
at
another
example.
In
this
example,
the
user
requests
for
sequel
proxy.
Instead
of
my
sequel.
This
is
used
to
connect
to
cloud
sequel
in
a
cloud
environment.
Cloud
sequel
is
a
more
production,
ready
database
and
when
he
applies
the
spec.
B
Let's
take
a
look
at
the
cluster
theory
cluster
series,
where
you
describe
the
actual
airflow
components
in
this
particular
example:
the
user
requests
for
us,
salary
executors,
thereby
needing
the
radius
as
the
broker,
and
he
also
under
request
for
two
workers
and
when
the
controller
deploys
this
CRD,
it
results
in
all
these
components
getting
deployed
the
way
the
controller
works
with
cluster
is
it
for
each
of
the
clusters.
Uri
creates
a
unique
username,
password
and
database
in
the
sequel,
DB,
and
thereby
this
allows
you
to
share
the
same
base
across
multiple
clusters
with
isolation.
B
In
this
example
of
the
cluster
CR,
you,
the
user,
requests
for
communities
executors.
If
you
notice,
we
don't
the
broker
and
the
worker
nodes
are
not
deployed
for
cumulative
executors
and
for
what
we
recommend
is
well
demoed.
We
use
the
local
executor
and
for
production
runs
use
the
kubernetes
executor.
B
B
With
the
operator
we
use
a
sidecar
or
any
container
pattern
to
inject
the
tags.
So
when
the
airflow
pod
is
started,
a
sidecar
syncs,
the
tags
from
a
the
GCS
get
or
NFS
or
any
of
the
sources,
that's
configurable
and
then
starts
the
airflow
part,
like
the
sorry
airflow
container,
like
the
scheduler
of
worker.
A
That's
already
been
set
up
in
your
cluster,
so
you
can
piggyback
on
things
like
if
your,
if
your
team
has
already
set
up
Prometheus
or
elasticsearch
for
logs
a
lot
of
these
will
automatically
transfer
from
air
flow
to
these
other
sources
to
easily
track
and
monitor
what
the
health
of
your
cluster
we
and
as
far
as
the
status
goes,
the
kubernetes
executor
has
been
released
in
airflow
1.10
in
experimental
mode.
We
have
multiple
companies
already
using
it
in
production,
a
home
chart
in
progress.
A
A
B
B
A
B
B
B
B
B
B
B
B
All
right,
if
you
notice
it,
creates
a
pod
dynamically,
that's
what
the
cabinet
is
executed
just
at
four
for
a
dag
task.
It
created
a
pod
dynamically
and
once
the
task
completes,
the
part
goes
away.
That's
that's
it
and
to
show
that
how
the
sequel
sharing
a
sum
I'm
just
going
to
show
a
bit
look
at
the
connection
string
here
for
my
sequel.
It
generates
dynamically,
the
username
password
and
database,
which
are
different
for
each
of
the
clusters.