►
From YouTube: Nicholas Schrock: Dagster - An open source Python library for building data applications at Crunch
Description
👉 Learn more about the talk and download the slides at https://crunchconf.com/speaker/NicholasSchrock#talks
📬 Sign up to our newsletter so you won't miss the updates about the next Crunch Data Conference: http://eepurl.com/dGwi1f
We introduce Dagster, an open source Python library for building ETL processes, ML pipelines, and similar software systems, all of which we call data applications.
Data applications are graphs of functional computations that consume and produce data assets. Dagster provides abstractions and tools for modeling the semantics of these applications by providing a unified type system, a data dependency graph, a configuration system, a structured API for emitting events such as data quality tests and materializations, and high-quality developer tools built on those abstractions. Builders can use the tools they know -- e.g. Spark jobs for data engineers, SQL statements for analysts, Python for data scientists -- and the application can be deployed to arbitrary orchestration engines -- such as Airflow, Dask, or Kubernetes-based execution -- in a pluggable fashion. - Captured Live on Ustream at https://www.ustream.tv/channel/JUMjvCF2ucj
A
Thank
you
this
thing
on
there
we
go
well
thanks
for
all
thanks
everyone
for
coming.
As
she
mentioned,
my
name
is
Nix
rock
and
the
founder
of
go
feed
back
here.
I'm,
the
founder
of
elemental
and
kind
of
our
purpose
is
a
company
right
now
is
to
host
this
open
source
project,
called
dag,
stir
and
gonna
talk
about
what
that
is,
basically
show
it
to
you
through
live
demos.
A
So,
let's
hope
that
works
and
also
talk
about
it
in
relation
to
other
technologies
in
the
space,
in
particular
air
flow,
because
that
comes
up
over
and
over
and
over
again,
so
we're
gonna
talk
about
what
its
relation
is
to
that
technology.
So
what
is
this
thing?
We
describe
it
as
a
Python
library
for
building
data
applications,
it's
an
open
source
project,
but
you
might
be
asking
that's
very
understandable.
What
on
God's
earth
is
a
data
application?
A
Why
did
you
make
up
a
new
word,
so
you
know
there's
all
this
existing
terminology
in
this
industry,
yeah,
there's
ETL,
which
is
extract,
transform
load.
Then
there's
the
dramatic
reversal
of
ELT.
This
whole
new
thing,
where
you
swap
the
team.
A
oh
and
there's
ml
pipelines,
and
you
know
we
really
think
that
a
lot
of
this
terminology
doesn't
really
make
sense
anymore.
So,
let's
talk
about
ETL
in
particular.
A
Right
ETL
is
historical
term,
and
it
comes
from
way
back
in
the
day
in
the
70s,
where
you
had
one
database
over
here
and
one
analytical
store
over
here
and
you
had
a
single
T
in
the
middle
and
you
extracted
from
the
transactional
database.
Did
one
transformation
process
usually
once
per
week,
say
and
then
loaded
it
into
the
data
warehouse
right?
A
You
typically
did
this
in
the
sort
of
like
drag-and-drop,
vertically
integrated
tool
and,
what's
weird,
is
that
people
still
use
this
term
ETL,
but
it
has
no
relation
to
what
it
previously
used
to
mean
right.
Etl
now
means
effectively
any
computation
which
moves
data
from
one
place
to
another
and
in
a
modern
context,
that
means
there's
generally
multiple
computational
runtimes
involve
multiple
stages
of
materialization
many
many
data
sources.
Some
of
them
are
transactional
databases.
A
Sometimes
my
web
scraping
some
might
be
ingest
from
a
SAS
app,
it's
just
an
incredibly
complicated
world,
and
this
stuff
is
also
generally
more
critical
to
the
functioning
of
your
generalize
application.
All
right,
then,
let's
talk
about
ml
pipelines,
so
you
hear
this
all
the
time
right.
A
data
scientists
will
say,
like
I,
spend
90%
of
my
time
doing:
data
cleaning
and
10%
of
my
time
doing
my
job.
What
they're,
really
saying
in
some
ways
is
that
the
bulk
of
every
ml
pipeline
is
an
ETL
process
right.
A
So
this
is
most
frequently
materialized
in
the
real
world
as
I'm
a
data,
scientist
and
I
got
a
task
and
I
was
given,
like
my
boss,
gave
me
a
CSV
file
and
they're
like
do
ml,
make
us
money
please,
and
then
you
go
off
and
you
work
inside
that
Jupiter
notebook
and
you
build
some
graph
and
you
can
show
it
to
a
decision
maker.
Be
like
look,
you
can
do
this
and
then
it's
handed
off
to
the
data
engineers
whose
responsibility
is
to
brush
on
eyes
it.
A
They
often
build
it
in
a
completely
different
technology
stack
and
this
silo
leme
and
this
relationship
between
these
personas
is
actually
very
unhealthy
and
what
you
really
want
is
more
of
a
world
that
looks
like
this,
where
data
scientists
have
skills
and
capabilities
of
data.
Engineers
don't
have,
and
vice-versa
is
also
a
true,
but
you
have
more
overlap,
so
they
can
kind
of
exist
within
the
same
framework
right
and
the
only
way.
A
But
I
know
how
to
make
this
happen
is
to
create
new,
tooling
and
new
abstractions
in
order
to
make
that
relationship
more
harmonious,
so
data
application
ml
is
an
instance
of
it
ETL
ELT
and
what
we
define.
These
things
ask
is
just
both
of
those
types
of
classes
of
systems
or
all
of
those.
They
are
all
graphs
of
functions
or
functional
computations,
as
it
says
up
there
that
produce
and
consume
data
assets
right
the
last
step
of
an
ETL
process,
typically
a
file
or
partition
in
a
data
warehouse.
A
Something
like
that
and
the
last
step
of
an
ml
process
is
a
model
right.
Everything
else
about
it
very,
very
similar,
and
the
thesis
here
is
that
the
current
tools
that
we
use
do
not
match
the
complexity
of
the
problems
that
we
face
right
and
you
kind
of
like
feel
this
instinctually
when
you're
scratching
either
one
of
these
ways-
data
infrastructure
projects,
it's
just
always
ends
up.
Is
this
disorganized
mess?
That's
we're
testing
isn't
really
the
way
it
should
be.
A
I
mean
everyone
knows
what
I'm
talking
about
here
and
you
can
see
it
in
the
end
product
from
the
data
teams,
and
so
in
particular,
I'm
gonna
talk
about
air
flow
and
why
air
flow
alone
does
not
save
us
here,
so
I
guess
before
I
go
into
this
whole
shtick.
How
many
people
have
heard
of
air
flow?
Okay,
okay,
great?
How
many
people
have
used
it?
This
is
amazing,
there's
actually
I
was
making
a
slide
deck
last
week
and
I
even
think
this
morning,
I
was
like
Harding.
A
A
So
that's
12
years
ago
now
it's
actually
a
fairly
old
system,
and
it's
very
it's
very
similar
to
that
system
that
was
created
in
that
time
period
of
Facebook,
and
you
know
the
thinking
in
the
industry
has
evolved
since
then,
and
no
less
than
max
the
inventor
max
Bo
Sherman's,
the
inventor
of
airflow
or
I'd
like
to
call
him
mr.
data
engineer.
A
2012-2015
he's
now
moved
on
to
greener
pastures.
He
started
with
startup
based
on
a
technology
called
super
sense,
so
he's
no
longer
that
directly
involved
in
air
flow,
but
he
wrote
this
really
interesting
article
a
couple
like
a
year
ago
about
how
we
need
to
bring
functional
programming
practices
to
the
data
engineering
ecosystem,
and
you
know
I
think
that's
totally
right.
So
he
talked
about
all
sorts
of
interesting
things
in
that
article.
He
talked
about
how
functional
programming
leads
to
better
reproducibility,
that
by
thinking
as
these
partitions,
that
are
immutable
data
structures.
A
You
can
kind
of
like
model
this
stuff
better.
You
model
everything
with
idempotent
computations,
which
means
that
you
can
kind
of
instigate
the
computation
again
and
be
confident,
that's
not
going
to
destroy
any
information
and
that
it
makes
it
easier
to
manage
the
concept
of
time
in
these
systems
which
I'm
not
gonna,
get
into
that.
But
it's
a
very
complicated,
and
so
you
know
I
loved
everything
max
is
saying
here
and
he
was
the
inventor
of
airflow,
but
then
I
got
me.
Thinking
is
airflow
really
functional.
A
Is
it
really
and
I
would
argue
that
it's
actually
not?
You
can
use
it
in
the
service
of
implementing
this
functional
and
functional
data
engine
data
engineering,
but
it
does
not
naturally
guide
you
to
do
it.
I
call
this
falling
into
the
pit
of
success
right.
You
can
use
airflow
to
do
this,
but
it's
not
the
natural
thing
to
do
so.
There's
a
few
core
reasons
for
this,
so
DAGs
and
tasks
an
airphone
will
demonstrate
this
are
not
parameterised.
A
They
are
functions
which
have
effectively
no
parameters
right,
that's
doesn't
that
makes
it
not
a
function
pretty
much.
They
also
don't
encapsulate
data
flow
in
any
sort
of
way.
If
you
look
at
an
air
flow
task
graph,
there's
no
way
to
know
what
data
is
actually
flowing
through
that
system.
It's
completely
the
tool
is
completely
blind
to
what
the
actual
nodes
of
computation
are
doing
and
then
there's
no
metadata
about
that.
A
There's,
no
type
information
and
so
on
and
so
forth,
and
another
issue
here
is
that
the
system
infrastructure
is
not
pluggable,
which
means
that
it's
actually
very
difficult
to
execute
your
flow
tags
and
tasks
in
isolation.
I
mean
that
they
always
have
to
touch
database.
You
have
to
local,
install
on
your
machine
running
it's
difficult
to
spin,
that
up,
ephemerally,
etc,
etc.
It
makes
it
difficult
to
test,
and
so
because
of
these
limitations.
There's
actually
some
consequences
here.
A
One
is
that
air
flow
tags
and
tasks
are
effectively
untestable
right,
as
I
said,
it'll
go
over
in
a
second
they're,
basically
functions
about
the
parameters,
and
if
your
function
doesn't
matter
parameter,
you
cannot
test
it
right.
You
can
only
invoke
it
and
you'll
only
ever
get
one
result
back.
So
it's
like
it's
like
designs,
not
to
be
testable,
which
is
kind
of
concerning,
and
then
because
of
this.
It's
really
only
a
tool.
A
A
You
kind
of
the
data
inputting,
that's
input,
changes
right
and,
whereas,
with
data
engineering,
the
only
parameter
that
matters
is
time,
I
mean
like
you're,
processing,
different
partition,
everyday
and
that
time
is
the
only
parameter
that
airflow
supports
natively,
and
the
other
thing
is
that
airflow
is
not
suited
for
the
full
developer
life
cycle,
meaning
that
you
don't
use
it
when
you're
doing
local
development.
Typically,
you
don't
test
within
that
system.
Only
once
you've
crafted
your
computation
and
you
want
to
embed
it
in
a
dag
in
production.
A
Does
air
flow
come
in
the
picture
and
we
think
there
should
be
a
system
that
does
the
full
development
life
cycle?
So
that's
this
slide.
You
want
a
framework
that
supports
the
full
lifecycle
of
day
applications.
That
means,
when
you
develop
it,
you
want
be
able
to
test
it
locally
test
it
in
a
CSV
pipeline.
Do
integration
testing
then
productionize?
A
It
then
schedule
it
right
and
you
want
to
have
one
system
that
can
kind
of
capture
that
whole
process
and
so
I'm,
going
to
demonstrate
kind
of
through
demos
see
if
this
works,
how
dax-ur
and
also
of
the
injuries
and
some
of
the
concepts
along
the
way
how
daxter
addresses
each
of
these
steps.
So
let's
go
to
develop.
A
So
this
is
kind
of
hello
world
for
gangster
here,
so
you
pip,
install
daxter,
it's
just
a
Python
program
and
the
core
abstraction
here
is:
we
call
it
a
solid
okay
and
the
solid
is
a
node
of
computation
that
consumes
and
produces
a
data
asset
right.
The
body
of
the
function
can
be
anything
meaning
that
you
can
use
any
computational
framework,
whether
that's
pandas,
PI
spark
or
you
can
even
remote
out
and
execute
a
date
command
in
a
data
warehouse
or
invoke
a
spark
scholar,
job
whatever,
and
you
define
inputs
and
outputs.
A
So
here's
like
the
first
thing
reads:
yes,
V
and
then
this
is
like
classic
computer
science,
blinking
lights
demo
like
doing
like
a
ad
operation,
but
this
takes
in
a
dataframe,
produces
another
data
frame,
and
then
you
string
these
together,
and
this
is
how
you
construct
that
dag.
So
we
have
this
function
like
DSL
that
actually
like
assembles
these
things
together
and
creates
a
dag
so
the
moment
that
you
have
your
code
in
that
format.
You
have
tools
at
your
disposal
so
hop
here.
A
This
is
just
a
web
app
running
locally
on
my
machine
and
it's
loaded
up
this
code,
and
now
we
have
all
this
tooling
here,
so
we
load
up
this
dag
yeah.
You
have
descriptions
of
what's
going
on,
so
you
can
like
view
this
and
actually
see
and
have
context
about.
What's
going
on
right
now,
let's
say
I
want
to
execute
this
thing.
A
Now
these
types
here,
one
of
the
things
we
have
to
do,
we
actually
have
to
parameterize
this
thing.
So
this
is
one
of
those
important
concepts
in
this
system.
Is
that,
unlike
airflow,
where
you
kind
of
the
dag
takes
in
the
config,
and
then
you
construct
the
dag
out
of
that,
this
separates
the
dag
from
its
parameters.
So
whenever
we
have
a
dag
that
we
want
to
execute,
we
have
to
have
the
parameters
in
order
to
execute
it.
So
in
this
case
right
we
need
to
give
it
the
path.
A
A
So
as
you
execute
this
thing,
you
get
a
live
log
of
everything,
that's
going
on.
It's
live
and
reactive,
even
gonna
get
a
sense
of
these
computation.
So,
for
example,
every
single
step
that
happens.
It
does
a
type
check
and,
as
part
of
that
type
check,
it
also
renders
metadata
for
you
in
this
structured
format,
and
so
this
is
all
very
nice.
Let's
see
what
else?
Oh,
so
this
is
a
now
like
kind
of
our
local
development
environment
right.
A
So
one
thing
I'm
going
to
do
is
I'm
gonna
set
this
so
that
one
of
the
things
our
type
system
allows
us
to
do.
Is
that
as
the
dag
executes,
it
will
actually
persist
all
the
intermediates
that
get
generated
along
the
way
and
this
unlocks
a
lot
of
power.
So
I'm
gonna
execute
this
and
now,
let's
imagine
do
go
here.
Oh
now,
it's
a
mention.
I
injected
an
error
here.
Let's
say
all
right:
I
screwed
something
up.
A
Okay,
now
I
execute
this
thing
and
now
there's
gonna
do
an
error
right,
so
I
can
click
here.
I
can
get
a
stack
trace
figure
out
what's
going
on,
but
what's
cool
about?
This
is
that
you
know
this
is
a
really
fast
trivial,
dag
so
takes
doesn't
take
much
time
to
execute
I
imagine
this
is
like
a
50
node
dag
and
it
was
like
a
whole
bunch
of
work
and
you're
just
about
to
go
home
for
the
day
and
then
the
50th
task
fails.
A
You
want
to
kill
yourself
because
you
have
to
fix
one
little
bug
and
then
re
execute
the
whole
thing
which
totally
sucks
right
here.
Instead,
you
can
just
re,
execute
this
single
step
over
and
over
again,
and
it
uses
the
intermediates
from
the
previous
run
in
order
to
deal
with
that.
So
we
run
this
thing
by
itself.
We
still
have
the
same
error
go
here.
A
Yeah
re
execute
this
single
step.
Now
it
works
again
right.
So
in
a
really
complicated,
dag
scenario.
You
can
see
how
useful
is
to
be
over
and
over,
and
this
is
actually
how
we
actually
get
incremental
computational
semantics
happening,
server-side
as
well
other
fun
things
with
this
system.
You
can
execute
sub
decks
right,
so
you
can
actually
go
here
and
say:
hey
I,
don't
want
to
read
I,
don't
want
to
execute
that
read
CSV
command
that
you
coded
by
hand
right,
but
now
you
have
to
provide
the
input.
So
now
we're
we
start
execution
here.
A
A
A
So,
let's
talk
about
testing,
so
one
of
the
nice
things
about
this,
so
this
contract
that
you
see
where
it
says
like
hey:
this
is
a
node.
It
has
these
inputs,
it
has
these
outputs,
etc,
etc.
What
that
says
is
that
it
is
a
contract
that
tells
you
what
the
state
of
the
world
needs
to
be
before
the
execution
proceeds
for
this
thing
to
work,
meaning
that
here
what
it
says
is
like
hey:
there
needs
to
be
pandas,
dataframe,
that's
somewhere
right,
and
if
that
thing
exists,
my
computation
won't
work.
A
Translate
that
to
any
other
task
management
system
that
doesn't
have
this
type
of
metadata
system.
You
have
no
idea
what
the
state
of
the
world
needs
to
be
for
that
task
to
execute
right.
If
there's
just
a
task
called
foo
in
the
middle
of
like
say
an
airflow
tag,
you
have
no
idea
what
the
state
of
the
world
needs
to
be
in
order
to
for
food
to
execute
properly
right,
having
this
contract
is
enormous,
ly
powerful,
and
what
that
allows
you
to
do
to
go
back
to
the
presentation
here.
A
So
now
we
move
on
to
test
and
I'm
gonna
dig
into
this
that
much,
but
we
provide
a
bunch
of
nice
kind
of
in
memory
api's,
so
you
can
just
within
our
run
time,
execute
a
solid
on
zone
or
execute
a
pipeline
on
its
own
right
and
what's
nice
about
this.
Is
this
plugs
in
and
we'll
go
into
this
a
second
yeah,
the
the
system
kind
of
mocks
up
all
the
internal
infrastructure
needed
to
run
this
thing.
A
So
this
these
runs
have
no
dependencies
on
any
external
runs
database
or
anything
like
that
right,
but
it's
still
executing
the
context
of
a
runtime.
So
the
point
of
this
not
so
you
understand,
like
this
code
sample.
The
point
is
that
we
have
designed
and
published
api's
that
are
specifically
designed
for
you
to
run
testing
in
your
local
environment,
testing,
your
CSUB
pipeline,
etc,
etcetera,
etcetera.
So
now,
let's
talk
about
production,
ization,
so
you've
written
your
pipeline.
A
You
have
a
dag
you're
thinking
about
your
dag
from
day
one
as
you're
developing
your
app,
which
is
good
my
opinion,
but
now
I
want
to
deploy
that
ok.
Well,
we
actually
come
out
of
the
box
what
kind
of
and
some
automatic
deployment
to
AWS.
So
if
you
run
some
command
right,
you
can
go
in
it'll
spin
up
a
node
for
you.
Security
groups
key
pairs
all
that
stuff,
but
the
mandate
to
the
team
was
don't
make
people
touch
the
Atos
console
because
who
wants
to
interact
with
that
thing?
A
So
now
we
want
to
be
able
to
schedule
these
things
right.
You
write
a
pipeline.
You've
tested
it
to
your
satisfaction,
you've
deployed
it
now.
You
want
to
schedule
it,
and
so
we
come
with
a
native
scheduler,
which
is
very
simple
right.
All
it
is,
is
a
Python
wrapper
over
a
cron
turns
out.
Cron
works,
cron
works
very
well,
it's
worked
well
for
30
40
years
and
we
have
a
way
where
you
can
say
like
hey.
Here's.
This
function,
which
contain
some
a
Python
data
structure,
would
represent
your
pipeline
on
a
schedule.
A
A
B
A
Is
like
a
live,
updating,
UI,
you
know
if
I
wait,
one
minute
for
that
next
schedule
to
run
you'll,
see
it
magically
appear
here,
which
is
fun
and
you
can
search
for
all
the
schedules
which
came
from
a
certain
one
like
all
this
stuff
right.
So
you
can,
like
you,
have
the
Box
like
a
very
simple
scheduler,
but
all
this
is
structured
on
api's
too.
So,
if
you
have
your
own
scheduler,
it's
very
easy
to
invoke
new
runs
based
on
that.
A
So
you
know
between
these
tools
that
I've
shown
you.
You
can
go
all
the
way
from
local
development
deployment,
to
scheduling,
doing
both
local
development
and
monitoring
all
with
the
same
same
tool
set.
It's
actually
quite
powerful.
So
I
want
to
dig
into
testability
here,
because
I
think
it's
an
interesting
lens
to
kind
of
explain
a
bunch
of
the
principles
of
the
system.
A
So
when
you
think
of
the
status
quo
today
in
data
systems
and
you
divide,
testing
and
to
say
unit
test,
which
is
like
you
can
run
tests
on
your
local
machine
without
touching
externalized
resources
to
integration
test
which
might
be
in
a
csv
pipeline.
But
you
might
have
to
spin
up
a
docker
container
or
something
to
a
staging
environment
if
you've
done
a
lot
of
work,
all
right
which
basically
a
replication
of
your
production
environment,
maybe
on
some
sample
data
and
then
your
full
production
environment.
A
If
you
talk
about
most
data
systems
today
who
build
them
but
kind
of
the
percentage
of
testing
done
at
stage,
looks
like
that
like
in
reality,
a
lot
of
most
testing,
if
the
eve
is
even
done
at
all,
is
done
at
production
and
there's
good
reasons
for
this,
and
the
primary
reason
for
it
is
that
testing
these
systems
is
really
hard.
They
have
very
specific
constraints
that
make
them
difficult
to
test,
namely
that
you
don't
control
your
input,
datasets
right.
A
So
it
takes
a
lot
of
work
to
simulate
those
and
to
most
your
business
logic
is
often
hosted
in
computational
runtimes
that
are
impossible
to
mock
locally
right.
You
cannot
like
mock,
spark
in
process
in
Python.
Doing
that
would
be
building
an
entire
company
unto
itself.
It's
like
not
enough
work.
It's
not
not
a
too
much
cost
for
the
benefit
you
get,
and
so
you
know,
I
think
testing
and
the
data
systems
can
get
a
bit
of
a
bad
name
because
possibility
people
in
software
are
very
opinionated
not
often
very
zealous.
A
They
start
giving
you
speeches
about
box
and
stubs
now,
you're,
not
doing
anything
right,
I,
don't
know
if
you've
ever
experienced
this.
This
has
been
my
experience.
We
go
away
and
I,
don't
think
they
really
appreciate
kind
of
what
the
problems
are
on
the
data
domain,
that
I
was
talking
about,
and
so
when
it
comes
to
testing.
This
is
a
couple
principles.
A
Is
that
one
is
that
things
that
can
be
caught
by
unit
tests
should
be
caught
by
unit
tests
right
so
I'm
talking
about
things
like
fixing
parsing
a
configuration
file
or
figuring
out
in
Python
that
you're
calling
a
method
that
doesn't
exist
or
things
of
that
nature
right.
These
are
things
that
should
be
caught
in
unit
tests
in
in
the
current
day,
data
systems
they're
typically
not
right.
A
You
don't
really
catch
that
stuff
until
you're
in
production,
but
there's
an
important
corollary
which
I
just
mentioned,
which
is
like
don't
attempt
to
catch
things
earlier
in
the
process.
Then
you
can
write
like
a
date
equality
test.
You
have
to
run
them
in
Prague
cuz
like
that's,
where
the
only
place
where
really
makes
sense
to
run
them
or
you
have
to
in
order
for
it
to
make
sense.
You
shouldn't
try
to
unit
tests
parked
in
too
much
detail,
it's
impossible
to
mock.
A
So
I.
Think.
If
you
have
this
in
mind
what
you
can
frame,
the
goal
is
not
to
like
say
that
Oh
everything
should
be
unit
testing
and
everything
should
be
caught
in
the
CSV
pipeline.
It's
not
realistic,
but
what
you
can
do
is
kind
of
shift
the
gravity,
so
that's
earlier
in
the
process
and
that
shift
in
the
graph
might
not
look
like
a
big
deal.
A
But
if
you
actually
succeed
in
moving
most
of
your
testing
to
unit
and
integration
testing
time,
that
is
like
an
order
of
magnitude
improvement
in
developer
experience,
because,
as
the
farther
you
push
testing
on
the
line,
it
becomes
more
costly
in
terms
of
the
developer
or
feedback
loop
to
catch
it
and
I
think
this
is
like
you
know.
The
other
thing
I
noticed
in
these
data
systems
that
frequently
the
code
quality
is
at
a
lower
bar,
and
this
there's
like
systemic
degradation.
It's
not
because
the
engineers
who
are
interacting
those
systems
are
worse.
A
You
take
the
same
engineer
you
put
in
one
of
these
data
systems
and
the
code.
Quality
isn't
as
good.
It's,
because
the
act
of
doing
like
a
refactoring
task
of
something
is
way
more
risky
because
you
don't
catch
the
error
until
you're
later
in
the
process.
So
it's
important
to
try
to
front-load
this
as
much
as
possible,
so
airflow
does
not
addressed,
develop
and
test
right.
A
Typically,
when
you
use
the
airflow
right
yours
working
in
your
own
little
universe
in
your
job
and
then
very
late
in
the
development
lifecycle,
you
plop
that
job
into
the
airflow
dag
construction
zone,
and
then
you
just
kind
of
ship
it
off
to
production
and
hope
and
pray
right.
That's
typically
the
workflow
and
it's
oh
really.
Airflow
only
addresses
the
production
eyes
and
the
schedule
part
and
therefore
doesn't
view
the
dag
really
is
a
first-class
programmatic
concept.
I
think
is
the
problem
here.
A
So
let's
just
dig
into
this
right:
airflow
tasks
are
what's
a
little
confusing.
Actually
the
airflow
documentation
hops
between
operator
task
and
task
instance.
It's
often
kind
of
hard
to
know
what
they're
actually
talking
about,
but
operators
when
you
create
one
that
actually
creates
a
task,
and
so
here,
if
you
look,
the
task
itself,
doesn't
have
any
user
defined
parameters
right
so
like
this
load
to
the
HDFS
instance
right
there.
Doesn't
it's
not
a
it's
a
function
with
no
parameters
effectively
and
that
task
instance
is
bound
to
an
environment.
A
A
And
so
it's
bound
to
an
environment
and
that
task
creates
a
dag
and
because
the
tasks
themselves
aren't
parameterised.
The
dag
is
not
parameterised
right.
So
these
things
have
no
parameters,
so
you
might
be
thinking
you're,
an
airflow
expert
like
wow,
there's
all
this
other
stuff,
so
yeah
there's
no
parameters,
but
you
can
kind
of
model
data
flow
through
XCOM.
But
it's
really
odd
right
because
air
flows
documentation
says
like
wow.
You
should
never
share
data
between
the
tasks.
A
First
of
all,
it's
a
little
wrong
because
all
you're
doing
is
not
expressing
some
dependency
that
exists
right
and
in
order
for
there
to
be
some
things
dependent
on
the
other
thing,
something
had
to
happen
first,
which
means
that
this
had
to
affect
some
sort
of
externalized
state
somewhere.
It
makes
sense
that
it
should
pass
data
to
the
next
task
about
what
the
hell
it
did
right
and
so
there's
a
feature
called
XCOM,
but
no
one
in
airflow
really
says
they
should
do
it.
It's
just
like.
Oh
absolutely
can't
be
avoided.
A
Don't
use
XCOM
right,
they're,
basically
saying
don't
use
this
feature
and
in
fact
the
creator
of
XCOM
kind
of
has
sworn
it
off.
So,
okay,
we
have
this
thing,
but
don't
use
it,
and
so,
in
the
end,
what
you
have
again
is
a
dag
of
parameters.
Functions,
there's
a
common
theme
in
airflow
land
about.
Like
don't
use
me,
you
know
we
have
this
real.
A
We
have
this
real
construct
for
a
composition
called
the
sub
dag,
but
every
airflow
practitioner
I
know
says
that
don't
use
that
thing
so
there's
like
there's
been
these
stabs
and
airflow
for
hire
lab
it'll
programmatic
constructs,
but
it's
always
kind
of
been
a
swing
and
a
Miss
and
which
is
interesting.
So
another
thing
that
I
mentioned,
which
I'll
go
through
quickly,
is
that
dag
stirrer
has
this
pluggable
system,
environment
so
airflow,
for
example,
they
store
runs
in
the
database
right.
A
So
all
this
testing
stuff,
the
kind
of
theme
is-
is
that
we're
not
claiming
this
is
automatic
right,
but
what
we
have
done
is
we
very
deliberately
built
the
system
from
first
principles,
so
that
testing
in
a
cohesive
way
is
very
possible
and
it's
gonna
be
up
to
likely
an
entire
ecosystem
of
people
to
kind
of
build
tools
and
help
stuff
out.
But
this
is
a
very
stable
foundation
with
assumptions
that
allow
for
this
testability,
so
stepping
back
for
a
second.
A
What
the
core
feature
that
allows
the
testability
to
happen
is
that
we
think
a
lot
about
parameterizing
these
computations
and
yes,
that
helps
testability
a
lot.
But
it's
also
like
the
core
difference
between
it
and
other
workflow
engines,
and
it's
gonna
be
that
property
which
enables
it
to
use
for
other
use
cases
as
well,
so
parameterization
actually
enables
data
science
right.
It's
really
really
critical
for
data
science
workflows
for
these
computations.
A
We
parameterised,
because
in
order
to
build
a
data
science
ml
pipeline,
you
want
to
be
able
to
run
it
50
times
with
different
parameters
in
order
to
say
do
hyper
parameter.
You
want
to
have
it
do
that
on
the
same
instance
of
code
in
a
verifiable
way,
so
you
have
reproducibility
and
whatnot.
So
parameterization
is
incredibly
incredibly
important
here,
metadata
I'm,
actually
going
to
skip
to
my
demo.
A
So
let's
go
to
a
more
complicated
dag
here,
so
this
is
a
more
real,
dag.
Okay-
and
this
is
a
couple
things
I
want
to
point
out
here.
So
one
is
that
there's
tons
of
metadata
that
was
very
richly
expressed
here
right,
there's
inputs,
there's,
outputs,
there's
user-defined
descriptions
right
and
each
one
of
these
is
a
function
right
and
the
function
says
that
hey
these
inputs
need
to
exist
in
order
for
my
computation
to
proceed
and
then
it
produces
as
output.
A
And
then
you
do
dependencies
based
on
those
that
dataflow
right
well,
because
these
are
functions
right.
What's
one
of
the
best
things
about
functions
well,
functions
compose
very
useful
property
of
functions,
so
just
as
functions
can
call
other
functions.
Solids
can
call
other
solids.
So
if
you
go
in
here,
so
you
click
on
this
mm-hmm
there
we
go.
That
is
actually
a
dag
of
solids
which
exists
within
another
solid
right,
and
this
arbitrarily
can
nest.
You
could
go
over
these
ones.
This
one
goes
down
and
here's
another
reusable
thing,
and
now
here's
this
like
reusable.
A
You
know
this
is
like
nested
and
it
actually
makes
it.
So
you
can
reuse
this
thing
all
over
the
place
so
like,
for
example,
this
one
solid,
which
is
cash,
a
file
from
s3
in
some
local
cash
that
is
used
like
eight
times
throughout
this
pipeline
in
different
parts
of
the
stack
right.
This
is
like
a
level
of
reusability
and
composability,
which
does
not
exist
in
the
existing
systems,
so
yeah
with
this
metadata.
If
you
compare
this
to
an
airflow
dag
right,
this
is
what
the
airflow
dag
looks
like
in
the
UI
right.
A
All,
it
is,
is
goals
with
a
label
and
then
arrows
between
them
write
tasks
and
then
the
execution
order,
those
tasks.
That's
it
that's
all
the
metadata
system,
that's
all
the
metadata
that
the
user
gets
just
to
view
it
and
understand
it.
But,
more
importantly,
that's
all
the
metadata
that
the
system
gets
in
order
to
actually
do
useful
things.
A
A
This
no
go
to
how
this
works
this.
Actually,
this
dag
up
to
the
left
here.
This
is
actually
compiling
and
actually
programmatically,
creating
that
airflow,
dag
and
I'll
go
into
how
that
works,
but
it's
actually
a
layer
abstraction.
It
can
be
used
as
a
layer,
abstraction
above
air
flow.
So
the
other
thing
that
I
wanted
to
show
so
you
have
four
minutes
left,
is
how
this
can
be
used
as
kind
of
a
multidisciplinary
pipeline.
A
So
the
the
example
I
showed
you
kind
of
before
was
just
basic
Python
code
right,
but
here
is
an
example
pipeline
where
the
first
thing
is
actually
executing
sequel
in
some
database
that
kicks
off
another
sequel
job.
This
one
actually
kicks
off
a
Jupiter
notebook
and
if
you
heard
the
first
talk,
Michels
talk
Michelle's
here,
but
this
actually
is
invoking
a
paper
mill
notebook
as
a
function
right,
and
so
what
this
represents
is
a
pipeline
that
gets
integrated
right
and
is
using
solids
crafted
by
different
participants
in
the
data
application.
A
A
And
so
the
system
right
at
the
end,
what
daxter
is
or
one
of
the
things
it
is.
Is
it
it's
an
API
by
which
you
can
query
these
computations
for
their
metadata?
So
you
have
a
abstract,
computational
graph,
which
is
represented
in
that
UI.
That
I
showed
you
accessible
behind
the
API.
Each
one
of
those
solids
can
be
implemented
in
arbitrary
processing
tools.
Again
from
the
standpoint
of
the
system,
the
implementation
of
a
solid
is
totally
black
box
function.
So
this
is
any
storage
any
language,
any
runtime.
A
You
have
our
daxter
native
tools,
so
that's
that
log
viewer
I
showed
you
the
graph
viewer.
The
live
log
viewer,
the
simple
scheduler
all
that
stuff,
but
we
also
integrate
with
existing
ops
of
infrastructure
right.
This
is
just
a
Python
library
and
just
a
software
abstraction,
which
means
you
could
deploy
it
it
on
arbitrary
computation
right.
So
you
can
execute
this
on
kubernetes.
A
But
what's
interesting
is
that,
like
I
showed
before,
we
also
have
the
ability,
for
example,
to
use
the
API
to
actually
say:
hey,
give
me
this
dag
here's.
This
configuration
program
recreate
me,
the
airflo
dag,
which
corresponds
to
that,
which
means
that
if
you
have
an
existing
airflow
cluster
and
you
don't,
you
want
to
experiment
with
this
up
or
abstraction
without
totally
thrashing
your
infrastructure.
Maybe
you
don't
even
want
to
talk
to
the
people
who
maintain
your
airflow
cluster
because
they
fear
all
change
right.
A
A
Those
properties
make
it
amenable
for
both
data
science
and
data
engineering,
and
we
really
do
think
that
those
these
tools
that
we
built
on
top
of
it
really
set
a
new
standard
for
kind
of
you
know
the
developer
experience
and
the
user
experience
on
top
of
these
beautiful
web
UI
tools
and
with
that
I
will
close
it
out.
Thank
you
for
listening.
B
A
Are
no
bugs
and
it
is
perfect,
obviously
no
I
mean
so.
This
is
definitely
like
on
the
early
side
of
her
framework,
and
you
know
so.
There's
advantages
disadvantage
that
it's
not
as
battle
tested
right
and
something
like
that's
been
around
for
ten
years,
but
it's
under
super
active
development
by
what
we
hope
are
reasonably
competent
engineers
who
respond
extraordinarily
quickly
to
both
feature
and
bug
requests.
A
A
great
question,
so
that
was
one
of
the
Scott
slides
I
skipped
actually,
but
metadata
is
a
very
broad
term.
The
metadata
Iowa's
focused
on
during
most
the
presentation
was
metadata
about
the
computations
itself,
meaning
that
you
can
like
query
it
and
ask
it
for
its
dag
structure.
I
am
like
the
types
and
all
that
kind
of
stuff,
but
there's
also
this
metadata
that
the
each
of
these
solids
emits
facts
about
what's
happening.
A
So,
instead
of
just
unstructured
logging,
it
will
say:
hey
I,
just
passed
this
data
quality
test,
I
just
created
this
materialization
that
will
outlive
the
scope
of
the
computation
right
now.
The
only
way
we
persist,
that
is
in
this
structured
event,
log,
but
we've
hoped
to
work
with
our
users
and
to
build
up
the
stack
such
that
that
can
easily
be
used
to
persist,
kind
of
an
artifact
store
or
a
system
of
record
for
metadata.
So
that's
a
great
question
and
that's
where
we're
going
with
the
system.
B
A
Fantastic
question
right
now:
natively,
it's
only
a
time-based
scheduler.
As
I
said
these
invoke.
We
have
like
a.
We
have
a
you
know:
spoiler
alert.
We
have
a
graphic
you'll
API
over
all
this
and
what
you
can
do
is,
if
you
have
some
invent
based
trigger.
You
can
just
invoke
that
API
to
in
queue
a
scheduled
execution.
A
A
Are
one
of
these
mystical
questions
about
when
to
call
something
one
point:
oh
I
mean,
for
example,
still
isn't
one
oh
and
it's
probably
responsible
for
the
entire
popularity
of
Python
in
the
data
ecosystem
in
general.
So
it's
a
relatively
arbitrary
term.
I
get
asked
this
all
the
time,
so
I
probably
should
come
up
with
some
sort
of
like
coherent
stick
about
when
we'll
call
it
one.
Oh,
it
is
relatively
arbitrary.
A
A
A
Think
about
that
I
mean
knowing
which
rows
have
changed
from
a
pipeline.
We
consider
part
of
the
business
logic
of
the
pipeline
itself,
like
I.
Think
it's
very
difficult
to
have
like
super
opinionated
viewpoint
on
that,
without
having
much
more
context
about
what
the
computation
is
doing
so
I'm
gonna
punt
that
and
say
like
right
now.
That's
not
a
concern
of
our
system
is.
We
would
consider
a
part
of
the
business
logic
in
your
pipeline.
Does.
B
A
A
great
question
so,
as
far
as
I
know
so
does
does
daxter
serialize
data
between
solids.
The
answer
is:
yes,
it's
part
of
its
capability,
so
you
know
you
can
express
things
in
terms
of
inputs
and
outputs
our
data
frames
and
then
the
system
will
be
responsible
for
serializing
the
contents
of
this
thing,
so
you
can
pass
them
from
node
to
node.
A
Now
catalyst
is
an
internal
implementation
detail
of
the
SPARC
engine,
so
it's
their
internal
in
memory
format,
that's
used
within
the
context
of
a
spark
cluster,
so
that's
really
only
relevant
within
the
context
of
one
spark
job
which
would
be
contained
within
a
single,
solid.
Now,
I
think.
The
question
you
maybe
wanted
to
ask
is:
can
we
use
a
system
called
arrow
which
is
an
open
standard
for
in-memory
data
frames
and
I?
Think
that
would
be
a
really
interesting
project
to
have
to
be
able
to
use?
What
I
would
probably
propose
is
there's
this
object.
A
There's
a
memory
object,
store
called
plasma
with
the
jannat
project
called
ray,
and
you
can
persist
the
arrow
within
that
and
you
could
use
that
as
the
intermediate
store
between
the
solids.
But
whoever
asked
that
question
should
come
talk
to
me,
because
that
was
very
good
and
maybe
in
come
work
for
us
and
on.