►
From YouTube: Rethinking Orchestration as Reconciliation: Software Defined Assets in Dagster | Elementl
Description
ABOUT THE TALK
This talk discusses software-defined assets, an approach to orchestration and data management that makes it drastically easier to trust and evolve data assets, like tables and ML models.
In traditional data platforms, code and data are only loosely coupled. As a consequence, deploying changes to data feels dangerous, backfills are error-prone and irreversible, and it’s difficult to trust data, because you don’t know where it comes from or how it’s intended to be maintained. Each time you run a job that mutates a data asset, you add a new variable to account for when debugging problems.
Dagster proposes an alternative approach to data management that tightly couples data assets to code - each table or ML model corresponds to the function that’s responsible for generating it. This results in a “Data as Code” approach that mimics the “Infrastructure as Code” approach that’s central to modern DevOps. Your git repo becomes your source of truth on your data, so pushing data changes feels as safe as pushing code changes. Backfills become easy to reason about. You trust your data assets because you know how they’re computed and can reproduce them at any time. The role of the orchestrator is to ensure that physical assets in the data warehouse match the logical assets that are defined in code, so each job run is a step towards order.
Software-defined assets is a natural approach to orchestration for the modern data stack, in part because dbt models are a kind of software-defined asset.
Attendees of this session will learn what it looks like to build and maintain a warehouse or data lake of software-defined assets with Dagster.
ABOUT THE SPEAKER
Sandy is a software engineer at Elementl, building Dagster. Prior, he led machine learning and data science teams at KeepTruckin and Clover Health. He's a committer on Spark and Hadoop, and co-authored O'Reilly's Advanced Analytics with Spark.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai/
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
A
So
a
couple
years
ago
I
came
back
to
the
world
of
data
tooling
to
try
to
fill
what
I
saw
as
the
biggest
gap.
I
joined
the
elemental
team
to
work
on
dagstar,
but
before
we
get
to
that
gap
and
before
we
get
to
dagstar
I'd
like
to
step
away
from
the
world
of
data
entirely
and
talk
about
front-end
engineering
and
cluster
orchestration
and
devops,
these
domains
have
something
in
common,
which
is
that
they've
all
gone
through
a
big
transition
in
the
last
decade
or
two.
A
So
we
used
to
build
websites
using
jquery,
but
now
we
use
react.
We
used
to
orchestrate
our
compute
clusters
using
yarn
or
mesos,
but
now
we
use
kubernetes
and
we
used
to
provision
infrastructure
using
high
powered
shell
scripts.
But
now
we
use
terraform,
so
technology
transitions
happen
all
the
time.
A
For
those
of
you
who
aren't
familiar
with
these
terms
here
are
some
crude
definitions,
so,
under
an
imperative
paradigm,
you
give
commands
telling
your
system
what
to
do
change.
This
change
that
with
the
declarative
paradigm,
you
describe
the
end
state
that
you
want
your
system
to
be
in,
and
then
you
let
a
control
plane
manage
change.
A
So
what's
going
on
here,
why
do
we
see
this
pattern
of
change?
If
you
take
a
closer
look
at
the
history
of
any
of
these
software
domains,
you
start
to
get
a
sense
of
it.
So
the
initial
challenge
is
about
speed
to
build
stuff.
It
takes
developers
a
long
time
to
get
basic
things
done
like
provision,
an
ec2
server
or
change.
What
shows
up
in
a
text
box
tools
come
along
that
speed.
This
up
dramatically.
A
A
A
A
A
So
now,
let's
talk
about
a
domain
with
complexity
and
change
of
the
wazoo,
and
that
domain
is
data
in
the
world
of
data.
Where
are
we
on
this
tooling
journey?
Well,
it
used
to
be
very
hard
to
build
stuff
10
years
ago.
If
you
wanted
to
calculate
some
statistic
over
every
event
on
your
website,
you
need
to
set
up
a
compute
cluster
with
20
different
services,
and
even
the
simplest
operations
could
take
tens
of
minutes
to
complete.
A
Since
then,
we've
built
powerful
databases
and
data
processing
engines
that
make
this
all
effortless.
So
now,
with
snowflake
or
spark,
you
can
write
three
lines
of
code
and
get
a
result
in
a
couple
of
seconds
and
we've
built
workflow
managers
that
allow
you
to
easily
chain
a
bunch
of
data
processing
steps
together.
A
A
So
there's
a
lot.
That's
great
about
this
picture.
Maintaining
intermediate
data
sets
lets
us,
reuse,
valuable
data,
cleaning
work
and
supporting
multiple
ways
of
processing
data
gives
us
the
flexibility
to
use
the
right
tool
for
the
job,
but
without
extremely
careful
shepherding.
This
complexity
naturally
devolves
into
chaos.
A
We
end
up
with
multiple
versions
of
the
same
things
created
by
different
people
or
even
by
the
same
people
at
different
times.
It's
hard
to
distinguish
trustworthy
maintained
data
from
one-off
artifacts
that
went
stale
months
ago
discovering
how
a
table
or
machine
learning
model
is
generated
often
requires
a
heroic
act
of
code.
Spelunking
and
debugging
requires
untangling
a
complicated
history
of
state
mutations.
A
This
chaos
interferes
with
our
ability
to
accomplish
our
goals.
We
can't
trust
our
data
because
we
don't
know
how
it's
generated
or
how
it's
expected
to
be
kept
up
to
date,
and
we
spend
most
of
our
energy
waiting
through
our
existing
system.
So
we
don't
have
the
mental
space
to
expand
it,
which
is
all
to
say
that
I
think
we're
solidly
in
this
upper
right
box.
Managing
all
the
data
in
pipelines
is
still
really
hard.
A
What
would
it
take
to
bring
a
declarative
approach
to
data
management
and
tame
this
chaos?
So
a
declarative
approach
fundamentally
revolves
around
the
declarative
entity
in
front-end
engineering.
The
entity
is
a
ui
component.
You
declare
the
ui
components
that
you
want
to
exist
and
how
you
want
them
to
be
rendered
in
devops.
It's
a
resource.
You
declare
the
servers
or
other
infrastructure
components
that
you
want
to
exist
and
what
you
want
their
properties
to
be.
A
It
could
be
a
source
asset,
that's
ingested
directly
from
some
external
system.
It
could
be
a
derived
asset.
It's
built
from
source
assets
or
other
derived
assets.
Creating
and
maintaining
data
assets
is
the
main
reason
that
people
build
data
pipelines
in
the
first
place.
Assets
function
as
the
interfaces
between
different
teams,
they're
the
objects
that
we
inspect
when
we
want
to
debug
problems
and
they're.
Ultimately,
the
products
that
data
teams
build
for
the
rest
of
the
world
managing
assets,
declaratively
means
defining
them
in
software.
A
A
Materialization
is
the
word
we
use
for
a
physical
manifestation
of
an
asset,
for
example:
example
an
actual
table
in
a
database
or
file
in
a
file
system.
If
this
all
sounds
a
little
bit
abstract,
that's
fine!
I'm
going
to
spend
the
rest
of
this
talk,
making
this
concrete
and
walking
through
the
implications,
and
I'm
going
to
do
that
through
the
lens
of
dagstar.
So
for
those
of
you
who
aren't
familiar
dagster
is
an
open
source
orchestration
platform.
A
A
A
A
A
A
A
Something
you
might
notice
here
is
that
I'm
not
showing
any
code
that
actually
reads
or
writes
to
or
from
physical
storage.
The
op
is
a
pure
function
to
support
this
functional
style
of
programming.
Dagstar
allows
the
code
that
writes
an
asset
to
storage,
to
be
provided
separately
and
shared
by
multiple
assets.
A
A
A
A
A
A
Once
you
define
a
set
of
assets
in
code,
either
directly
in
python
or
by
loading
them
from
an
external
tool,
you
can
start
working
with
them
in
dagster's
web
ui.
So
here's
how
you
tell
dexter
to
scan
a
python
package
and
make
all
the
assets
inside
of
it,
loadable
from
the
web
ui
and
here's
that
web
ui.
We
have
a
visual
representation
of
all
the
assets
that
we
were
just
looking
at
in
code.
A
A
A
A
The
second
is
that
there's
no
central,
dag
artifact
that
holds
the
dependency
structure
each
asset
on
its
own
knows
about
the
upstream
assets
that
it
depends
on.
This
makes
it
scalable
to
large
numbers
of
assets.
You
don't
need
a
central
team
to
maintain
a
code
artifact
that
captures
all
of
them,
so
here
we're
looking
at
just
the
assets
that
I
showed
you.
The
definitions
of
we've
also
got
links
to
assets
that
these
assets
depend
on,
but
that
are
loaded
by
dexter
separately.
A
They
correspond
to
code
in
our
git
repo,
not
artifacts,
in
physical
storage,
that
is,
they
have
no
associated
materializations,
but
dexter
doesn't
just
track
asset
definitions.
It
also
allows
you
to
materialize
assets
and
tracks
those
materializations,
so
we
can
click
this
button
on
the
upper
right
to
materialize,
all
the
assets
that
are
shown
here
that
launches
a
run
which
contains
a
log
of
everything
that
happens
as
we
execute
the
computations
that
materialize
these
assets.
A
A
Here's
the
page
for
an
individual
asset.
Our
activity
forecast
asset
that
we
defined
in
python.
It
contains
details
about
the
asset
and
its
history
of
materializations,
a
materialization
isn't
just
a
record
saying
that
a
computation
ran
it's
a
description
of
the
object
that
was
produced
by
the
computation
dexter
lets
users
plug
in
code
that
records
runtime
metadata
about
the
asset,
that's
being
materialized.
A
So
let's
imagine
that
we're
in
our
bi
tool
we're
using
this
asset
to
try
to
understand
what
the
future
holds
for
our
website.
We
notice
something
fishy,
maybe
some
of
the
data
that
we
expect
to
be.
There
is
missing,
so
we
can
pull
up
this
search
box
in
dagstr
type
in
the
asset
that
we
care
about
and
get
to
the
asset
page
that
we
were
looking
at
before
from
that
page.
A
A
New
upstream
data
is
constantly
arriving
and
we're
constantly
changing
the
code
that
derives
our
assets
from
that
upstream
data.
The
result
is
that
our
asset
materializations
naturally
drift
from
their
definitions
over
time.
When
that
drift
occurs,
we
need
to
launch
computations
to
bring
those
materializations
in
line
with
our
definitions
and
launching
computations
at
the
right
job.
At
the
right
time
is
typically
the
job
of
an
orchestrator.
A
A
A
But
if
you
try
to
pose
these
questions
to
a
traditional
orchestrator,
you'll
immediately
run
into
the
issue.
They
have
no
idea
what
an
asset
is.
They
can
tell
you
what
tasks
have
completed,
what
tasks
are
meant
to
run
after
each
other,
but
not
about
how
any
of
that
has
to
do
with
the
assets
that
these
tasks
are
reading
and
writing.
A
A
Dexter
supports
two
approaches
to
orchestration
for
assets.
The
first
one
looks
very
similar
to
how
scheduling
works
in
traditional
orchestrators,
except
that
it
focuses
on
assets.
Instead
of
tasks,
you
can
select
a
set
of
assets
re-materialized
at
some
regular
cadence
or
via
a
sensor
based
on
arbitrary
logic.
A
A
A
Putting
the
schedule
on
the
asset
instead
of
on
a
task
makes
it
easy
for
stakeholders
to
answer
questions
about
when
assets
will
be
updated
next,
when
they
navigate
to
the
page
for
an
asset,
they
can
see
the
frequency
that
it's
going
to
be
updated
at
so
just
by
framing
scheduling
in
terms
of
assets
instead
of
tasks.
The
orchestrator
becomes
an
ideal
system
for
answering
these
questions
that
we
looked
at
earlier.
A
A
A
Another
discrepancy
is
when
the
contents
of
an
asset
don't
reflect
the
contents
of
the
upstream
assets
that
it
depends
on.
So
if
the
most
recent
materialization
of
the
asset
occurred
earlier
than
the
most
recent
materialization
of
the
upstream
asset,
then
it's
likely
that
the
asset's
contents
are
stale.
Dagster
raises
its
discrepancy
with
an
upstream
changed
indicator.
A
A
A
A
For
example,
in
the
past
to
build
a
derived
table,
you
would
write
an
airflow
task
that
issues
a
create
table
statement
to
your
database
nowadays
you're
more
likely
to
define
the
table
directly
as
a
dbt
model
in
the
past.
If
you
wanted
to
ingest
the
table
into
your
data
warehouse
from
your
production
database,
you
would
write
an
airflow
task
that
executes
a
sync
oops
nowadays,
you're
more
likely
to
just
tell
5tran
or
airbyte
or
miltano
that
you
want
that
table
to
exist
in
your
warehouse.
A
With
these
declarative
approaches,
you
get
all
the
declarative
benefits
that
we
discussed
earlier.
So
it's
understandable
why
they're
attractive
and
why
we're
moving
in
this
direction,
but
there's
something
that's
lost
as
well
and
to
understand
it.
We
need
to
go
back
in
time
and
take
a
look
at
the
pre-modern
world
in
the
pre-modern
world.
You'd
use
an
imperative
orchestrator
like
airflow,
to
invoke
computations
across
a
variety
of
frameworks
as
we
covered
earlier.
A
This
centralized
orchestration
layer
makes
it
easy
to
understand
everything,
that's
running
in
one
place
and
to
make
sure
that
everything
is
running
in
the
right
order.
Even
across
tools
in
the
world
of
the
modern
data
stack
purpose-built
tools
for
each
of
the
verticals,
essentially
roll
their
own
orchestrators.
A
A
So
it's
easy
to
lose
track
of
what
ran
what's
supposed
to
ran
and
whether
things
ran
in
the
right
order,
and
you
might
also
notice
that
python
has
largely
been
left
behind
from
this
picture.
By
focusing
on
specialty
tools
that
simplify
common
data
patterns,
we
risk
losing
our
ability
to
run
generalized
computation
when
we
need
it,
so
we're
forced
to
choose
between
a
a
unified
approach.
That's
stuck
in
an
imperative
paradigm
or
b
a
declarative
approach.
Without
a
unified
control
plane,
I
don't
think
there's
any
fundamental
reason.
We
need
this
trade-off.
A
A
The
assets
in
the
middle
with
the
orange
tags
are
dvt
models,
materializing
them
in
dagster
clicking
on
the
button.
You
know
on
the
asset
page
results
in
a
call
to
dbt
either
cli
rbc,
server
or
dbt
cloud,
and
the
asset
with
a
light
blue
tag
on
the
lower
right
is
an
asset.
That's
computed,
using
python
downstream
of
one
of
these
dbt
models.