►
From YouTube: Dagster: Workflows for Data Science, Machine Learning, and Data Engineering | Elementl
Description
ABOUT THE TALK (https://www.datacouncil.ai/talks/dagster-workflows-for-data-science-machine-learning-and-data-engineering)
Dagster is an open-source, Python library for building data applications: ML, Analytics, ETL, and more. We will describe how Dagster can enable a unified stack of tooling across a wide dynamic range of use cases: from a laptop to a Kubernetes cluster; from local development, to monitoring in production; from data science, to data engineering, to machine learning.
ABOUT THE SPEAKER
Nick Schrock is the founder and CEO of Elementl, a company aiming to reshape the data management ecosystem, and the creator of Dagster, a new programming model for data processing. Previously, Nick was a Principal Engineer and Director of Engineering at Facebook. In that time, Nick co-created GraphQL, and led its implementation and adoption across the entire organization and product line. He also formed the Product Infrastructure group, whose engineers, in addition to GraphQL, created React, React Native, and many other broadly-used developer technologies, both inside Facebook and the technology industry at large.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
A
A
So,
first
of
all
we're
an
orchestrator.
Our
job
is
to
manage
and
orchestrate
the
graph
of
computations
that
comprise
what
we
like
to
call
data
applications
and
I'll
go
into
that
term.
In
a
second
data
applications
week
yeah
we
effectively
view
ml
pipelines,
etl
elt,
et
cetera
as
effectively
the
instances
of
the
same
software
problem.
A
Next
we
are
metadata
and
data
aware,
and
we
think
this
is
kind
of
like
our
you
know
what
really
distinguishes
us
from
adjacent
systems
in
this
space,
which
are
generally
agnostic
to
the
data
flowing
through
their
systems
and
just
about
the
computations
and
managing
dependency
order.
We
also
have
a
much
more
opinionated
programming
model
which
allows
us
to
build
much
more
rich,
tooling
and
then.
A
So
let's
talk
about
this
data
application
concept,
so
we
define
a
data
application
as
a
graph
of
functional
computations
that
produce
and
consume
data
assets.
In
the
end,
that's
what
we're
all
trying
to
do
here,
we're
taking
files
we're
doing
computation
on
them,
we're
producing
databases,
database
tables
or
something.
A
So
if
you
kind
of
google
around
right,
look
for
the
more
familiar
terms
in
the
space,
you
have
stuff
like
ml
pipelines,
and
this
is
literally
the
images
I
found
when
I
googled
around,
and
you
have
like
the
sequence
of
steps
where
you
start
with
a
file.
You
end
up
with
a
model
that
is
being
served
or
you
have
like
a
classic.
You
know.
If
you
look
for
etl,
you
get
some
real
graphic
design
monstrosities
there,
and
this
is
a
classic
of
ingesting.
A
You
know
csv
and
using
set
and
awk
to
a
multi-threaded
java
process
which
I've
found
quite
charming
and
then
there's
kind
of
the
more
modern
du
jour
elt
stage,
which
is
effectively
etl
except
extend.
Instead,
you
just
persist
the
raw
data
before
you
process
it
right
and
effectively
most
modern
systems.
A
Just
kind
of
do
this,
as
a
matter
of
course,
these
days
and
so
what's
interesting
about
this-
is
that
we
think
these
are
all
fundamentally
the
same
systems
right
they're
all
these
kind
of
like
data
factories
that
consume
data
and
like
flow
it
through
they're
all
graphs
of
functions
at
the
core,
that's
what
they
do
yeah.
I
had
to
insert
this
slide
after
last
week's
talks,
where
literally
four
out
of
four
of
the
presentations
all
put
up
this
slide.
A
So
I
want
to
keep
that
streak
alive,
but
it's
actually
very
germane
because,
like
if
people
ml
people
sometimes
hear
this
and
like
wait,
I
don't
do
etl.
That's
that's
someone
else's
job,
but
probably
the
most
like
famous
graphic
in
ml,
explains
how
all
of
the
work
is
everything
which
isn't
which
isn't
ml
right.
If
you
take
out
serving
infrastructure
and
ml
code
in
this
chart,
you're
basically
left
with
what
hitherto
has
been
known
as
etl
right
data
processes,
the
core
of
all
these
systems.
A
A
So
by
artificially
siloing
these
pieces
of
software,
you
actually
compartmentalize
a
system
which
should
actually
be
treated
like
one
logical
system,
okay.
So
what
are
the
properties
of
these
things?
Well,
I
can
say:
they're
multi
right.
It's
a
very
heterogeneous
world,
multi-persona
they're,
logical
systems
that
people
is
varied
of
analysts,
data
scientists
and
data
engineers
interact
with
their
multi-tool
each
one
of
those
people
I
mentioned
has
very
different
cultures.
They
have
very
different
skill
sets.
They
use
very
different
tools.
They
have
different
preferences.
You
have
to
go
to
where
the
people
are.
A
You
can't
force
everyone
to
encode
their
business
logic
in
one
system.
It's
also
multi-team,
just
kind
of
like
gets
more
and
more
complicated
right.
These
systems
span,
the
entire
enterprise
need
to
subdivide
the
application
of
the
teams
and
then,
as
a
result,
those
teams
often
end
up
running
their
own
environments
right
and
it's
just
like
an
extremely
extremely
complicated
domain.
A
The
last
thing
to
say
is
that
this
is
software
engineering
and
what
I
mean
by
that
is
that
a
lot
of
times
traditionally,
especially
when
you're
talking
about
the
etl
term,
the
the
move
has
been
to
try
to
like
move
kind
of
analysts
or
people
who
aren't
engineers
into
a
gui
tool
that
they're
comfortable
with
and
get
them
to
do
engineering
tasks
in
a
non-engineering
way.
We
at
dexter,
finally,
don't
believe
that's
how
the
world
works.
A
You
know
one
example
of
this
that
I
love
is
a
project
that
I'm
a
huge
fan
of
which
is
dbt,
which
has
taken
analysts
and
instead
of
trying
to
like
jam
them
into
a
gui
and
say
you
don't
have
to
be
an
engineer.
They
say
no,
here's
a
tool
that
makes
sense
to
you.
That
makes
you
into
an
engineer,
and
you
know
this
data
is
a
software
engineering
discipline.
It
needs
tools
that
acknowledge
the
complexity
of
that
and
embrace
it
all
right.
A
So,
let's
talk
about
the
the
data
application
life
cycle
so
another
one
of
our
beliefs
is
that
this
core
orchestrator,
which
is
the
thing
that
knows
about
everything
in
the
system
right?
It
knows
like
the
orchestrator,
a
data
where
orchestrator
can
know
about
every
single
runtime
that
interacts
in
the
data
application.
A
Every
single
piece
of
data
that
gets
produced
every
single
piece
of
data
gets
consumed.
It's
kind
of
the
fundamental
structure
of
this
thing,
and
we
think
that
this
graph,
the
the
computational
graph
of
the
data
application
models,
should
be
involved
and
support
every
stage
of
the
software
life
cycle.
So
you
should
be
fundamentally
using
the
same
tools
to
both
locally
develop
on
your
laptop
to
some
degree
locally,
developing
your
laptop
to
operating
this
thing
in
a
cluster.
A
So,
let's
talk
about
develop
here,
so
we're
going
to
do
one
of
these
classic
like
toy
examples
right
where
we're
going
to
load
some
data
data
frame
into
pandas,
we're
going
to
take
that
data
frame
and
augment
it
with
call
it
a
feature.
That's
what
sugar's
for
cup
is
we're
going
to
compute
something
based
on
that
feature
and
then
do
some
business
logic.
So
all
we
do
here
is
we
find
like
the
top
25
of
cereals
based
on
sugarcoat
per
cup.
A
We
compute
the
75
cut
off
and
then
we
filter
everything,
and
then
we
write
it
back
to
a
csv
file
right
classic
blinking
lights
demo.
So
this
is
just
a
raw
python
script
that
does
that
right
and
for
those
of
you
who've
done
it.
This
is
pandas,
it's
a
like
very
well
known
data
processing,
library
and
and
yeah.
So
it's
just
like
basic
code.
A
On
your
existing
python
code,
so
here
all
we
do
is
we
add,
we
call
our
unit
and
we'll
go
and
do
the
second
we'll
call
our
kind
of
like
node
in
our
graph
a
solid
and
we
annotate
it
with
a
solid,
and
then
we
add
this
first
parameter,
which
you
can
go
into
later
and
then
to
orchestrate
this
thing.
You
just
kind
of
like
naturally
call
python
code,
but
it
this
actually
constructs
a
representation
of
the
debt
of
the
computation
rather
than
executing
it
eagerly.
A
So
the
same
before
we
have
the
solid.
We
call
it
like
a
functional
unit
of
computation
in
the
orchestration
graph.
In
the
end,
it's
just
a
function
with
some
specific
semantics
that
that
framework
knows
how
to
interact
with,
and
it's
specifically
designed
for
reuse
for
testability,
and
I
hope
I
can
show
you
those
features
as
we
go
more
and
then
we
have
a
pipeline,
which
is
a
graph
of
solids.
I
think
the
thing
to
note
here
is
that
we
are
connected
via
data
dependencies.
A
So,
for
example,
if
you
look
at
filter
below
cutoff,
it
has
like
a
serials
input,
which
is
represents
a
data
frame,
and
you
have
to
you
know,
route
that
from
a
previous
computation
which
outputted
a
data
frame.
So
we
you
know,
encourage
the
user
to
encode
their
data
dependencies
in
this
way,
and
we
think
this
leads
to
much
more
reliable,
far
more
testable
systems.
A
Let
me
share
my
screen
here,
all
right,
I'm
just
going
to
assume
everyone
can
see
this
and
if
we
hop
here
so
this
is
what
the
you
know.
This
is
what
the
gui
looks
like
when
we're
just
looking
at
this
thing.
So
we
have
a
representation
in
this
web
app
you'll
notice
that
we
have
data
dependencies,
note
how
we
have
the
any
type
anywhere
we
kind
of
like
automatically
assign
the
any
type
so
that
so
that
ends
up
being
gradual
and
optional.
A
And
now
so
we
can
view
this
and
we
can
also
execute
it
right,
and
so
I'm
actually
going
to
configure
this
thing
to
run
each
of
the
steps
in
its
own
process,
and
this
is
actually
here
live
on
my
laptop.
This
is
actually
kind
of
terrifying,
because
the
computational
expense
of
might
totally
shut
this
thing
down
and.
A
A
So
we
go,
and
so
this
is
this
beautiful
kind
of
gantt
chart
viewer
that
we
have
which
live,
updates
and,
like
I
don't
know
how
good
it
ends
up
looking
over
zoom,
but
it's
it's
very
pleasant
to
use
I'll
give
you
that
we
have
logging.
We
have
all
this
stuff.
You
know.
One
interesting
thing
is
that
one
of
the
things
the
system
is
doing
is
it's
automatically
managing
kind
of
shuffling
the
intermediate
values
from
one
node,
in
that
my
case
on,
my
laptop
is
just
to
process
the
machine
to
another.
A
This
allows
us
to
do
things
like
okay.
We
can
just
select
this
node,
which
is
this
one
thing
we
can
see.
A
log
message
is
happening.
We
can
just
like
re-execute
that
step
right
so
by
even
just
this,
like
minimal
decoration
of
all
these
kind
of
functions,
you
get
like
a
ton
of
capabilities
right
outside
the
box.
You
can
see
we
actually
like
link
retries
versus
run,
lineage
and
group
them
together.
These
end
up
being
views
in
a
run
viewer.
You
can
see
all
this
information.
A
You
can
go
back
to
the
run,
so
it
really
is
a
joy
to
use
okay.
So
we
have
you,
know
nice
demo,
nice
ui
has
some
interesting
capabilities,
but
you
know
we
haven't
really
done
that
so
much
interesting
yet,
and
so
I'm
going
to
take
this
one
second
year,
there's
this
classic.
You
know
what's
the
next
problem,
we
want
to
deal
with.
Well,
there's
this
classic.
You
know
line
about
a
founders
fund
which
is
like
we
wanted
flying
cars.
We
got
140
characters
right,
but
this
was
like
10
years
ago.
A
I
think
the
slogan
needs
updating
and
I
would
suggest
in
2020.
The
right
answer
would
be
this,
which
is
we
got
100.
We
wanted
flying
cars,
we
have
yaml,
oh
my
god
do
we
have
yaml.
I
mean
everyone
talks
about
yeah
with
kubernetes,
and
this
like
this,
like
cambrian
explosion
of
yaml,
but
it's
equally
bad,
not
bad.
It's
equally.
It
is
what
it
is
in
the
data
application
space
and
you
know
just
a
huge
amount
of
apple.
A
huge
amount
of
configuration
goes
into
any
sort
of
internal
platform
of
any
sort
of
significance.
A
A
So
we
thought-
and
in
our
approach
for
testability
we
were
probably
going
to
make
configuration
more
complicated,
so
we
thought
we
need
to
build
software
abstractions
and
manage
that.
So
we
have
a
configuration
system
now
go
over
this
quickly,
but
what
it
effectively
is
is
schema
over
python
dictionaries.
A
A
You
catch
errors
earlier
in
the
process
and
you
have
an
auto
completing
yaml
editor,
which
is
really
nice
so
just
to
demo
this
very
quickly.
You
know
I
wanted
to
figure
out
how
to
demo
the
config
system
like
well.
I
would
actually
like
to
well.
This
is
before
the
thing
slowed
down
my
computer,
but
I
wanted
to
inject
computational
delays
into
this
into
this
demo
to
demo
the
gantt
chart
better,
I'm
like
okay,
why
don't
we
use
the?
A
Why
don't
we
use
the
you
know
this
config
editor
to
do
that,
so
we
go
there
and
instead
we
oh
switch
right
in
here.
That's
faster,
go
the
next
step
in
the
demo
you'll
see
here.
You
have
like
this
config
right
here
and
then
you
can
go
to
the
playground
here
and
I
preloaded
some
config.
But
let's
pretend,
like
we
hadn't
typed
that
and
now
you
can
see
on
the
right
hand,
side.
You
have
the
schema
right
there
about
how
you
can
configure
this
thing
and
we
can
say
actually
I
want
to
delay
this.
A
Actually,
let
me
show
you
see
how
the
documentation
is
right
in
there
so
nice.
So
now
we
can
do
this
and
we
can
launch
it
and
now
you'll
see
that,
like
we
have
been
able
to
configure
this
delay
configure
this
delay.
The
next
one
should
be
a
lot
slower
boom.
There
we
go
so
you're
able
to,
like
you,
know,
you're
able
to
adorn
different
artifacts
in
our
system
with
config
it's
self-describing.
It
can
be
very
complicated
and
dusty,
and
you
get
all
this
tooling
support
around
it.
A
Next,
we
have
our
type
system,
so
you
notice
in
the
previous
demo
that
all
the
connections
between
the
nodes
were
annotated
with
the
any
type.
That's
because
we
have
a
type
system
it
has
to
have
a
type
and
the
way
we
make
this
thing
gradual
and
optional.
So
you
can
like
kind
of
like
opt
into
it
is
by
defaulting
everything
to
any.
A
A
Right
with
you
know
the
fact
that,
like
it
takes
this
type,
it
spits
out
this
type,
etc.
It's
great
for
documentation,
you
know,
and
you
can
like
trust
it
more,
and
you
know
what
this
gets
to
a
world
where
we
were
you
want
to
get
to
as
a
dexter
developer.
Is
you
want
your
solids
to
kind
of
describe
themselves?
You
want
to
look
at
this
and
understand
exactly
what
the
state
of
the
world
is
necessary
for
this
computation
to
complete.
A
So
in
this
case
and
we'll
build
this
up
over
time,
but
add
sugar
per
cup
says
like
hey:
if
I
have
access
to
a
data
frame
I'll,
do
computation
on
it
and
spit
out
a
data
frame.
That's
like
my
contract
with
the
world.
So
you
know.
One
interesting
thing
is
that
now
this
data
isn't
coming
from
anywhere,
and
so
you
might
be
thinking.
Where
does
that
come
from
and
one
of
the
behaviors
that
we
have
is
you
can
actually
specify
where
to
source
data
for
types
from
config?
A
So
the
moment
that
you
do
this,
you
can
go
to
the
next
demo
here,
and
so
we
have
this
dag
right.
You
have
this
data
frame
type
and
you
click
on
here.
It
says
it's
like
there's
just
all
this
gobbledygook
over
here
and
what
that
means
is
that
you
can
actually
specify
via
config
and
this
whole
config
schema
and
the
behavior
around.
It
is
totally
customized
customizable
by
the
user.
A
You
know
so
if
this
was
commented
out,
but
but
effectively,
what
you
can
say
is
like
hey
actually
for
this
thing,
execute
and
load
directly
to
csv,
the
user
doesn't
have
to
write
that
code
and
that
gets
incredibly
useful
once
you
start
talking
about
executing
subsets
of
these
things.
So
here
I'm
not
going
to
go
into
it,
but
you
can
like
select
sub
strokes
of
this
dag
and
execute
them
and
then
use
that
type
system
to
load
to
parameterize
those
computations.
A
Okay,
so
let's
talk
about
testing,
so
we've
talked
about
development
right
we've
kind
of
annotated,
a
relatively
simple
pipeline
with
the
dijkstra
primitives
we've.
We
have
this
kind
of
a
really
nice
local
ide
experience.
You
can
retry,
you
have
a
nice
developer
workflow,
but
now,
let's
talk
about
testing,
so
a
couple
words
on
testing
one
testing
data
applications
is
uniquely
challenging
and
you
know
it's
a
combination
of
things.
One
is
like
often
the
data
is
really
big.
A
So
testing
is
a
challenge
and
the
issue
with
that
is
that
the
status
quo
is
kind
of
like
if
you
think
about
all
the
stages
where
you
can
catch
defects
in
software
in
data
world,
it's
way
lopsided
towards
prod,
and
you
know
how
bad
that
is
right,
because
defects
that
are
caught
in
prod
a
bug
showing
up
in
a
ceo
dashboard
is
way
worse
than
one
showing
up
at
a
test.
Suite
right,
it's
kind
of
like
obviously
true,
and
you
know
our
goal
is
not
to
say,
like
everything
has
to
be
100
testable.
A
We
want
to
be
clear
that
we
want
to
be
practical
on
this.
Our
goal
is
to
bend
this
curve,
there's
still
with
the
nature
of
these
systems.
You
cannot
change
them.
There
is
some
testing
that
can
only
happen
in
production,
right
performance
testing
to
some
degree
like
you're,
only
able
to
catch
changing
prod
data
in
prod.
It's
just
the
nature
of
the
beast,
but
what
we
want
to
do
is
be
able
to
move
testing
farther
upstream
right,
like
with
our
config
system.
A
As
you
saw,
you
should
never
catch
a
configure
at
production,
never
zero
times
right.
Everyone
without
guardrails
like
this,
has
pushed
a
configure
and
a
prod.
It
just
happens
right
and
this
shift.
This
curve
shift
might
not
seem
like
a
big
deal,
but
it
is
because,
as
I
mentioned
before,
the
cost
massively
increases
at
each
stage
so
moving
because
you
know
the
number,
if
you
kind
of
like
multiply
these
two
curves
together,
that's
like
the
total
number,
the
total
amount
of
productivity
loss
and
damage.
A
As
a
result
of
defects
in
your
system,
bending
the
curve
like
we
just
showed,
you
is
a
huge
deal.
It
can
represent
like
an
order
of
magnitude,
magnitude
improvement
in
both
productivity,
because
your
dev
cycle
is
better
and
business
impact
costs
because
you
catch
errors
at
places
where
it's
way
less
terrible.
A
So
what
do
you
need
in
order
for
these
things
to
be
testable?
You
need
like
kind
of
three
things
and
if
any
one
of
them
aren't
there
you're
kind
of
out
of
you
you're
kind
of
in
trouble,
so
one
you
need
parameterized,
computations,
meaning
that
these
things
can
be
functions.
You
can,
for
example,
substitute
sample
data
for
real
data.
Two,
you
need
a
plugable
environment.
You
need
to
be
able
to
figure
out
a
way
to
isolate
yourselves
from
those
super
heavy
dependencies.
A
Three,
the
system
itself
has
to
be
pluggable,
so
like
stuff,
like
a
system
like
daxter,
has
a
database
where
you
track
runs
and
all
this
stuff.
You
know
in
order
to
run
that,
in
an
isolated
environment
you
need
to
be
able
to,
for
instance,
cut
your
dependency
on
a
database
in
some
cases
in
order
to
do
like
effective
unit
testing
as
an
example
so,
and
we
have
kind
of
abstractions
that
try
to
attack
all
these
layers
and
I'm
going
to
start
to
speed
up
now,
because
I
just
realized
that
I
have
too
much
content.
A
So
we
have
this
notion
of
a
resource
and
it's
a
different
abstraction
and
I'll
talk
about
two
things
here,
one
into
every
solid
we
pass
in
this
context
object
and
you
can
think
of
context
as
like.
This
is
the
way
that
you
interact
with
the
environment
that
you
have
to
interact
with,
meaning
that
you
have
here
this.
This
solid
says
that,
like
I
need
this
thing
called
a
data
lake
in
order
to
succeed
and
the
data
lake
is,
you
know,
presumably
some
like
internal
abstraction
that
someone's
written
at
this
company
to
save
data
files
somewhere.
A
So
this
solid
is
saying:
I
need
this
data
light
to
succeed
right
and
then
you
always
access
something
like
a
resource
off
of
this
context:
object,
which
means
that
you
can
change
the
version
of
it
that
happens
at
runtime,
so
this
is
kind
of
like
a
built-in
coarse-grained
dependency
injection
system.
If
you
know
that
term,
if
you
like
dependency
injection,
this
is
like
a
nice
pythonic
way
to
do
it.
If
you
don't
like
dependency
injection,
this
feels
nothing
like
dependency
injection,
but
it
ends
up
trying
to
accomplish
the
same
thing.
A
So
if
you
remember
that
couple
slides
ago,
I
was
saying
that
the
system
itself
is
pluggable
and
I
was
using
it
to
talk
about
testability,
but
this
property
of
the
system
is
key
to
both
testing
and
the
flexible
deployment
right.
We
explicitly
don't
believe
in
vertical
integration,
which
means
that
you
can
use
any
database.
We
want
you
to
be
able
to
use
any
execution
substrate
you
know,
and
so
on
and
so
forth.
The
world
is
very
complicated
and
heterogeneous
and
people.
You
know
there
is
no
one-size-fits-all
in
terms
of
the
infrastructure
stack.
A
So
what
we're
going
to
show
here
is
that
we
have
this
local
development
thing
and
now
we're
going
to
use
the
capabilities
that
we've
built
one
instance
of
our
stack,
which
we
have
support
for
to
deploy
that
pipeline
using
help.
So
we're
going
to
use
database
for
postgres
for
the
component
that
shuffles
data
between
the
computations
for
on
the
user's
behalf,
we're
going
to
use
s3
and
we're
going
to
actually
use
kubernetes
plus
celery
as
an
execution
substrate
so
kind
of
like
pretty
standard,
docker,
helm
stuff.
A
You
have
to
build
a
docker
file
which
is
like
your
daxter
dependencies
and
your
project.
You
have
to
parameterize
this
template
that
we
give
you
so
you
push
that
you
run
hell
and
then
boom
you're,
spinning
up
a
cluster
and
what
the
cluster
ends
up.
This
is
kind
of
what
it
gives
you
for
free.
I
just
want
to
talk
through
this
quickly
because
I
want
I
want
to
explain
like
how
we
think
you
should
run
the
system.
A
We
really
believe
in
spitting
up
ephemeral,
compute
resources
that
isolate
from
other
user
processes
in
the
system,
so
we
spit
up
a
kubernetes
job
per
run
and
we
also
do
purse
and
that's
kind
of
what
we
call
the
run
coordinator
and
we
also
isolate
every
single
step
of
execution
in
its
own
kubernetes
job
and
all
this
coordinates
via
a
postgres
database.
That's
also
deployed
using
this
helm
chart,
so
you
can
really
go.
I
mean
it's
definitely
some
work
to
get
this
set
up.
A
A
A
You
run
a
user-defined
function
and
it
produces
a
chunk
of
config
to
run
on
a
specific
pipeline
and
that's
what
it
does.
We
really
like
to
use
this
config
code
separation
to
enforce
some
testability
constraints.
You
have
this
chunk
of
code
that
is
fixed
and
then
you
only
vary
on
the
config
aspects.
This
also
helps
like
backfilling
because
effectively,
if
you
want
to
backfill,
you
kick
off
n
things
and
you're
providing
different
dates
into
this
function
and
queuing
up
a
bunch
of
runs.
A
Lastly,
we
have
what
we
call
the
asset
management.
Okay,
thank
you
pete.
I
am
free
to
go
to
955,
but
I
will
end
a
little
earlier
than
that
pete,
I
promise
so
asset
management.
You
know
we
talked
about
data
awareness
and
assets
are
really
our
way
of
helping
out
with
data
management
at
an
operational
level.
A
So
what
am
I
exactly
talking
about
here?
Well,
solids
can
also
look
like
this.
So
one
interesting
fact
here
is
that
the
core
api
for
a
solid
is
actually
it's
a
data
computation
or
just
a
computation
that
yields
a
stream
of
these
structured
events
in
the
dagster
api
that
inform
the
runtime
about
things
that
are
happening
right.
So
what
this
says
is
yielding
an
asset
materialization.
A
A
A
Give
it
a
second
here
I
never
get
tired
of
looking
in
this
gantt
chart.
If
I
have
to
be
honest
here
the
so
we
know
that
this
thing
is
yielded
what
we
call
materialization,
we
actually
our
log
viewer
is
nice
right.
You
can
like
search
for
these
structured
events,
and
it
says
here:
oh
it,
it
emitted
this
asset,
materialization
and
you
might
be
asking
what
does
that
do
well
as
a
result
of
this,
it
actually
shows
up
in
this
thing
called
the
assets
manager.
A
A
There
we
go
so
here
we
can
see
like
hey
here's,
the
yeah,
here's
this
asset,
here's
some
metadata
about
it
here
are
the
last
runs
that
touched
it,
and
this
might
not
seem
that
profound.
But
when
you
start
producing
thousands
and
hundreds
of
thousands
of
assets-
and
you
have
no
idea
where
they
came
from
this
capability
of
being
able
to
link
computations
and
accents
and
and
assets,
is
enormously
powerful.
We,
you
know
the
use
case
you
should
really
think
about.
Is
we
have
users
who
are
allowing
their
analysts
to
self-serve
ops,
wise
their
pipelines?
A
A
So
yeah,
just
to
summarize,
you
know
this
is
kind
of
a
whirlwind
presentation,
thank
you
for
bearing
with
me,
but
you
know
we
went
through
when
we,
you
know,
at
least
in
demo
form
developed
a
pipeline.
We
made
it
more
testable
and
then
we
deployed
to
kubernetes
and
we're
tracking.
We
scheduled
it
and
we're
tracking
what
assets
it's
created
and
so
on
and
so
forth.
A
Now
what
I
showed
you
was
a
very
narrow
example
that
only
involved
pandas
and
kubernetes,
but
we
have
an
entire
world
of
tools
that
we
interact
with
right,
like
we
have
users
who
you
know
are
using
our
integration
with
paper
mills
a
great
effect.
We
have
pi
spark
users,
we
have
people
using
snowflake,
we
have
people
using
dbt,
we
have
people
using
pandas.
A
We
have
folks
who
are
ingesting
data
from
google
sheets
who
are
shooting
out
events
to
slack,
and
you
know
using
all
of
our
features
to
track
these
complicated
interrelations,
and
it's
just
like
it's
too
much
to
go
into
all
that
right
now,
but
this
really
is
turning
into
a
full-fledged
platform,
and
so
with
that
I
will
end
the
presentation
thanks
so
much
for
joining.
You
know
this
is
an
open
source
project,
so
you
can
go
and
find
it
on
github
you
can
pip
install.
A
You
can
come
say
hi
in
our
slack
and
ask
us
more
questions.
We
have
a
totally
new
revamp
doc
site.
If
you
looked
at
it
a
while
ago
and
the
docs
were
a
little
janky.
I
think
we've
made
a
ton
of
progress
on
that
and
yeah
and
we're
about
to
kind
of
announce
a
new
major
release
next
week.
We'll
kind
of
you
know
talk
about
this
thing
as
a
data
orchestrator,
I
have
a
bunch
of
different
content
around
that.
So
I'm
really.