►
Description
In our sixth Dagster Community Meeting, the core team presents upcoming 0.11.0 features, including Dynamic Orchestration, a system to establish Asset Lineage, and a new documentation site.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Dynamic Orchestration and the new `collect` step: 1:48
Dynamic Orchestration Demo: 6:40
Dynamic Orchestration tl;dr: 10:05
An introduction to Asset and Asset Lineage: 11:05
An example pipeline producing assets: 14:11
Inferring Asset Lineage from the Execution Graph: 20:40
Dagster Features using Asset Lineage: 22:31
Dagster Asset Lineage tl;dr: 24:23
New Backfill and Partition Features Demo: 26:03
New documentation infrastructure and design: 33:31
Q&A: 36:56
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
A
A
So
we
actually
have
our
doctor
o110
major
release
coming
out
next
thursday
march
18th.
We're
really
we're
all
really
excited
about
it,
and
today
we
have
four
presentations
about
some
of
the
features
you
can
expect
in
the
release.
So
first
we
have
alex
talking
about
dynamic
orchestration
and
the
new
collect
step.
Then
we'll
have
owen
talking
about
upcoming
plans
for
the
asset
system,
including
asset
lineage.
A
A
So
here's
here's
the
premise:
let's
imagine
that
I'm
somebody
who
has
a
very
special
python
script
that
I
run
it
takes
a
directory
of
files
that
people
send
me
and
produces
profound
truths
about
the
universe.
I
can
do
this
on
any
directory
that
someone
sends
me.
I
like
this.
This
is
my
data
application.
I
care
about
this,
but
I'm
you
know
I'm
just
running
this
script
in
a
shell
every
once
in
a
while,
I
want
to
like
do
something
a
little
more
robust,
so
I
google
around.
A
I
look
on
hacker
news
and
I
find
dexter
and
I
decide
to
check
it
out,
so
I
create
just
basically
an
entry
point
into
my
script.
It's
a
single,
solid
pipeline
and
it's
my
way
to
get
started.
This
is
already
pretty
great
right.
I've
got
the
history
of
every
time.
I've
run
this
now
I
set
it
up
so
that
it
takes
structure
config.
It
does
type
checks
and
I
can
even
like
run
this
on
a
schedule
or
hook
this
up
to
a
sensor
to
kind
of
automate.
A
The
part
of
you
know
ingesting
these
directories
that
people
send
me
so
that
I
can
produce
these
profound
truths
about
the
universe,
but
I
want
more
right.
This
is
like
data
pipelines
and
stuff,
but
I
don't
really
have
a
pipeline
yet.
So
I
look
at
trying
to
like
flush
this
out
a
little
bit,
so
I
structure
the
pipeline
the
same
way
that
my
my
code
is
factored
you
know.
First,
I
get
the
directory
of
files
for
you.
A
You
know
that
I
process
all
the
files
and
then
I
collect
over
that
and
you
know,
synthesize
my
final
report
over
the
whole
directory
and
while
this
feels
a
little
more
organized
operationally,
I
haven't
really
improved
the
situation
right
like
process
files
is
still
the
bulk
of
the
work.
If
one
thing
fails
in
there,
the
whole
step
fails.
I
can
you
know
retry
that
step
or
re.
You
know
resume
failure
for
that
step.
That's
gonna
have
to
do
all
the
files
over
again.
A
A
If
I
was
if
the
directors
were
always
the
same
size,
maybe
this
would
actually
be
pretty
good
and
I
think,
there's
certainly
a
set
of
problems
where
dynamically
creating
the
pipeline,
when
the
fan
out
so
to
speak,
is
known
ahead
of
time.
An
approach
like
this
works
all
right,
but
these
directories
are
any
size
right,
so
this
doesn't
really
work
that
great
and
it's
like
also
pretty
cumbersome
and
like
not
intuitive,
and
if,
when
someone
else
looks
at
this,
it's
kind
of
confusing,
that's
not
I'm
not
stoked
about
it.
A
What
I
actually
want
to
do
is
something
that
looks
more
like
this
right.
I
want
to
for
each
file
that
I
determine
at
runtime,
do
some
operation
and
then
once
those
are
all
done,
do
something
else
pretty
straightforward,
and
this
would
be
great,
and
this
is
where
dynamic
orchestration
comes
in.
So
what
we
have
have
that
partially
released
in
a
0.10
with
dynamic
output
and
coming
soon
in
o
11
collect
is
what
we
call
dynamic
orchestration.
A
This
solution
is
meant
to
do
fine-grained
kind
of
mapping
and
for
reaching
you
know
if
you
need
to
do
some
clever
data,
throughput
on
a
data
frame
and
stuff
like
that
spark,
etc.
Our
system's
designed
for
that
problem,
they're
going
to
do
much
better
than
what
we
do
here,
but
when
you
consider
orchestrating
a
set
of
tasks-
and
you
have
a
problem
with
this
nature,
I
think
this
will
work
for
that.
A
So
we
have
is
a
different
output
type
called
dynamic
output
which
will
cause
the
downstream
dag
from
that
dynamic
output
to
clone
for
each
copy
of
the
dynamic
output
that
is
returned.
Each
dynamic
output
has
a
unique
mapping
key,
which
is
how
we
identify
the
distinct
clones
of
the
subgraph
and
then,
when
that's
all
done
at
some
point,
you
need
to
do
something:
wait
for
it
to
be
done
or
collect
all
the
results,
and
that's
where
this
collects
that
comes
in,
which
closes
the
cloning
property
and
we'll
collect
over
those.
A
A
A
A
A
The
output
from
invoking
a
solid
that
returns
a
dynamic
output
is
a
special
type.
You
can't
just
take
a
dependency
on
this
directly.
You
need
to
unpack
it
so
to
speak
or
explicitly
map
over
it,
and
so
here
we're
going
to
invoke
the
process
file
solid,
for
you
know
for
each
dynamic
directory
that
gets
submitted,
and
then
this
output
type
also
special
to
take
a
dependency
on
it.
We
have
to
do
another
explicit
call
in
this
case.
Collect
is
the
proposed
api,
and
that
lets
us
know
that
we're
going.
B
A
Join
this
information,
so
this
is
a
rough.
You
know
this
is
a
very
simple
example,
but
kind
of
gives
you
a
flavor
of
how
this
will
look
in
practice
and
then
let's
go
see
this
in
action.
So
let
me
move
the
zoom
window
here
all
right,
so
here's
where
we
started
right
here
was
the
version
of
the
pipeline
that
wasn't
that
great
here's,
that
kind
of
crazy
manual
parallelism
thing
and
here's,
here's
the
real
stuff,
here's
the
dynamic
output!
A
So
here
we
have,
you,
know
kind
of
our
first
pass.
Ui,
which
is
some
we
need
to.
This
will
be
improved
on
both
the
collect
and
dynamic
output
side,
but
we
have
some
notion
ahead
of
time
that
we
know
that
this
is
going
to
happen,
so
we
can
mark
the
dependencies
on
both
sides.
Currently,
it's
this
asterisk.
I
think
our
ui's
here
will
improve,
as
we
continue
to
iterate
on
this
and
we
can
go
and
implement
or
go
and
run
this.
Let's
go
ahead
and
kick
this
off.
A
Let's
start
with
the
default
just
to
get
a
taste
of
it,
and
here
we
see
we've
there's
a
directory
that
we
point
out
by
default.
It's
got
three
files
in
it.
We
process
them
all.
This
is
happening
linearly
because
I'm
using
the
in-process
executor
if
I
was
using
multi-process,
these
would
be
in
parallel.
A
A
A
A
These
apis
aren't
going
to
go
away,
but
we
might
change
some
of
the
arguments
or
change
a
little
bit
of
behavior
so
still
want
to
make
it
clear
that
you
know
there
may
be
call
site
updates
that
need
to
happen
as
we
go,
but
this
is
getting
getting
close
to
being
officially
stamped
and
sealed
so
check
it
out.
Give
us
feedback
on
what
you
think
and
if
you
have
any
questions
there
will
be
time
for
questions
at
the
end
of
all
the
presentations.
A
C
C
Okay,
can
people
see
my
screen
cool,
so
yeah?
I'm
owen,
I'm
an
engineer
at
elemental,
and
I
want
to
talk
about
a
much
more
experimental
and
early
stage
sort
of
set
of
features
around
asset,
lineage
and
as
just
sort
of
a
forward.
We
are
looking
for
design,
partners
and
potential
feedback
and
use
cases
on
this.
C
So
if
anything
jumps
out
at
you,
while
I'm
presenting
or
afterwards
feel
free
to
reach
out
we're
totally
happy
to
talk
more
about
this,
so
just
so
we're
all
starting
on
the
same
page,
we
should
define
what
an
asset
is
in
the
in
the
concept
of
dagster,
so
we're
thinking
of
assets
as
these
sort
of
persisted
objects
that
are
relevant
and
persist
outside
the
scope
of
single
pipeline
run.
So
a
classic
example
would
just
be
a
table
in
a
database.
So
imagine
you're
running
a
pipeline.
You
produce
some
data.
C
You
store
that
in
a
table
in
a
database,
that's
your
asset
and
then
maybe
you
later
on
in
the
pipeline.
You
read
from
that
database
again
or
maybe
not
either
way
you
want
to
keep
track
of
that
asset,
but
it's
not
just
tables.
You
know
it
could
be
files
on
s3.
It
could
be
some
like
trained
ml
model
that
you
produce.
C
These
are
all
sorts
of
things
that
are
like
important
after
you
run
your
pipeline
and
one
of
the
things
that
we
strongly
believe
is
that
as
much
as
possible,
we
don't
want
people
to
have
to
think
about
the
pipeline
that
you're
creating.
So
while
the
pipeline
and
all
the
individual
steps
are
really
important
to
the
person
developing
that
pipeline
to
people
outside
of
of
that
team
or
that
group.
It's
not
really
relevant
right.
C
So
we
want
to
enable
people
to
just
think
about
the
assets
as
much
as
possible,
rather
than
thinking
about
all
these
intricacies.
So
then
asset
lineage
would
just
be
the
dependency
relationships
between
these
different
assets.
So
you
can
imagine
being
able
to
say,
like
hey
this
table
in
this
database.
Has
a
data
dependency
on
these
files
in
s3
or,
like
this
ml
model,
was
trained
using
using
data
from
this
table,
etc.
So
these
sorts
of
relationships
and
the
reason
we
think
dagster
is
a
good
place
to
be
putting
this
sort
of
functionality.
C
Is
that
you've
already
sort
of
encoded
a
lot
of
this
information
already
just
by
creating
your
pipeline?
So
you've
said
that
here's
the
solid
it's
going
to
transform
the
data
in
this
way
and
it
has
this
output
and
then
there's
this
other
solid
down
the
line
that
will
take
that
output
as
input
and
so
on
so
you're,
defining
all
these
dependency
relationships
and
then
because
dagster
is
asset
aware,
you
also
in
many
cases,
have
already
defined
assets
and
and
told
dagster
about
these
assets
that
you're
creating.
C
So
right
now
that's
done
through
asset
materializations,
and
it
seems
intuitively
that
you
should
be
able
to
combine
this
sort
of
information.
Hey
I'm
creating
this
asset
at
this
point
in
my
pipeline
with
this
dependency
information,
this
point
in
the
pipeline
depends
on
this
other
point
in
the
pipeline
into
some
sort
of
asset
dependency
graph,
which
then
people
can
operate
over.
So
that's
that's
what
we're
working
with
and
I
think
it'll
make
more
sense
as
we
go
into
a
simple
example.
C
So
this
is
just
a
very
sort
of
dumb
two
pipeline
or
a
two
node
pipeline
in
the
first
step.
It's
downloading
some
user
action
data
and
it
puts
it
into
a
data
frame
not
super
important
where
it
gets
it,
but
then
it
passes
that
on
to
the
next
step
and
again
it's
a
toy
example.
So,
let's
just
imagine
it
has
two
types
of
actions,
comments
and
reviews,
so
it'll
split
those
into
two
different
data
frames
and
then
use
those
as
outputs
and
notice.
C
C
So
let's
imagine
we're
in
some
production,
environment
and
we've
defined
in
I
o
manager
for
all
of
these
outputs
and
for
people
that
aren't
familiar
an
io
manager
just
defines
how
you
take
an
output,
store
it
and
then
load
it
again
as
input
by
default.
This
is
just
done
in
memory
right.
You
pass
an
in-memory
object
from
an
output
to
an
input,
but
that
doesn't
really
work
if
you
have
different
nodes
that
aren't
sharing
sharing
memory
or
if
you
want
to
do
some
like
fancier
stuff
in
between.
C
So
in
this
case
we're
defining
an
I
o
manager
and
its
protocol
is
on
an
output
it'll.
Take
that
data
frame
object
in
memory
and
it'll
save
it
in
a
database.
We'll
say
it
saves
it
in
the
mydb
database
and
then
in
order
to
load
that
input
it'll
read
directly
from
that
table,
so
it'll
go
to
my
database.
It'll
say:
what's
in
this
table,
read
that
into
a
data
frame,
and
then
you
can
do
your
your
further
like
processing
in
the
next
step,
so
we're
working
that
io
manager.
C
Now,
let's
the
the
goal
here
is
to
say:
hey
this
io
manager
is
actually
creating
assets
right.
It's
writing
two
tables.
It's
creating
these
assets.
We
want
to
keep
track
of
them,
so,
in
particular,
we'll
say
that
it's
going
to
write
the
output
from
this
one
to
mydb.raw
actions
and
then
here
it'll
write
the
output
from
comments
to
comments
and
reviews
to
reviews
and
the
whole
end
goal
here,
is
to
be
able
to
say
automatically
without
too
much
user
input
that
hey
this
comments.
C
C
So
let's
look
at
some
code
that
enables
you
to
do
this,
so
this
is
some
hypothetical
code
for
a
solid
and
I'll
walk
through
some
key
things
here.
So
the
first
thing
is
this:
is
our
actions,
data
frame
output
and
we're
defining
this?
I
o
manager
key
on
it.
So
this
is
saying
this
is
the
I
o
manager.
I
want
to
use
for
this
output
and,
in
this
case
we're
using
something
called
mydbio
manager.
C
Remember
the
implementation
can
swap
out
between
test
and
prod
mode,
but
we're
just
thinking
about
the
prod
mode
right
now
and
then
it
has.
This
thing
called
metadata
if
you
haven't
seen
that
before
it's
because
it's
experimental-
but
this
is
basically
just
a
really
simple,
unstructured
dictionary
where
you
can
tell
us
some
attributes
about
this
output.
So
in
this
case
we're
saying,
there's
some
attribute
table
name
and
that's
raw
actions
for
the
specific
output.
C
Then
you're
just
grabbing
some
data
frame
of
data,
somehow
not
super
important
and
then
you're
yielding
your
returning
output.
So
in
this
case
this
is
all
really
standard,
except
for
this
metadata
entries
thing,
which
is
new
and
experimental,
but
this
is
just
a
way
to
attach
some
sort
of
structured
metadata
onto
this
output
object.
So
if
you
know
some
interesting
things
that
you
want
to
store
and
have
dijkstra
represent
you,
then
this
would
be
one
way
to
do
it.
C
So
you
can
just
say:
here's
this,
like
maximum
score
of
this
data
frame
persist
that
around
so
this
is
just
a
little
bit
nicer
than
doing
a
context.log
and
it's
also
useful
for
the
asset
stuff
which
I'll
get
to
so
now.
Let's
talk
about
the
specific
implementation
of
the
mydbio
manager.
So
first
there's
this
handle
output
function
and,
like
I
said,
the
protocol
is
to
just
take
your
data
frame.
This
object
and
write
it
to
a
table
and
the
way
we
know
what
table
to
write
to
is
by
looking
at
the
metadata.
C
So
we
know
to
write
to
the
raw
actions
table.
Then
this
is
really
similar
to
the
metadata
entries
on
the
output
that
I
just
showed,
but
you're
allowed
to
yield
some
arbitrary,
structured
metadata
as
well
describing
what
you
just
did
there.
So
in
this
case
the
database
returned
a
response
and
said:
here's
the
number
of
records
I
wrote
and
you
might
want
to
persist
that
and
have
that
show
up
in
the
log.
So
this
is
one
nice
way
to
do
it
then
load
input,
like
I
said,
you're
just
reading
from
the
table.
C
You
know
what
table
to
read
from
because
of
the
table
name
and
that
will
return
a
data
frame
and
then
finally
get
output
as
a
key.
This
is
sort
of
the
key
to
the
asset,
lineage
part
of
the
story,
so
this
is
where
you
say
when
I
do
my
handle
output.
This
is
what
asset
key
I'm
going
to
be
writing
to
or
materializing.
C
So
in
this
case,
we're
saying
when
I
run
when
I
do
this
protocol,
I
will
be
creating
an
asset
mydb
dot
and
in
this
case,
because
the
table
name
is
raw
actions,
mydb.raw
actions
and
notice
that
this
is
actually
not
taking
the
object
as
a
parameter.
So
this
is
something
this
context.
Thing
is
something
that
we
know
when
we're
planning
the
execution
of
the
pipeline
so
before
we've
actually
run
any
solid
code,
or
anything
like
that.
C
We
materialized
this
asset
that
we
described
here
and
it
has
a
set
of
metadata
entries
and
that
metadata
set
of
metadata
entries
includes
everything
that
you
define
on
the
handle
output,
as
well
as
everything
you
defined
on
the
output,
because
these
things
are
sort
of
interrelated.
So
in
that
way
you
can
get
metadata
from
the
solid
body
to
show
up
on
this
asset
thing
that
you're
creating
okay.
So
if
we
go
back
to
the
original
pipeline
graph,
remember
how
I
said
this
doesn't
include
any
specific
information
about
the
implementation
of
the
I
o
manager.
C
So
we
said
that
hey
this
thing
is
going
to
generate
some
asset.
That's
going
to
be
this
mydb.actions
and
we
can
do
the
same
thing
down
here.
So
we
know
the
set
of
actions
or
set
of
assets
that
will
be
generated
by
running
this
pipeline
and
now
because
we've
layered
this
asset
information.
On
top
of
the
execution
graph,
we
can
start
to
create
lineage
information,
inferences
using
the
dependency
information,
that's
already
encoded
on
this
execution
graph.
C
So,
for
example,
we
can
say
here's
this
comments,
thing
it's
going
to
be
produced
by
the
scion
manager,
which
uses
comments,
df,
which
is
produced
inside
split
actions
which
uses
df
as
input
which
is
loaded
from
this.
I
o
manager,
which
is
associated
with
raw
actions
here,
so
simplifying
that
down.
We
can
just
make
the
inference
that
comments
depends
on
my
db.r
actions,
same
thing
for
reviews.
C
C
So
we
did
that
work.
We
got
this
asset
linear
graph.
What
can
we
do
with
it?
So
this
is
still
an
early
stage
feature,
so
we're
trying
to
be
cautious
and
conservative
and
keep
things
limited
so
right
now
we're
just
surfacing
that
in
the
asset
catalog
on
the
asset
page.
So
if
you
navigate
through
the
asset,
catalog
find
mydatabase.com,
we
can
surface
the
information
that
it
depends
on
my
database.raw
actions.
C
C
So
one
of
the
things
is
that
a
lot
of
people-
you
know
if
you're
migrating
from
old
code-
that's
not
in
daxter
and
you
have
like
or
or
even
your
you
have
dijkstra
code,
but
it's
not
using
io
managers.
Yet
it
can
be
a
little
heavy
way
to
refactor
all
of
your
code
in
order
to
hook
into
this
asset
linear
stuff.
So
we
also
allow
you
to
bind
these
asset
keys
directly
to
an
output
definition.
C
C
Finally,
there's
also,
I
talked
a
bit
about
that
structured
metadata
stuff.
It's
also
useful,
regardless
of
if
you
have
an
acid
or
not
so
in
this
case,
when
you
add
these
to
an
output
regardless
of
asset
or
no,
these
will
show
up
on
the
step
output
event
in
the
event
log,
so
you
can
see.
It'll
show
up
like
that
and
same
thing
with
handle
output
in
an
ion
manager.
These
will
show
up
on
the
handled
output
event,
so
it's
just
a
little
prettier
than
doing
context.log
and
a
little
easier
to
scan
cool.
C
The
execution
graph
already
has
all
this
really
rich
information
about
how
things
how
data
depends
on
other
bits
of
data,
and
we
think
we
can
leverage
that
to
great
effect
and
really
help
people
from
duplicating
all
this
knowledge
that
they
already
kind
of
encoded
by
doing
that
sort
of
transformation
that
I
was
describing
so
finally,
like,
I
said,
still
super
early
stage.
Still
still
experimental,
there's
plenty
of
stuff
that
I
sort
of
skipped
over
in
this
presentation.
C
One
of
those
interesting
like
tidbits
is
there's
this
whole
concept
of
partitioned
assets,
so
these
are
assets
that
maybe
you're
generating
one
day's
worth
of
data
at
a
time
with
every
pipeline
run,
and
you
might
want
to
represent
asset
relationships
differently
than
if
you're,
just
creating
a
whole
new
thing.
Every
time
you
run
the
pipeline
and
we
do
have
some
affordances
for
that
going
out
in
0110,
but
it's
definitely
something
we
want
to
think
more
about,
but
yeah
in
conclusion,
something
we're
super
excited
about.
I
hope
you
are
too
feel
free
to
reach
out.
D
D
D
So
one
of
the
sort
of
the
important
things
about
petition
pipelines,
the
petition
set
abstraction
lets
you
work
with,
is
that
you
usually
care
about
having
a
successful
run
for
every
partition.
So
if
a
run
for
one
of
the
partitions
fails,
that
means
the
table
is
missing
data
for
that
partition,
and
you
want
to
be
able
to
detect
that
and
fill
in
those
failed
runs.
D
D
There
are
a
few
reasons
you
might
want
to
backfill
a
pipeline,
so
maybe
an
upstream
data
source
has
changed.
Maybe
the
code
that
defines
the
table
changed.
Maybe
the
table
was
accidentally
deleted.
You
need
to
regenerate
it,
or
maybe
some
recent
ones
failed
and
you
want
to
fill
those
in
up
until
now.
If
you
wanted
to
use
these
dijkstra
features,
you
might
have
run
into
a
couple
issues.
D
Many
of
these
changes
have
landed
in
oceano,
so
you
might
have
already
had
the
chance
to
play
around
with
some
of
them.
We
waited
to
advertise
them
while
we
stabilized
them
adding
a
button
to
the
product
that
lets
someone
kick
off.
A
thousand
runs
at
once
adds
a
whole
set
of
scalability
concerns
that
we
had
not
yet
encountered.
So
we
wanted
to
finish
ironing
those
out
which
we've
done
with
this
release.
D
So
here's
our
pipeline.
This
is
a
toy
pipeline
that
might
exist
inside
of
a
company
like
youtube,
so
it
ingested
on
video
views
and
trains,
a
model
to
recommend
videos
to
users
at
the
end
of
each
day.
We
process
the
data
for
that
day,
so
this
means
that
the
pipeline
has
a
partition
for
each
date,
as
we
talked
about
before.
D
D
What
we're
seeing
here
corresponds
to
the
partition
set
that
we've
defined
for
this
pipeline,
so
it
centers
on
this
matrix
of
solids
and
partitions.
So
you
can
see
in
the
middle
of
the
screen.
Each
row
in
the
matrix
corresponds
to
one
of
the
solves
in
the
pipeline
and
each
column
corresponds
to
one
of
the
partitions.
So
you
can
see
here
that
each
column
is
labeled
with
a
date
which
is
the
name
of
that
partition.
D
If
there's
a
red
box,
it
means
that
the
latest
run
for
that
solid
and
that
partition
failed.
If
there's
a
white
box,
it
means
we
never
even
got
to
run
that
solver
for
that
partition
and
a
green
box
means
that
the
latest
run
for
that
solid
partition
combination
was
a
success,
so
we
can
click
on
any
of
these
boxes
to
see
the
let's
look
at
one
of
these
over
here.
B
D
The
history
of
all
runs
for
a
particular
step
in
partition.
So
while,
like
the
state
of
that
the
the
shading
of
the
box
indicates
the
latest
run,
we
can
actually
still
go
back
and
see
any
historical
run
which
might
be
useful
for
debugging
in
this
case,
I'm
interested
in
what
caused
the
failure
in
this
particular
step
partition.
So
I'm
going
to
click
on
the
box
and
then
I
can
click
on
the
run
id
to
actually
go
to
the
logs
for
that
pipeline
and
notice.
The
cause
of
the.
D
Failure
because
we
have
these
fellow
runs,
I'm
interested
in
launching
some
new
runs
to
get
the
job
done.
I
think
the
bad
ones
are
over
here,
so
let's
focus
on
the
partition
for
january
22nd
and
january
20
is
that
yeah
january
22nd
and
january
24th?
What
I'm
going
to
do
is
I'm
going
to
launch
a
backfill
to
fill
those
in
so
to
do
that,
I
click
on
this
launch
backfill
button,
which
takes
me
to
a
screen
where
I
get
to
select
the
partitions
that
I
want
to
launch.
D
D
The
filter,
next
to
the
run
matrix
text
up
here
indicates
this
that
we're
looking
at
this
particular
backfill,
the
page
periodically
refreshes
the
capture
state
of
the
backfill
as
it
unfolds.
So,
as
you
can
see
in
this
case,
the
backfill
is
actually
completed
already,
and
luckily
we
had
successful
executions
for
all
of
the
steps.
If
we
launched
a
larger
backfill,
which
maybe
let's
do
it
just
for
fun,
we
can
do
it
over
an
entire
set
of
partitions.
D
A
A
A
So
for
our
11
0
release,
we
focused
a
ton
of
energy
into
creating
a
improved
documentation.
Experience
from
everything
we
learned.
We
improved
the
documentation,
structure
and
information
hierarchy
and
even
refreshed
the
underlying
infra
to
make
it
easier
for
both
the
team
and
external
contributors
to
write
docs.
A
You
can
check
out
the
early
preview
of
this
url
and
it
will
officially
go
live
with
the
o110
release
next
week.
So
here's
a
new
design
what's
actually
new.
So
first
up
we
have
consistency.
We
found
that
it
was
a
bit
hard
to
navigate
a
bunch
of
our
pages,
because
each
page
had
its
own
unique
style
and
format.
This
meant,
you
often
had
to
read
the
whole
page
to
extract
the
information
needed
so
to
solve
this
problem.
We've
created
a
standardized
template
for
all
of
our
pages,
so
each
page
has
consistent
information.
E
A
A
So,
for
example,
on
the
pipelines
page,
we
have
a
bunch
of
common
pipeline
layouts,
so
you
can
just
copy
and
paste
the
pipeline
definitions
for
them
and
finally,
we
have
search
and
seo
one
of
the
biggest
complaints
that
we
had
was
how
hard
it
was
to
search
for
information
using
the
search
bar.
We
actually
used
to
have
two
search
indexes,
one
for
the
docs
content
and
one
for
the
api
docs.
It
was
never
clear
which
one
to
use
so
also
finding
new
information
through
google
didn't
give
back
great
results.
A
A
We've
also
got
a
ton
of
interest
in
contributing
back
to
docs.
So
next
I'll
talk
about
different
ways.
You
can
do
so
number
one
way
is
to
give
feedback.
We
absolutely
love
feedback
and
take
it
very
seriously.
There's
two
ways
you
can
do
so
going
forward.
One
is
you
can
refor
report
feedback
directly
on
the
docs
site?
There's
a
share
feedback
right
on
the
top.
You
can
just
click
it
and
it'll
come
to
us.
Next
is
the
edit
page
on
github
button.
You
can
just
click
that
it'll.
A
Take
you
to
a
page
where
you
can
edit
the
markdown
online
and
you
can
send
a
pr
away,
we'll
be
sure
to
review
it
quickly
and
get
it
back
up
on
the
live
site.
Next,
we
have
issues.
If
you
see
anything
missing,
please
don't
hesitate
to
open
up
an
issue.
We
also
have
a
new
issue:
template
specifically
docs,
so
you
just
click
it
get
started
and
then
fill
it
out
and
we'll
make
sure
that
the
content
that
is
missing
will
be
on
the
doc
site
and,
finally,
we're
writing
new
docs.
A
If
you're
feeling
especially
motivated
and
want
to
write
content
for
a
new
doc
site,
please
reach
out.
We
have
a
pretty
streamlined
process
internally
now
involving
templates
and
topics
list.
So,
if
you'd
like
to
write
or
if
you
have
any
special
unique
topics
that
you
feel
passionate
about,
reach
us
to
reach
us
to
us,
reach
out
to
us
on
slack
and
we
can
help
guide
you
and
that's
all
we
have
in
the
dark
site
and
we're
super
excited
to
show
you
next
week
and
that's
all
we
have
for
presentations.
E
D
Yeah,
this
is
something
that
we've
talked
about
a
decent
bit
and
I
think
the
workflows
that
are
required
right
now
are
a
little
bit
more
involved.
So
I
think
you
have
two
options.
One
would
be
to
use
the
sort
of
running
queueing
capabilities
that
we
have
in
that
data
server,
so
you
can
basically
set
that
there's.
Only
one
concurrent
run
allowed
in
that
particular
queue
and
then
submit
backfill
runs
with
the
tag
that
sends
them
to
that
queue.
D
Another
one
would
be
to
use
a
sensor
with
a
sensor.
You
can
basically
launch
sets
of
runs
which,
with
whatever
complex
or
interesting
dependency
logic
that
you
have,
we
acknowledge
this
is
a
more
involved
workflow,
so
it
you
know,
will
often
involve
pushing
code
instead
of
just
clicking
a
button
on
the
ui,
so
we're
thinking
about
ways
to
make
that
more
streamlined.
E
Cool
and
then
the
follow-up
peter
did
a
follow-up
in
the
chat
sandy
to
your
question.
I
was
thinking
about
pushing
dashboard
and
their
dependency
on
the
data
set,
so
encoding
encoding
a
dependency
between
a
dashboard
which
is
kind
of
just
the
external
arbitrary
entity
and
a
data
set
that
would
be
computed
by
dagster
got
it.
E
Please
forgive
that
pronunciation
of
your
username.
How
do
asset
metadata
events
interact
with
asset
materializations?
It
seems
like
new
feature
in
the
I
o
managers
depends
on
inputs
and
outputs,
but
the
materializations
are
relatively
free
form.
So,
yes,
I
will.
Let
owen
weigh
in
that,
because,
yes,
we,
this
is
a
topic
of
frequent
discussion.
C
So
yeah
yeah!
No,
that's
that's
a
super
good
question.
So
right
now
we
think
about
attaching
an
asset
key
to
an
input
or
an
output
as
the
way
to
propagate
that
information
down
the
line.
So
the
way
we
have
it
right
now,
if
you
are
emitting
asset
materializations
like
manually
in
the
body
of
a
solid
those,
can
inherent
and
inherit
the
dependencies
from
those
inputs
that
have
been
like
propagated
down.
But
they
don't
that
information
won't
go
anywhere
else
outside
of
the
solid.
C
E
Yeah-
and
I
want
to
emphasize
this-
is
all
open
to
feedback.
I
think
both
you
know
peter
and
alokian
have
asked
excellent
questions
and
feel
free
to,
and
please
engage
with
us
on
this
subject,
because
these
are
the
type
of
issues
we
want
to
suss
out.
Why
we
even
talk
about
this
at
the
community
meeting.
We
want
feedback
from
you
all
hugo
asks.
I
remember
seeing
somewhere
on
the
dagster
slack,
that
there
was
some
work
being
done,
how
user
code
is
deployed
to
kate's,
perhaps
involving
helm,
sub
charts.
F
A
Yeah,
the
current
the
current
restriction
is
that
you
can
only
be
you
can
only
be
in
like
one
cloning
subgraph
at
a
time,
so
you
can
only
be
downstream
of
one
dynamic
output
at
any
given
time.
I
think
that's
a
restriction,
we're
going
to
lift
eventually,
but
it's
a
little
trickier
earlier
to
be
specific,
it
was
tricky
in
the
initial
implementation
that
got
landed
in
0.10.
A
Some
of
the
revisions
that
are
happening
now
to
make
collect
work
should
make
that
easier,
but
I,
but
there's
still
a
little
bit
to
solve
there
to
support
being
downstream
of
multiple
so
that
that's
the
current
state.
I
think
o110
you'll
still
only
be
able
to
do
one
be
downstream
with
one
dynamic
output
at
a
time
and
in
the
future
that
is
likely
to
be
removed
as
a
restriction
thanks
and
yeah.
Once
once
we
make
it
work.
E
Okay,
are
there
any
special
interest
groups
for
devops?
I
assume
you
mean,
like
a
slack
channel
rather
than
a
group
to
lobby
congress
about
devops
issues.
The
I
don't
actually
know
feel
free
blaze
to
reach
out.
We
don't
really
have
a
process
for
community
requested
channels
cat.
Maybe
if
you
could
reach
out
to
blaze
about
this
issue,
and
we
can
talk
about
that
or
if
anyone
feels
free
to
weigh
in
there's
some
channel.
I
don't
know
about
there's
a
kate's
deployment
channel,
but.
E
A
E
Okay,
if
there's
no
further
questions,
I
think
we
can
end
this
about
10
minutes
early.
I'd
like
to
personally
thank
shashank
owen,
alex
and
sandy
for
putting
this
much
effort
and
giving
such
excellent
presentations,
and
I
just
want
to
emphasize
that
we
really
try
to
talk
about
features
in
their
early
stages
to
the
community,
because
we
are
very
interested
in
feedback
on
on
these
things.