►
Description
In our eleventh Dagster Community Meeting, we are happy to introduce Mollie Pettit, who is our new Developer Relations Lead at Elementl! We also hear from David Laing, a Staff Data Engineer at VMware on monitoring-driven asset development. Owen from the Elementl team covered recent work on software-defined assets and dbt.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
0.14.0 Update: 1:55
Monitoring-driven Asset Development - David Laing from VMware: 6:08
Software-Defined Assets & the Modern Data Stack: 29:38
Q&A: 44:42
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
A
Right
welcome
everybody
to
dagster's
11th,
roughly
every
two
monthly
community
meeting
wave.
You
can
hear
me.
Can
you
hear
me?
Yes,
thank
you.
We've
got
a
pretty
packed
agenda
for
you
today
at
the
start
of
the
agenda.
I
want
to
introduce
molly,
who
is
our
new
developer
relations
lead
for
dagstar
molly?
Will
you
hop
on
and
say
a
couple
words.
B
Yeah
hi
yeah,
so
I
just
joined
elemental
first
day,
was
yesterday,
so
very
fresh
I'll,
be
heading
up
the
developer
relations
team.
Previously
I
worked
as
developer
relations
manager
at
observable,
as
well
as
a
database
developer
before
that
a
data
visualization
engineer
at
netflix-
and
I
also
am
the
co-founder
and
events
director
of
the
database
society,
so
launched
a
big
conference
with
them.
Yeah
excited
to
be
here.
A
A
Oh
and
I'm
sandy,
for
those
who
don't
know
me,
I
am
an
engineer
at
elemental
after
that
david
lang
from
vmware
is
going
to
talk
about
monitoring,
driven
asset
development
and
demo.
What
he's
been
building
and
working
on
and
then
owen
from
elemental
is
going
to
talk
about
software-defined
assets
and
the
modern
data
stack.
A
All
right,
0-14-0,
our
target
release
date
is
less
than
two
weeks
from
today,
the
thursday.
After
this
coming
thursday,
it
includes
honestly
a
ton
of
stuff,
so
we
talked
about
software-defined
assets
in
our
last
community
meeting.
It
was
something
like
a
demo-able
prototype
and,
if
you're
not
familiar
with
software-defined
assets,
owen's
gonna
give
a
brief
background
introduction
in
his
talk.
A
Since
then,
since
our
last
community
meeting,
we've
hardened
the
apis
considerably
and
introduced
a
rich
new
feature
set,
including
partitioned
assets,
asset
metadata,
asset
observations,
tight
integrations
with
modern
data
stack
tools
like
dbt,
5tran
and
airbyte,
and
in
o140.
We
believe
the
apis
are
ready
for
prime
time
and
encourage
you
to
try
them
out.
A
Dagit
is
going
to
have
a
new
home
page,
this
home
page,
which
we've
been
affectionately,
calling
the
factory
floor
view
or
the
job
status
view
I'll,
give
you
an
overview
of
all
the
jobs
that
you
have
across
your
entire
instance,
including
a
fancy
gantt
chart
near
the
top,
which
helps
you
understand
when
these
jobs
are
executing
over
time.
Right
now,
you
can
turn
this
on
as
an
experimental
flag
and,
unfortunately,
we're
gonna
make
it
daggett's
default
homepage.
A
A
A
Daggett
is
going
to
be
able
to
render
table
schema,
so
you'll
be
able
to
set
provide
a
set
of
column
names,
a
set
of
column
types,
column,
constraints
for
a
dagster
type
or
a
software
defined
asset
as
a
standard
kind
of
metadata
entry,
similar
to
how
you
can
include
markdown
or
text
or
or
a
float,
and
this
is
going
to
be
rendered
really
beautifully
in
a
bunch
of
different
places.
Inside
daggett.
A
C
So
so
I
think
that
in
the
data
engineering
space,
we're
kind
of
watching
a
set
of
paradigm
shifts
unfold
if
you're.
Like
me,
you
probably
started
with
a
kind
of
project
mindset,
so
you
were
thinking
about
projects
where
you
were
delivering
some
kind
of
data
artifact,
maybe
a
report
to
a
stakeholder.
C
There
was
always
they
wanted
some
some
level
of
some
number
of
features
by
some
specific
date,
and
we
all
know
how
that
goes
and
success
was
that
your
stakeholder
accepted
the
data.
You
know
artifact
that
that
you
gave
them,
and
then
you
moved
on
to
the
next
project,
then
I
think
as
the
ultimate
as
the
automation
tools
started
to
mature.
C
We
were
just
doing
more
automation
when
delivering
that
project,
so
we're
still
trying
to
deliver
something
automatically,
but
by
a
certain
date,
success
moved
from
acceptance
by
the
stakeholder
to
your
pipeline,
succeeding
or
not
succeeding,
and
when
you
were
finished,
you
moved
on
to
making
the
next
pipeline-
and
you
know
replay
that
cycle
a
couple
of
times
and
I'm
sure
we've
all
had
the
experience
where
you
have
a
whole
bunch
of
pipelines.
Some
of
them
are
failing
and
no
one's
quite
sure
who
cares
whether
the
failure
is
a
problem
or
not.
C
We
also
shift
instead
of
having
automation,
that's
just
used
to
create
the
asset.
We
also
have
automation
that
validates
that
that
the
asset
is
available
and
has
the
quality
criteria
that
our
stakeholders
expect,
and
only
once
we
have
all
of
that
in
place.
Do
we
do
the
familiar
work
of
creating
pipelines
to
create
the
asset,
and
then
we
have
a
much
more
iterative
approach
where
we
look
at
well.
Are
our
stakeholders
asking
for
more
data
in
this
asset?
C
C
C
We
also
can
start
by
writing.
This
is
the
kind
of
monitoring
driven
start
by
by
writing
the
monitors
that
validate
that
our
asset
is
some
level
of
quality
and
we
expect
it
to
fail
in
the
beginning,
but
now
we
have
something
that
we
know
once
we've
implemented
some
logic.
We
know
when
we're
finished,
because
now
these
monitors
are
succeeding.
C
It's
also
a
great
way
to
sort
of
demonstrate
to
our
stakeholders
why
our
data
assets
are
trustworthy,
we
can
sort
of
you
know,
show
them.
This
is
the
automation
that
we
have
that
validates
that
the
monitor
that
the
data
assets
are
good,
and
this
is
how
frequently
they're
good
and
how
frequently
they're
bad,
and
only
at
that
stage
do
we
get
on
to
actually
implementing
the
asset
creation
logic.
So
this
is
the
traditional.
C
Now
we
think
of
now
we
iterate
on
that
on
that
asset
we
get
feedback
from
the
stakeholders
from
the
automation
and
we
use
that
to
prioritize
adding
more
data,
perhaps
improving
the
quality
and
so
forth,
and
so
success
in
this
paradigm
looks
like
a
team
owning
some
data
products
and
long-lived
data
assets
with
good
quality
and
monitoring.
And
you
always
know
you
know
what's
important
to
the
team,
where
you
should
focus
your
your
your
attention.
C
C
C
C
Come
on
there
we
go
mouse
control
there.
We
go
all
right.
So
look
at
this,
so
we
want
to
kind
of
publish
this.
This
very
basic
asset
schema
with
no
actual
data
in
it
yet
into
our
asset,
catalog
and
get
our
our
stakeholders
engaged
about.
You
know
is
this
the
thing
that
they
were
expecting.
Is
this
the
definition
when
you
know
when
they
talk
about
a
customer
being
active,
for
example,
you
know:
is
this
data
that's
presenting
helpful
for
them?
C
C
They
can
come
back
to
this
asset
catalog
page
to
to
see
this.
You
know
the
state
of
the
data
asset,
so
here
we
go
so
we
filled
in
in
one
partition,
and
here
we
go.
We
can
see
that
you
know.
One
of
the
six
partitions
is
now
filled
with
some
test
data,
but
more
importantly
than
that,
we
can
also
start
showing
them
how
we're
going
to
validate
the
quality
of
of
this
data,
and
so
in
this
case
the
implementation
that
I
did
was
the
simplest
possible
thing.
C
So
I
can
write
a
dagster
pipeline
that
is
focused
not
so
much
on
the
creation
of
the
data,
but
on
the
validation
of
the
data.
That's
in
the
mart,
and
so
I
can
have
a
set
of
graphs
around
validating
specific
data
assets
and
within
those
they
can
have
a
particular
set
of
you
know,
conditions
so
should
conform
to
some
schema.
It
should
contain
perhaps
some
current
date
and
then
the
actual
implementation
of
these.
C
C
Oh
and
then-
and
then
finally,
I
mentioned
that
we
can-
we
can
use
that.
You
know
this
same
the
same
job
that
we
ran
manually
over
here
and
that
we
can,
you
know,
use
dexter's
scheduling,
capabilities
to
run
for
us
automatically.
We
can
run
that
same
job,
but
kind
of
in
a
pi
test,
pi
test
context
to
help
drive
our
our
actual
implementation
development.
C
C
C
All
right,
so
that's
phase
one
just
to
recap.
So
we
talked
about
publishing
the
asset
as
one
of
the
first
things
that
you
do
in
the
asset
catalog,
and
what
we're
trying
to
do
is
get
early
feedback
from
our
stakeholders
and
trigger
the
kind
of
questions
that
that
you
know
sort
of
come
up.
Can
you
actually
access
this
location
in
the
data?
Mart
it's
a
big
one.
I
bumped
into
a
lot
the
perennial.
C
Oh,
what
do
you
mean
when
you
say
a
customer,
especially
if
you're
in
a
corporate
environment
you
know,
there's
a
customer,
the
subsidiary
or
the
parent
company
or
the
one
that
pays
the
bills,
and
you
can
have
conversations
about
about
the
required
grain
we've
assumed
daily
here.
Maybe
actually,
this
is
something
that
needs
to
be
done
hourly
or
maybe
it's
much
slower.
Maybe
we
just
need
something
quarterly,
but
having
those
conversations
early
on
can
really
impact
how
you
go
about
implementing
things.
C
Then
we
talked
about
writing
a
validation
job
where
each
operation
is
a
check
for
the
data
quality
as
a
person
that
makes
data
and
using
the
asset
observation
type
to
kind
of
link
that
metadata
from
this
validation
run
into
your
into
the
asset,
catalog
and
then
finally,
using
the
the
same
validation
pipeline,
but
run
inside
a
unit
test
context
to
service
your
acceptance
test,
while
you're
implementing
the
features
all
right
phase.
Two.
C
Now
that
you
have
you
know,
set
the
stage
now,
it's
time
to
get
on
to
the
bit
that
used
to
be
step,
one
which
is
actually
developing
the
logic
to
put
the
put
the
asset
in
in
the
right
place.
So
what
I
think
we
should
be
doing
under
this
paradigm
is
to
first
deploy
our
monitoring
and
see
that
it's
failing
and
failing
in
a
way
that
that
we
expect
you
know,
are
the
error
messages
we
see
in
the
monitoring
helpful.
C
Do
they
tell
us
which
environment,
for
example,
the
the
failure
is
occurring
on,
and
then
we
just
need
to
implement
the
simplest
first
thing
to
to
sort
of
pass.
The
validation
that
that
we
have
running
simpler
is
better
is
always
my
approach,
and
maybe
we
begin
just
by
having
some
some
manual
steps.
C
Maybe
we
don't
have
a
fully
automated
process
in
the
beginning,
because
we're
still
learning
about
the
problem
that
we
want
to
automate,
you
can't
automate
something
you
you
don't
understand
and
you
definitely
don't
want
to
be
doing
things
like
worrying
about
performance,
optimizations
or
any
of
those.
Those
are
you
know
things
once
this
asset
is
used
and
important.
That's
the
time
to
invest
in
that,
not
at
this
early
stage
and
then
finally,
we
can
use
the
monitoring-
and
you
know
the
associated
acceptance
test
to
tell
us
when
our
implementation
is
good
enough.
C
Looking
at
the
time
I'm
going
to
skip
over
the
the
phase
2
demo,
but
let
me
just
show
you
sort
of
by
by
the
recap,
so
you
would
have
deployed
your
monitoring
and
it
would
have
failed
for
some
amount
of
time
and
then,
finally,
once
you
deployed
the
your
implementation
that
worked
now,
you
would
have
started
to
have
your
monitoring
started.
Starting
to
succeed
your
mind,
your
your
first
implementation
can
be
really
simple.
C
And
then
phase
three
is
to
is
to
iterate
on
your
solution
and
to
prioritize
where
you,
where
you
focus
based
on
the
feedback
that
you're
getting
from
your
stakeholders,
are
they
asking
for
more
data?
Are
they
asking
for
a
different
schema?
Is
the
update
schedule?
Maybe
they
want
to
change
that
or
your
your
automated
asset
monitoring?
C
C
C
Some
further
ideas,
you
could
add
notifications
based
on
the
monitoring
job
to
let
you
know
about
problems
before
your
stakeholders
notice
them.
There's.
No
reason
that
your
monitoring
job
should
only
be
limited
to
monitoring
your
assets.
You
know,
as
we
saw
the
monitoring
operations
are
just
a
python
function.
You
can
do
whatever
you
like
in
them.
Maybe
there's
a
set
of
upstream
assets
that
you
depend
on
that
would
be
helpful
to
to
monitor,
or
maybe
you
want
to
monitor
something
downstream.
C
C
C
All
right
so
recapping,
so
we
talked
about
this
whole
idea
of
monitoring
driven
development,
is,
is
part
of
this
paradigm
shift
that
we're
seeing
and
that
dexter
is
helping
to
drive
this
move
away
from
pipelines
and
towards
assets
and
the
approach
that
you
follow.
Is
you
start
by
publishing
information
about
the
asset
upfront?
So
you
can
get
stakeholder
feedback,
nice
and
early?
You
can
try
drive
that
in
engagement
and
you
can
you
can
make
changes
when
it's
cheapest
to
make
changes.
C
Where
you
know
based
on
feedback,
we
add
more
data,
or
maybe
we
improve
the
quality
or
the
reliability
of
something
and
success
in
this
new
paradigm
looks
like
a
a
team
that
has
a
a
bunch
of
long-lived
data
assets
with
a
well-known
quality,
some
kind
of
sli
driven
by
our
monitoring.
And
then
we
look
we're
thinking
about
selling
the
same
assets
to
additional
stakeholders
rather
than
being
on
the
perpetual
treadmill
of
just
doing
the
next
project.
C
A
D
All
right,
well,
once
the
pinwheel
death
goes
away,
maybe
I'll
be
able
to
present
a
little
better
but
yeah
thanks
thanks
david
for
introducing
some
of
these
concepts.
I'm
still
gonna
go
over
and
do
sort
of
a
high
level
recap
of
some
of
the
stuff
david
was
talking
about
before
I
jump
into
a
demo,
and
this
demo
is
going
to
focus
on
how
the
software
defined
assets
and
these
concepts
relate
to
some
more
external
tools.
D
So,
whereas
david's
content
was
focused
on
how
you
can
create
these
assets
using
python,
there
are
also
external
tools
that
you
might
want
to
interface
in
which
you
define
your
own
assets.
So
in
particular,
this
demo
is
going
to
go
over
airbite
and
dbt,
but
there
are
a
bunch
of
other
tools
that
these
same
concepts
would
apply
to
all
right,
so
yeah.
The
high-level
goal
here
is
to
expose
interfaces
that
allow
you
to
think
more
about
the
assets
that
you
want
to
produce,
rather
than
the
tasks
that
you
want
to
run.
D
So,
as
david
showed,
assets
are
essentially
just
products
that
your
data
platform
is
producing
any
sort
of
persistent
thing
that
you
might
want
to
refer
back
to
after
a
job
completes,
so
the
classic
example
is,
of
course,
a
table
in
a
database,
but
this
can
also
be
a
little
bit
more
general,
something
like
a
machine
learning
model
or
a
report,
a
dashboard
that
sort
of
thing-
and
these
are
often
sort
of
the
entire
point
of
writing
a
job.
You
don't
necessarily
care
about
arranging
tasks
in
a
particular
way
just
for
fun.
D
However,
traditionally
working
with
assets
can
be
kind
of
difficult.
Deploying
changes
can
feel
dangerous
or
irreversible.
After
you
push
a
change,
it
can
be
hard
to
know
what
other
downstream
assets
need
to
be
updated
as
well.
If
you
push
code
that
has
a
bug
in
it
or
otherwise,
mysterious
error
fixing.
This
may
involve
tedious
backfills
that
may
have
to
be
kicked
off
manually
or
even
have
unknown
knock-on
effects.
D
Performing
operations
often
requires
mentally
translating
between
assets
and
tasks.
Understanding
how
to
refresh
a
particular
asset
requires
that
you
know
what
tasks
produce
that
asset
and
what
tasks
are
relevant
to
that
downstream
task.
Understanding
when
an
asset
will
be
updated,
similarly
requires
understanding
when
a
task
will
be
updated
and
then
translating
that
back
to
the
asset
layer.
D
All
these
issues
culminate
in
a
problem
where
assets
are
difficult
to
trust,
knowing
if
something
is
up
to
date
or
had
was
created
with
the
right
version
of
code,
is
often
challenging
or
even
impossible.
Given
a
particular
toolset,
we
believe
that
orchestrators
are
in
a
unique
position
to
help
with
these
issues.
As
long
as
the
orchestrator
can
know
what
asset
a
task
will
produce,
they
can
know
if
an
asset
is
up
to
date,
simply
by
inspecting
when
a
task
was
run.
D
D
However,
traditional
orchestrators
often
are
completely
ignorant
of
the
assets
that
a
task
will
produce,
so
they
may
know
all
these
different
dependency
links
between
different
tasks.
They
may
know
when
a
task
is
set
to
run,
but
they
don't
have
any
insight
onto
the
actual
point
of
those
tasks.
We
believe
that
this
is
a
huge
opportunity
for
orchestrators
to
actually
expose
the
underlying
purpose
of
a
job.
D
So
the
high
level
goal
is
to
take
data
from
some
online
transactional
database,
put
it
in
a
data
warehouse
and
then
once
it's
in
the
warehouse
do
transformation
on
it
and
the
tool
we're
going
to
use
to
move
data
from
point.
A
to
point
b
is
airbite,
which
is
akin
to
5tran,
if
you're
not
familiar
with
airbyte
and
the
tool
we're
going
to
use
to
transform
data
once
it's
in
the
warehouse
is
going
to
be
dbt.
D
D
It's
a
postgres
database
that
in
demo
land
is
some
online
transactional
thing
in
reality
to
peel
back
the
curtain.
This
is
just
some
postgres
database
running
on
my
local
machine.
You
connect
sources
with
destinations.
In
this
case
I
have
a
single
destination
called
postgres
replica
in
real
life.
This
would
probably
be
a
more
a
database,
that's
more
suited
to
analytical
workloads
like
snowflake,
but
again
this
is
just
another,
a
different
database
running
on
my
local
machine.
D
D
If
we
look
into
this
particular
connection
and
at
the
settings
we
can
see
here
that
there
are
three
tables
that
I've
selected
to
be
replicated
over
so
event,
logs
organizations
and
runs,
and
if
you
think
about
what
I've
done
here,
I've
kind
of
just
defined
three
assets.
So
I've
defined
these
three
data
assets
that
I
expect
airbite
to
manage.
For
me,
I
expect
it
to
be
able
to
take
data
from
some
place
and
create
this
data
asset
in
my
replica
database
event
logs
and
then
this
one
for
organizations
etc.
D
D
Gross
simplification
is
that
it's
a
sql
templating
tool
that
allows
you
to
create
sql
models
that
do
transformations,
which
reference
other
sql
models.
So
inside
a
single
dbt
project,
you
can
get
a
whole
web
of
interconnected
models.
So
I've
done
that
here,
and
this
is
what
that
looks
like
in
dbt
land.
So
this
is
all
just
using
dbt
as
as
usual,
and
you
can
see
these
three
green
boxes
are
the
sources
that
we're
using.
D
So
these
are
the
tables
in
the
replicated
replicated.
Database
organizations
runs
event
logs
and
then
information
from
these
tables
is
queried,
joined,
meshed
together
into
a
set
of
once
again
data
assets.
So
these
are
all
tables
in
a
database
that
an
analyst
might
be
interested
in
or
curious
about
or
want
to
refer
back
to,
and
now,
if
we
go
into
dagster
and
look
at
a
sort
of
traditional
task
oriented
view
of
what's
going
on
here,
we
see
that
there
are
just
two
steps.
D
D
D
Then,
if
I
click
re-materialize,
all
this
will
kick
off
a
run
of
this
job
and
you
can
see
once
again.
The
job
just
has
two
steps,
and
this
will
take
a
little
bit
to
run.
But
if
I
come
back
to
the
job
we
can
see,
we
already
get
some
operational
information
about
what's
going
on.
So
while
this
run
is
kicking
off,
we
can
see
right
here
on
this
asset
view
that
a
run
is
currently
rematerializing,
this
particular
asset,
so
we
get
the
spinner.
D
D
Similarly,
this
dbt
step
now
knows
that
a
run
has
started
that
will
soon
re-materialize
this
asset.
This
step
isn't
running
yet,
which
is
why
it
doesn't
have
the
spinner,
and
you
can
imagine,
while
this
is
already
useful
for
a
quick
running
task,
you
can
imagine
a
world
in
which
these
sinks
take
a
significantly
longer
amount
of
time.
D
30
minutes
an
hour-
and
this
gives
you
a
really
clear
view
into
the
exact
state
of
everything
that's
happening,
so
the
airbite
sync
just
completed
and
now
dagster
knows
that
hey
the
upstream
of
this
asset,
as
well
as
everything
up
here,
has
updated
more
recently
than
this
asset
has.
So
that
means
that
there's
probably
new
data
there,
that
this
model
has
not
taken
into
account,
so
we
get
this
little
upstream
changed
flag
and
that'll
go
away
as
soon
as
this
dbt
run
completes.
D
D
For
example,
if
you're
curious
about
the,
if
you're
working
on
this
steps
annotated
model,
you
know
for
a
fact
that
daily
step
stats
depends
on
it
and
therefore,
if
you
change
it
daily
substance
might
need
to
change.
This
is
something
that
dbt
can
already
tell
you
in
the
tool,
but
what
dbt
can't
tell
you
is
that
if
you
remove
a
column
from
event
logs,
then
you
probably
want
to
also
update
the
steps
annotated
model
to
take
that
into
account.
You
get
this
very
quick,
cross-dependency
or
cross-technology
dependency
structure.
D
You
can
also
imagine
if
you
defined
another
python
asset,
just
some
random
python
code
that
consumes
this
event
logs.
Maybe
it
does
some
bot
filtering
or
something.
This
would
also
show
up
here
so
again
if
you're
working
on
this
event
logs
thing,
it's
very
quick
to
determine
what
effect
changing
this
will
have.
D
Similarly,
it's
not
always
the
case
that
this
is
the
view
you're
interested
in
sometimes,
if
you're
an
analyst
say-
and
you
don't
particularly
know
that
a
particular
asset
lives
in
this
job.
You
might
just
want
to
search
directly
for
something
that
you
work
with
every
day
and
know
about
so,
for
example,
I
might
be
an
analyst
that
works
with
the
step
stats
table
and
I
can
just
search
for
that
directly
and
be
brought
to
this
page
in
the
asset
catalog
here.
D
Not
only
do
you
see
the
most
recent
materialization
information
and
whatever
metadata
that
dbt
has
decided
to
include
on
it,
but
you
also
see
a
historical
record
of
every
single
time.
This
has
ever
been
materialized,
so
if
you're
debugging,
you
can
always
go
and
click
into
the
particular
run
that
generated
this
particular
materialization,
which
is
an
extremely
powerful
debugging
tool.
D
If
you're
working
on
the
step
steps
model,
you
can
directly
kick
off
runs
of
this
or
re-materialize
this
particular
asset
from
this
page
directly
and
showing
that,
from
this
view,
if
I
go
here
to
the
step
stats
model,
I
can
show
off
a
particular
dbt
specific
integration
that
we've
included
in
this
and
I'll
answer.
Some
of
these
questions
at
the
end,
so
with
dbt.
D
D
D
So,
to
reiterate,
the
idea
here
is
that
a
lot
of
these
modern
data
stack
technologies,
your
ingest
tool,
your
your
transformation
tool
like
dbt
are
in
essence,
ways
to
define
assets.
Daxter
doesn't
need
to
know
the
exact
computations
like
the
down
to
the
source
code
that
are
being
run.
All
it
needs
to
know
is
that
hey
this
operation
will
produce
these
assets,
and
when
you
expose
that
information
to
your
orchestrator,
you
get
a
ton
of
value
unlocked.
D
So,
for
example,
if
you
compare
this
use
case
where
I
want
to
know,
hey
like
what
event
where
what
effect
will
modifying
this
particular
asset
have,
if
you
try
to
answer
that
question
purely
with
the
tasks
you
really
can't
do
it
all
you
know
is
that
something
in
erabite
happens
and
then
something
in
dbt
happens.
But
by
exposing
this
sort
of
rich
operational
information,
you
get
a
really
detailed
view
into
the
reality
of
the
complexity
of
your
data
platform,
so
that
is
the
the
demo
I'm
happy
to
answer
questions
but
yeah.
D
D
B
D
That
can
basically
be
an
arbitrary
python
or
set
of
python
code
that
you
run
for
each
dbt
model
to
add,
for
example,
like
row
count
etc
onto
there
and
then
there's
a
question.
Is
there
a
way
to
modify
the
dbt
command
that
is
being
run
when
an
upstream
model
has
changed
right
now
the
full
refresh
and
options
like
that
are
not
exposed,
but
I
don't
see
a
particular
reason
why
that
would
be
impossible.
D
A
A
Yeah
I
mean,
I
think
the
short
answer
to
this
is
that
you
can
use
dagster's
existing
schedule
and
sensor
or
abstractions
to
sort
of
build
as
complex
logic
here
as
you
want.
So
the
you
know,
step
one
might
be
to
define
a
asset
job
which
is
essentially
a
selection
of
assets
that
you
want
to
all
execute
at
the
same
time.
A
A
David,
I
had
a
question
for
you
actually,
which
was
you.
You
talked
about
those
three
stages
of
sort
of
development
from
like
project
based
the
pipeline
base
to
asset
based.
I
I'm
curious,
like
what
did
monitoring
look
like
for
you
in
the
project
based
phase
like
what
was
there
like?
Absolutely
no
monitoring
was
there
ad
hoc
monitoring.
C
There
was
stakeholders
shouting
at
you
monitoring,
yeah,
I
mean
in
the
in
my
experience
in
the
project
base,
it's
all
about
to
deliver
a
certain
amount
of
thing
by
a
certain
date,
you're,
obviously
late.
So
you
cut
scope,
you
deliver
the
best.
You
know
the
best
that
you
could
do
and
then
it's
on
to
the
next
thing:
there's,
never
any
monitoring,
there's
just
stakeholders
that
are
upset
and
there
isn't
any
anybody
who
really
remembers.
C
You
know
how
it
was
supposed
to
work.
They've
moved
on
to
different
projects,
so
I
think
you
know
part
of
the
idea.
Is
this
notion
of
data
becoming
a
product
with
a
team
that
knows
all
about
that?
You
know
the
data
asset
product
and
cares
about
it
and
monitors
it,
and
you
know
detects
problems
before
the
stakeholders.
A
D
I
can
show
the
code
for
the
the
air
by
dbt
demo.
I
completely
forgot
to
do
that.
Let
me
share
my
screen
again,
so
this
is
what
it
looks
like.
The
air
by
integration
is
not
in
open
source.
At
the
moment
it
will
be
by
of
ortino,
but
the
basic
idea
is,
you
know,
for
dbt,
it's
really
easy
to
generate
all
of
the
assets.
Just
because
dbt
contains
all
the
metadata
that
we
need
to
generate
the
dependency
information
for
airbite.
D
D
A
All
right
with
that,
I
think
that
is
the
end
of
our.
We
have
one
more
question
from
a
soft
lobby.
I'm
just
gonna
read
it
out
loud.
Are
there
plans
to
implement
something
like
kubernetes
liveness,
slash,
readiness
pro
apis
for
software-defined
assets,
kind
of
formalizing,
some
of
the
stuff
that
david
discussed
like
a
monitoring
job
as
official
dexter
apis.
A
F
A
F
F
A
A
I
think
we
are
definitely
very
interested
in
offering
the
ability
to
have
kind
of
like
first
class
asset
tests
or
like
asset
quality
checks,
arbitrary
code,
that
users
can
write
that
looks
at
an
asset
and
determines
whether
something
is
wrong
or
not.
And
so
then
I
think
what
that
would
probably
look
like
for
us
to
complete
the
circle
would
be
to
connect
those
to
sensors,
so
allow
you
to
to
look
at
the
results
of
those
tests
inside
of
a
a
sensor,
evaluation
to
figure
out
what
to
launch
next.