►
From YouTube: An Introduction to Dagster: The orchestrator for the full data lifecycle - UDEM June 2021
Description
Nick Schrock covers the principles and origin of Dagster. Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster considers the entire data application lifecycle. Practitioners in Dagster build data-aware dependency graphs designed for local development and testing; deploy those graphs to a multi-tenant, cloud-native orchestration engine; and then monitor and observe the data assets produced by those computations.
In this talk, Nick discusses how Dagster differentiates itself across the three stages (dev & test, deploy & execute, monitor & observer) of the application lifecycle. Through a demo and code snippets, the talk shows how the Dagit web UI and Dagster programming model can power a variety of data practitioners.
Related video:
Data Driven NYC talk on Fundamentals of Data Engineering (https://youtu.be/mPSzL8Lurs0)
#data #dataengineering #airflow #prefect #cloudnative #cloudcomputing #cloud #fundamentalsofdataengineering
-------------------------
About Nick Schrock
Nick Schrock is the founder and CEO of Elementl, the company behind Dagster. Previously, Nick worked at Facebook, where he co-created GraphQL. Nick believes deeply in the power of well-designed developer tools to make engineers more productive, accelerate their careers, make their lives more enjoyable, and transform the organizations in which they work.
A
All
right,
hi,
everyone,
my
name
is
xrock
and
I'm
the
founder
of
elemental,
the
company
behind
dagster,
thanks
for
thanks
so
much
for
having
me,
it's
really
exciting
to
be
here
and
yeah
just
before
I
get
into
the
technology
just
wanted.
To
give
a
brief
background
on
me.
In
a
previous
life,
I
worked
at
facebook,
where
I
found
a
team
called
product
infrastructure
which
built
abstractions
and
frameworks
for
our
product
engineers.
A
A
A
So
briefly,
I'll
go
over
this
quickly,
because
I
think
this
crowd
is
very
familiar
with
the
term.
But
what
is
data
orchestration
and
effectively?
You
know
it's
two
things
right:
it's
whenever
a
person
or
team
puts
a
data
asset
into
production
and
by
asset
I
mean
a
file
that
gets
produced
a
table,
a
pickled
model,
whatever
it's
not
done
in
a
vacuum
right
data
has
to
come
somewhere
and
go
somewhere,
and
that
means
dependencies
right.
So
that
has
to
be
managed
and
then
furthermore,
another
interesting
thing
is
about.
A
Even
as
an
individual
practitioner,
you
are
building
dependency,
graphs
right,
you're,
building
this.
What
we
like
to
call
data
application
and
you're
paralyzing
things
you
want
to
make
sure,
there's
checkpoints:
you
can
start
where
you
left
off.
So
even
if
there's
another
team
involved,
you
should
just
you're
just
developing
the
dag.
A
So
these
orchestration
systems
and
workflow
managers
are
deeply
ingrained
at
both
kind
of
the
individual
at
practitioner
level
and
at
the
team
at
organizational
level
and
in
the
traditional
definition
of
workflow
managers
is
that
it
simply
orders
schedules
and
orders
computations
in
production,
but
we
think
that
limited
definition
has
a
huge,
missed
opportunity
and
misses
a
huge.
You
know
point
of
leverage.
A
You
know
we
think
really
orchestration
matters,
because
we
view
it
as
really
the
center
of
gravity,
both
the
data
platform,
as
well
as
the
kind
of
data
life
cycle,
the
software
development
life
cycles.
It
comes
to
data,
so
there's
a
bunch
of
different
people
that
interact
with
the
orchestrator
right
one
is
you
have
practitioners?
A
There's
other
people
too
right.
There's
platform:
these
are
folks
who
are
responsible
for
supporting,
said
practitioners,
there's
tons
of
infrastructure
needs
and
really
you
know
what
we
think
that
our
goal
should
be
is
to
enable
end-to-end
ownership
of
data
products
by
practitioners
all
the
way
from
developing
tests
to
monitor
and
serving
those
those
produce
assets.
And,
lastly,
you
have
stakeholders
who
are
interested
in
data
assets
and
even
our
participate
in
the
operation
of
data
systems.
A
You
know
we
have
lots
of
partners
who
end
up
incorporating
non-technical
stakeholders
to
in
a
self-serve
way,
monitor
operations
and
asset
production,
and
at
the
center
of
this
is
the
orchestrator
our
cute
little
friend,
dexter
know.
So
we
just
think
this
is
a
critical
leverage
point
in
these
systems,
so
I
referred
to
the
life
cycle.
A
You
know
in
terms
of
developing
tests
we
freaking
hear
like
frequently
here
like
there's
no
way
I
can
develop
my
tags
locally
right.
All
my
bags
are
bound
to
the
production
environment,
the
sk.
You
know
I
have
to
have
the
scheduler
running
in
order
to
do
anything,
just
local
development
isn't
a
thing.
A
Next,
once
you
deploy
and
execute
these
things,
you
know
like
airflow
in
particular,
was
not
designed
to
be
a
multi-tenant
multi-team
environment,
all
of
the
dags
in
a
particular
airflow
installation.
All
share
the
same
process,
the
same
python
environment.
That
means
there's
like
massive
dependency
conflicts.
A
And,
lastly,
there's
a
modern
observation
aspect
of
this,
and
we
hear,
like
you
know,
like
things,
are
difficult
to
bug
some.
Some
of
this
is
ui
issues,
but
at
more
core
is
the
lack
of
the
lack
of
data
awareness
and
effectively
all
the
existing
orchestrator
systems
that
we
know
of
really
limits
in
terms
of
both
in
terms
of
high
quality
debugging
tools,
as
well
as
this,
the
ability
to
know
where
data
comes
from.
A
So
these
are
kind
of
the
things
we
hear.
So
what
I'm
going
to
do
is
hop
directly
into
demo
to
demonstrate
what
we're
it's
kind
of
a
tautology.
Yes,
I'm
demonstrating,
usually
a
demo.
The
yeah,
we're
going
to
you
know,
show
this
demo
to
show
off
the
system.
It's
going
to
be
this.
You
know
we're
going
to
build
kind
of
a
classic
data
application,
a
hacker
recommendation
engine
and
it
has
a
few
stages.
A
One
is
that
you
have
the
stage
kind
of
written
by
data
engineers
say
where
you're
downloading
data
from
the
public
after
news
api
you're
using
spark
to
compact
that
into
parque
and
upload
it
to
bigquery,
and
then
we'll
show
how
downstream
consumers
of
that
data
product
use
this
system
to
build
a
recommendation
engine
and
also
build
analytics
dashboards
using
dbt.
So
this
is
kind
of
one
of
those
classic
multi-persona
data
platforms,
where
you
got
to
get
everyone
to
look
together
and
you
want
to
have
a
unified
data
platform.
A
So,
let's
start
with
a
devon
test
like
devon
test
life
cycle.
I
introduce
you
here
to
who
we
call
dag,
stir
s-t-I-r,
the
icon
of
dexter
with
the
chef
which
I
think
is
particularly
cute.
So
why
do
you
think
devon
test
is
so
important
and
such
an
opportunity
for
the
data
domain,
and
the
reason
is
because
we
think
this
is
a
status
quo
with
most
data
teams
that
too
many
of
their
errors,
software
errors,
programmatic,
etc,
are
caught
way
too
late
in
their
life
cycle.
A
A
You
know
our
goal
is
to
flatten
the
curve,
and
you
know
we're
not
claiming
that
all
development
can
be
local,
because
there
are
just
certain
things
in
data
systems,
as
you
know
that
you
can't
test
locally
right.
They're,
like
production
issues,
performance
issued
on
your
production
data
data,
quality
tests,
et
cetera,
but
that
leaves
tons
of
errors.
A
So
this
flattening
of
that
curve,
because
the
errors
are
so
much
more
expensive
down,
the
line
represents
like
a
order
of
magnitude
reduction
in
cost
order,
magnitude,
magnitude,
increased
the
quality
of
life
of
developers,
practitioners
and
on
and
on
a
lot
of
goodness
comes
from
this.
So
I've
been
talking
for
a
while.
Let's
talk
about
some
code,
so
this
is
what
daxter
looks
like
excuse
me:
open
source
python,
module
pip
installed
extra.
You
can
write
code
like
this.
We
have
you
decorate
a
function,
you
call
it
a
pipeline,
that's
what
constructs
the
dependency
graph!
A
So
that's
not
actually
invoking
the
hello
world
function,
it's
simply
creating
a
dag
structure,
and
then
we
have
our
node
of
computation,
which
we
call
a
solid
which
actually
performs
a
compute.
And
then,
as
you
see,
we
have
a
lightweight
api
where
it
can
just
execute
that
pipeline
and
that's
what
hello
world
looks
like.
So
let's
hop
and
do
a
slightly
more
real
example.
A
So
this
is
a
piece
of
code
which
takes
a
range
of
ids
downloads
them
from
a
public
acronym
news,
api
and
then
plops
into
a
data
frame
right
classic
step
in
the
atl
prices
process.
A
few
things
to
note
here,
one
our
nodes
of
compute
are
unlike
tasks
in
traditional
or
workflow
engines,
which
are
simply
just
encode
execution
dependencies.
We
have
data
dependencies,
so
our
nodes
declare
inputs
and
outputs
and
they're
optionally
typed.
We
think
dags
should
be
typed
once
they're
put
into
production
and
we
think
there
should
be
data
dependencies.
A
Second,
the
body
is
arbitrary
python.
You
can
do
whatever
you
want,
invoke
any
computational,
runtime,
any
tool
etc.
Lastly,
is
separation
of
I
o
and
compute
you'll
notice
that
there
is
no
call
to
dot
two
csv
or
dot?
Two
part
k
right.
These
are
functions
which
perform
a
computation
and
return
their
logical
representation.
A
A
A
Let's
see,
are
we
seeing
this
cool
the
moment
that
you
have
these
computations
in
format?
You
have
all
this
tool
in
this
disposal,
so
this
is
a
live
demo.
On
my
laptop,
all
I
typed
was
daggett,
which
is
the
name
of
our
web
tool.
You
don't
need
to
have
the
schedule
or
process
running
you,
don't
need
to
have
a
database
running
and
all
of
a
sudden
you
have
this
kind
of
you
know
what
we
like
to
think
is
a
beautiful
tool
to
you
know
and
we're
gonna
start
by
inspecting
this
stuff.
A
So
one
nice
thing
we
like
to
think
this
is
a
consumer
grade
ui,
so
you
can
just
kind
of
do
a
global
search
and
search
for
things
and
kind
of
you
know
you
can
think
and
just
type
things
and
things
pop
up,
which
is
very
nice.
So
here's
a
representation
of
this
pipeline
and
you'll,
see
here
that
here's
that
solid,
that
I
showed
in
the
slide
and
look
at
all
this
rich
metadata
right.
A
It's
extremely
empowering
just
be
able
to
get
clone
one
of
your
colleagues
work
and
then
just
type,
dag
it
and
like
see
what
they
did
right
without
having
set
up
infrastructure.
It's
really
really
powerful,
but
this
isn't
just
a
view
only
thing
you
can
actually
run
ad
hoc
computations
from
here.
So
we
have
this
thing.
We
call
the
playground-
and
so
let's
imagine
that
I
was
developing
on
this
and
I
want
to
have
a
fast
developer
loop
and
we
like
we
want
to
treat
this
as
almost
like
an
ide
for
dags
flip
over
here.
A
B
A
Oh,
thank
god,
all
right
everything
is
fine,
and
so
you
notice
here
this
is
actually
executing
my
laptop.
We
have
this
live,
updating,
gantt
chart
and
boom.
Lo
and
behold,
we
have
a
failure,
so
you
can
click
on
this
right
here.
We
have
this
thing
called
the
structured
event
log.
So
this
is
not
just
pure
text
logging.
We
actually
log
these
structured
events,
so
we
can
have
rich
viewers
of
this.
So
here
we
have
a
step
failure.
You
can
view
this
message
it
formats
it
as
a
stack
trace.
A
A
Again,
it's
like
this.
Nice
live
updating,
gantt
chart
which
is
actually
quite
fun
to
use,
and
lo
and
behold,
we're
often
races.
So
you
know
we
have
this
nice
web
tool.
We
can
render
things
quite
beautifully
and
have
this
local
development
cycle
that
incorporates
it,
but
not
everyone
wants
to
work
in
ide
all
the
time.
Understandably,
so
we
also
have
very
convenient
apis
for
locally
developing
things.
So
here
we
do
an
invocation
which
actually
just
tests
the
downloads
out
and
I'll
talk
about
what
that
resources
think
makes.
A
But
these
are
just
vanilla,
python
functions
which
you
can
distribute
the
computation,
but
you
can
just
treat
it
effectively
like
a
physical,
vanilla,
python
function
right
there,
english
build
build
unit
tests
and
do
tdd
and
all
the
fun
stuff.
If
that
is
up
your
alley,
okay,
so
one
of
the
things
I
talked
about
was
early
cycle
detection.
Pardon
me
early
error,
checking
an
example
of
that
is
cycle
detection.
A
So
you
have
a
dag
a
goes
to
b
and
you
want
to
detect
to
make
sure
it
never
loops
back
and
goes
back
to
a
because
that
would
be
bad
things.
Would
infinitely
execute-
and
you
know,
dag-
is
a
directed
acyclic
graph,
so
this
slide
is
from
a
presentation
from
last
year's
airflow
summit
and
it
was
about
testability,
and
this
was
the
code
they
had
to
write
to
successfully
check
whether
or
not
there
was
a
cycle
in
the
dags
in
there
in
their
airflow
installation
and
effectively,
because
it's
so
tied
to
the
scheduler
at
airflow.
A
A
You
know
in
daxter
they're
standalone
artifacts,
which
means
that
you
simply
cannot
construct
them
with
a
cycle.
So
literally,
once
you
can
do
this,
you
know
you
don't
have
a
cycle.
You
can
move
on
with
your
life.
You
know
it's
just
like
one
example
of
many
of
like
moving
error
detection
up
the
curve
by
default.
You
know
we
like
to
call
that
a
pit
of
success
developers
fall
into
you
know,
but
you
want
to
go
beyond
what
we
call
structural
testing.
A
It's
not
local
against
production
resources,
so,
instead
of
executing
your
local
cluster
you're,
instead
launching
emr
job
and
persisting
the
intermediates
to
your
s3
data
link.
So
how
do
we
accomplish
that?
The
critical
thing
is
this
abstraction:
we
have
called
the
resource,
so
we
flow
a
context
object
through
all
of
our
computations
heading
back
to
the
download
items
thing
we
talked
about
and
you'll
notice
here
that
there's
a
context
object.
We
have
this
notion
of
resources,
so
the
the
solid
declares
what
it
needs,
and
then
you
invoke
those
resources
through
that
context.
A
What
this
allows
you
to
do
is
late
bind
what
you
actually
call
so,
for
example,
in
test,
you
might
just
hit
the
snapshot
of
the
data
right,
which
is
what
I
just
did
locally,
which
made
me
more
confident
that
it
was
going
to
work
super
reliably
and
you
can
run
a
csd
pipeline
without
incurring
errors
from
the
public
api,
but
in
pride
you
still
want
to
go
against
requests
right.
You
know
that's
a
python
library,
but
you
want
to
go
against
the
public
api
for
hacker
news.
A
A
This
is
known
to
be
different
from
airflow
most
of
those
systems
that
have
centralized
task
schedulers,
so
dame
the
damon's
only
responsible
for
runs
and
we'll
go
into
that
workflow
for
a
second.
This
is
another
critical
thing
which
you
know
makes
dijkstra
suitable
to
be
a
multi-team
multi-tenant
platform,
which
is
that
any
code
that
the
user
writes
the
practitioner
writes
is
not
in
process
with
the
scheduler
or
the
web
server.
A
It's
invoked
over
an
api
that
means,
for
example,
if
a
user
pushes
up
a
syntax
error,
the
centralized
core
system
simply
notes
that
they
can
launch
this
container,
and
then
you
know
there's
a
little
red
dot
in
the
ui
and
says,
like
you
have
to
do
something
rather
than
the
whole
thing
coming
crashing
down
and
this
process
isolation.
Aspect
of
this
really
is
the
I
would
say,
the
the
foundation
for
why
this
is
such
an
operationally
robust
system.
A
Our
run
scheduling
is
designed
to
be
fault
tolerant.
So
it
kind
of
follows:
a
reconciliation
based
model
similar
to
something
like
kubernetes,
which
means
that
we
do
scheduler
ticks
and
it
computes.
What
runs
should
exist
and
then
the
system
reconciles
that
that
means
that
the
run
scheduler
process
goes
down.
It
will
always
catch
up
once
once
it
reboots,
also
as
a
distinction
from
most
of
the
systems
that
we
don't
just
do
time-based
scheduling
in
this
matter,
but
we
also
do
event-based
scheduling
this
matter.
A
So,
instead
of
saying
having
a
task
that
runs
forever,
we
actually
integrate
sensors
or
event-based
schedules
into
the
run
scheduler
itself,
and
then
every
time
we
launch,
we
launch
a
new
unit
of
computation
to
mean
like
a
process
or
kubernetes
pod.
You
know
this
is
an
infrastructure
neutral
technology,
so
you
can
adapt
it
to
your
own
system,
but
it
runs
this
piece
of
ephemeral
compute
which
it
is
responsible
for
test
scheduling.
A
So
this
is
much
more
amenable
to
having
infinitely
elastic
compute
available
in
a
public
cloud
and
distributes
task
scheduling
which
generally
helps
the
robustness
of
the
system.
So
this
is
very
customizable,
but
we
also
come
out
with
a
kind
of
easy
to
use
out
of
the
box
production
grade.
Kubernetes
deployment
with
the
prefab
help
chart
that
you
can
get
spun
up
with
quite
quickly.
A
So
what
does
this
look
like?
I'm
just
going
to
go
quick
back
to
this
and
I'm
going
to
switch
tabs
you'll
notice
instead
of
local
host.
Here,
I'm
actually
running
on
demo.elemental.show,
which
is
our
kubernetes
cluster,
one
of
our
kudu
united's
clusters
in
gcp.
So
I
can
do
the
same.
I
can
use
the
same
tools
go
to
that
download
pipeline.
Go
to
the
playground.
A
You
notice
I'm
in
what
I
call
what
we
call
prod
mode,
and
this
is
actually
not
going
to
run
like
on
that
local
box
right.
This
is
actually
going
to
kick
off
kubernetes
jobs
and
launch
and
launch
the
run
on
this
ephemeral
compute
so
for
every
run,
there's
a
job
launch
and
then
the
compute
happens
in
the
cloud
same
tools,
you're
familiar
with
and
so
on
and
so
forth.
So
this
is
taking
a
bit
to
spin
up
the
kubernetes
job
and
now
we're
off
to
the
races.
A
So
this
is
not
just
your
local
development
experience.
This
is
also
your
operational
pipeline
and
we'll
return
back
to
that
soon
enough.
Okay.
So
let's
talk
about
the
third
life
cycle
stage
that
I
mentioned.
Observing
and
monitoring
and
critical
in
dagster.
Is
that
we're
not
just
observing
and
monitoring
computations
we're,
also
observing
and
monitoring
the
produced
assets,
and
we
think
this
is
critical
to
a
highly
functioning
orchestrator.
A
A
A
They
never
have
to
bother
everything.
Everyone
is
happier
that
way.
So
what
does
it
look
like
to
build
a
little
more
advanced
data
pipeline?
So
this
is
this
story
recommender
pipeline,
which
is
doing
a
very
ml
thing.
It's
doing
some
data
processing
and
it's
building
a
recommender
model,
and
then
you
know
building
a
bunch
of
components
in
a
in
in
a
date
in
a
pardon
me
ml
life
cycle.
So
this
is
what
building
a
pipeline
looks
like
it's
kind
of
this
function
x,
dsl.
This
is
not
invoking
the
computations
directly.
A
This
is
constructing
the
dependency
graph
and
it's
doing
a
data
ware
way,
so
this
ends
up
building
a
dependency
graph
like
this.
What
does
the
code
look
like?
So,
if
you're,
a
data
scientist
you're
using
dagster
you're
writing
what
is
mostly
looks
like
vanilla
code
right
you're
using
familiar
abstractions
and
whatnot,
but
at
the
bottom
here
there's
something
a
little
unusual.
A
You
know
this
is
that
structured
event,
stream
that
I
was
talking
about
and
what
this
code
is
doing.
Is
it's
informing
dexter
that
hey
I'm
about
to
create
this
asset
that
lives
at
this
spot
and
it
has
these
properties
and
that's
a
really
powerful
thing
to
have
an
orchestrator,
be
aware
of
that,
and
I
will
show
you
why
the
what
just
a
reminder:
what
do
we
think
of
as
assets?
So
it's
effectively
anything
that
outlives
the
scope
of
the
computation
right.
A
It
could
be
a
data
warehouse
table,
a
parquet
file,
a
data
lake,
a
pickled
model,
and
what
we've
found
is
this
core
insight
that
peop
your
stakeholders
and
people
only
care
about
the
data
assets?
I
hate
to
break
it
to
you.
They
don't
care
about
your
pipelines.
They
don't
care
what
their
structure
is.
They
don't
care
what
they're
named?
No
one
cares:
they
only
care
about
your
assets
and
how
does
that
end
up
materializing
in
the
system
to
overuse
that
phrase?
A
A
A
It
does
that
by
querying
the
metadata
in
our
asset
catalog-
and
you
see
how
this
is
like
a
live,
updating,
ui,
it's
very
fun,
and
here
it
kicked
off
this
run.
Okay,
so
if
we
go-
and
this
is
the
run
that
got
kicked
off
and
look
at
it,
what
I'm
going
to
do
here
is
we're
talking
about
assets
right,
so
this
pipeline
has
generated
a
bunch
of
assets
and
it's
done
it
with
our
structured
event
stream.
So
I
can
go
here
and
type
and
search
for
all
the
asset
materializations
and
boom
there.
A
It
shows
up
here
are
all
the
things
that
have
been
generated
by
by
this
by
this
computation.
So
you
can
go
here.
We
have
all
the
structure
information,
so
we
can
say
like
okay,
here
are
the
data
frame
columns?
That's
interesting,
but
then
there's
also
this
link
here
and
we
can
go
and
click
on
this.
Let's
go
to
the
recommender
model
and
I
want
to
see
what
the
history
of
this
asset
is,
and
I
load
this
and
like
there's
all
this
interesting
information.
So
look.
We
have
spark
lines
here.
A
We
can
track
what's
been
happening
to
the
step
execution
over
time,
see
what
the
total
explained
variance
ratio
is
see
what's
going
on
here
and
it's
just
structured
event,
log
that
tracks
what's
been
happening
to
the
stuff
over
time.
You
know
the
other
really
interesting
thing
about
having
the
orchestrator
be
asset
aware.
Is
that
it's
it's
entirely
new
way
to
index
the
orchestrator.
So
imagine
that
you're,
a
stakeholder-
and
you
don't
know
anything
about
these
pipelines.
A
You
don't
know
you
don't
care,
but
what
you
do
know
is
that
you're
looking
at
a
stories
table
in
snowflake
and
something's
off,
and
you
want
to
know
what
the
hell
is
going
on.
Well,
you
can
go
right
in
here
search.
Oh
look!
Here's
this!
Here's
the
stories
table
in
snowflake
boom
launch
into
that
all
of
a
sudden.
I
see
this
in
front
of
me
interesting
information.
A
Imagine
it
had
been
run
for
a
few
days
or
something-
and
you
see
that's
out
of
date
and
then
you're
like
wait.
What
pipeline
actually
did
this
and
click
on
this
and
see
like
okay?
This
is
the
pipeline.
You
did.
Oh
here
are
the
owners
right.
So
this
really
allows
stakeholders
it's
kind
of
indexed
into
the
orchestration
system
by
assets.
A
To
this
demo
again
wanted
to
talk
about
the
sensors
concept,
because
we
think
it's
really
powerful.
You
know
the
way
that
these
pipelines
operate.
Is
that
there's
an
upstream
pipeline
there's
an
upstream
team
that
manages
a
pipeline
and
then
a
downstream
team-
and
you
know
a
very
common
pattern-
is
to
create
these
mega
dags,
where,
like
your
whole
data
platform,
is
in
one
or
very
few
huge
decks,
and
we
think
you
should
be
able
to
componentize
this
a
little
more,
but
without
tight
coupling,
so
they
should
be
able
to
deploy
those
pipelines
independently.
A
They
should
be
able
to
use
different
python
environments,
etc,
etc.
But
the
nice
thing
about
these
sensors
and
what
these
sensors
do
is
they
agree
on
an
asset
key
in
our
asset
catalog.
So
literally,
when
we
built
this
demo
pipeline.
What
happened
is
that
the
data
science
person
and
the
data
engineering
person
they
simply
agreed
hey
the
stories
table
is
our
contract
and
then,
from
that
point
on,
there
was
very
little
coordination
right
team
one
developed
their
pipelines
completely
independently.
They
determine
names,
they
determine
the
structure
same
likewise
for
team
two,
but
these
asset
sensors.
A
A
I'm
gonna
quickly
talk
about
analytics
engineering,
because
I
assume
this
crowd
is
familiar
with
dbt
and
you
know:
let's
talk
about.
Okay
now
the
dbt
analytics
engineering
team
wants
to
hook
onto
this
and
construct,
take
the
assets
produced
by
the
data
engineering
team
and
then
construct
a
bunch
of
tables
in
their
cloud
data
warehouse
for
analytics
and
for
analytics
dashboards,
etc.
So
if
you
go
back
to
this
picture
and
you're
a
dbt
user,
you
might
see
like
oh
okay.
A
What
goes
in
the
devon
test
box,
because
I
really
like
dbt-
and
I
don't
want
to
change
that-
I
develop
in
it-
I
think
in
it,
and
you
know
we
agree
that
you
should
continue
to
do
that.
Dagster
does
not
proclaim
to
take
over
the
kind
of
dbt
developer
experience.
We
consider
dbt
a
highly
philosophically
aligned
system
that
really
thinks
about
data
computations
in
the
same
way,
meaning
that
you
know
the
nodes
of
computation
should
be
pure
functions.
You
should
depend
on
upstream
data,
and
you
know
it's
really
in
a
lot
of
ways.
A
So
this,
whatever
dbt
integration,
looks
like
it's
very
straightforward
to
create
dbc
solid
you
just
kind
of
effectively,
you
know
provide
where
your
profiles
and
your
project
live,
you're
good
to
go
and
then
just
quickly.
You
know
this
isn't
super
profound,
but
it's
working.
We
hooked
up
the
dbt
pipeline
in
the
same
way
with
sensors,
so
you'll
see
that
this
run
just
happened
a
few
minutes
ago,
and
if
we
go,
we
can
check
out
the
dbt
on
h,
download
finished
and
oh,
we
missed
it.
A
So
we
go
one
day
here
these
these
are
really
fun
uis.
Actually
you
can
see.
This
is
where
it
got
kicked
off.
Okay,
that's
interesting
go
here
and
then
you
can
use
the
same
tools
to,
for
instance,
track
all
the
things
that
are
created
by
a
given
dbt
run,
so
you
can
say
okay,
I
want
to
see
the
asset
materializations
here.
Okay,
look
here
interesting
here
are
all
the
models
you
know
here
all
the
database
tables
that
were
created.
Here's
all
this
information
about
it.
A
So
once
you
actually
hook
up
the
dbt
integration,
you
kind
of
get
all
this
stuff
for
free.
You
can
keep
track
of
execution
time.
You
can
keep
track
of
how
many
rows
happen.
You
know
you
can
even
see
sample
data.
You
can
configure
that
if
you
want,
if
there's
pii
or
whatever,
but
you
know
this,
this
tool
is
a
very
rich
way
for
the
orchestra
to
become
data
aware
and
our
users
really
find
that
it
improves
debug
ability
and
whatnot.
A
Oops,
you
know,
we
think
that
orchestration
is
a
huge,
both
problem
in
data
currently
and
a
huge
point
of
leverage
and
opportunity,
because
the
orchestrators
shouldn't
just
order
structure
computations
in
production.
We
think
it
should
be
data
aware
that
computations
should
be
connected
with
data
dependencies.
A
F
B
So
what
would
you
be?
What
would
be
your
advice
for
someone
who's
coming
from
airflow?
So
there
are
several
parts
here
right
like
there's:
there's
learning
the
tool,
reading
the
documentation,
that
kind
of
thing,
but
also
they're,
supporting
a
lot
of
old
airflow
code
that
you
have
laying
around
or
actually
running
in
production
over
to
dexter.
What
would
be
your
general
perspective
on
that.
A
So
funny
enough,
we
have
a
I
can.
I
have
some
additional
slides
just
for
this
type
of
question:
great
okay,
so
we
built
a
very
flexible
pluggable
system
and
one
of
the
things
that
users
have
done
to
kind
of
dip
their
toes
in
the
water,
especially
in
cases
where
there's
a
grumpy
infrastructure
team
that
doesn't
want
to
deal
with
a
new
tool,
which
I
completely
sympathize
with.
One
second
here
is
that
so
we
have
the
ability
for
you
to
write,
dexter
pipelines
and
actually
compile
them
to.
A
A
So
how
that
works
is
like
imagining
this
airflow
installation
there
are
dags
encoded
in
airflow,
and
you
know
data
science
they're
like
oh,
my
god.
I
want
to
use
a
new
tool
and
the
ops
people
are
like
get
get
out
of
here
like
another
tool.
No
and
again
I
understand
that
so
we
have
this.
What
we
call
you
know,
dag
injection,
so
the
leaf
developer
will
develop
code
in
you
know
in
the
daxter
api
and
then
what
they
could
do
is
in
their
airflow,
dag
construction.
A
Run
over
infrastructure
we
have
mapbox.
Does
this
and
they've
actually
recently
done
a
blog
post?
So
what
was
cool
about
that
is
that
they
actually
50
of
their
compute
spend
was
actually
spent
in
r
d
because
they
were
developing
against
their
production
clusters
and
they
were
able
to
switch
the
dagster
and
do
more
local
development
and
local
end
development,
also
on
ephemeral,
ur
clusters
and
it
actually
cut
their.
A
It
actually
cut
their
spend
so
much
that
the
managers
thought
that
there's
something
wrong
and
then
they're
like
no.
No,
it's
actually
fine.
So
this
has
been
effective
for
a
bunch
of
our
users.
Then
you
know
is
more
kind
of
well.
I
can.
I
can
start
there,
so
you
know,
in
terms
of
you
know,
that's
kind
of
one
strategy
for
dipping
your
toes
in
the
water
in
terms
of
using
that
injection
thing.
But
you
know,
I
think
you
know
when
you
first
start
out,
I
would
just
kind
of
build.
A
You
know,
take
something
that
you
would
have
built
an
airflow
and
build
it
in
the
system.
You
know
and
work
through
our
docks,
and
then
you
know,
I
think
what
I
was
describing
was
kind
of
more
how
to
you
know,
introduce
it
incrementally
into
our
organization,
because
I
am
an
ideological
incrementalist
and
thinks
that
you
should
have
multiple
points
of
ingress
into
these
types
of
systems
feel
free
to
follow
up
with
questions.
B
C
Yeah,
first
off
thanks
for
the
the
presentation
nick,
this
has
been
awesome.
It's
exciting.
It
mirrors
a
lot
of
the
conversations
we've
had
internally
on
some
of
the
the
problems
we're
facing
at
weave
and
how,
as
an
infrastructure
team,
we
want
to
kind
of
enable
people
better.
So
that's
really
exciting.
One
of
the
issues
that
comes
up
for
us
is
kind
of
empowering
teams
to
use
their
own
language
of
choice.
Right,
we
run
a
lot
of
our
stuff
in
kubernetes
and
python.
A
Yeah,
so
it's
a
good
question.
You
know
there's
nothing
that
prevents
you
from
writing
a
solid.
It
only
does
launch
as
an
external
container
right.
So
the
you
know,
I
believe
so.
Yes,
you're
kind
of
like
opting
out
a
lot
of
the
model
effectively
by
doing
that,
but
you
can
at
least
integrate
other
things.
A
I
think
in
the
short
term,
we're
very
focused
on
python
I
mean
we
have
users
who
like
invoke
scholar,
jobs
and
computations
in
our
languages,
and
it's
effectively
kind
of
just
launches
an
external
process
and
dykstra
doesn't
know
that
much
about
it.
You
know
we're
kind
of
we
don't
want
to
bite
off
more
than
we
can
chew,
but
certainly
the
long-term
vision
of
this
is
multi-language.
A
In
terms
of
you
saw
the
if
you
recall
the
chart
our
tools
interacted
with
our
user-defined
computations
over
an
api,
and
so
what
we
want
to
get
to
is
get
daxter's,
stable,
api
and
abstraction
layer
where
that
api
can
become
a
standard,
and
then
you
can
imagine
effectively
dagstr
being
implemented
in
other
languages
so
that
you
can
interact
with
this
in
a
kind
of
have
the
tools
cross
different
different
domains.
You
know,
I
think
the
first
instance
of
that
we
actually
have.
A
You
know
we
like
we
have
like
a
library
which
can
launch
the
scholar
job
a
spark
job,
that's
encoded
in
scala,
for
example,
but
you
know,
as
the
system
grows,
I
really
do
want
to
mature
it
so
that
we
can
incorporate
multi
language
in
a
much
more
rich
way.
So
like,
for
example,
I
think
the
first
step
would
be
getting
that
scala
spark
job
to
be
able
to
emit
those
structured
events
and
provide
inject
rich
information
into
the
observability
stream
as
an
example.
C
E
D
B
So
so
another,
oh
I'm
new
to
myself.
Okay,
you
guys
can
hear
me.
So
let
me
ask
you
a
pragmatic
question.
I
think
the
complaint
I've
heard
with
from
people
mostly
switching
to
prefect-
I
I
kind
of
heard
about
prefect.
Before
I
heard
about
you
guys,
the
main
complaint
was
not
about
the
tool
itself.
It
was
mostly
about
mind,
share
and
then,
from
my
perspective,
the
other
big
issue
is
managed
services.
B
So
those
are
like
two
advantages
that
airflow
has
right
now,
huge
mine
share
lots
of
people
using
it,
and
then
there
are
some
several
vendors
that
have
managed
services,
so
you
don't
have
to
manage
clusters
or
anything
like
this.
You
may
have
said
this
in
your
talk,
but
like
what
do
you
guys
have
in
terms
of
managed
services?
And
then
I
guess
this
talk
is
part
of
part
of
the
move
to
expand
mindshare.
But
what
are
your
perspectives
on
growing
the
mindshare
for
daxter.
A
There's
not
gonna
be
enough
as
much
mind
share
as
the
legacy
tool,
and
you
have
to
believe
that
the
value
is
big
enough
to
warrant
that.
So
you
know
this
is
kind
of
classic
adoption
curve
in
technology
like
not
everyone's
going
to
adopt
everything
all
at
once.
Everyone
has
different
risk
tolerance
and
the
willingness
to
deal
with
newer
technologies
so
like
that
is
what
it
is.
You
know
we
strive
to
have
super
high
quality
documentation.
A
A
We
know
we
have
delivered
something
of
value
and
the
people
using
the
system
currently
are
getting
a
ton
of
benefit
out
of
it
and
kind
of
the
next
stage
of
the
company
is
two
things.
One
is
really
accelerating
our
growth
trajectory
and
two
is
a
we
without
divulging
too
much
there's
a
commercialization
in
the
works,
and
I
think,
certainly
a
managed
service
will
be
a
component
of
that.
So
you
know,
airflow
was
actually
you
know.
Airflow
was
started
in
2014
at
airbnb.
A
It
actually
stems
from
a
system
called
data
swarm
at
facebook,
which
was
created
in
2007,
so
airflow
is
very,
very
similar
to
data
swarm,
at
least
at
the
time
that
it
was
originally
created.
So
you
know
that
that
is
what
it
is.
It's
the
primary,
if
your
own,
if
your
primary
evaluation
criteria
is
like
I
wanna,
you
know,
I
want
something
that
you
know
has
the
most
history
kind
of
definitionally.
A
new
thing
is
not
for
you,
and
that
just
is
what
it
is.
It's
incumbent
on
us
to
provide
enough
value
to.
B
Very
cool
yeah
I
mean
I've,
I've
done
a
lot
of
consulting
airflow
work
and
I
hope
to
help
to
deploy
it
at
overstock.com.
When
I
was
there
a
few
years
back
before,
you
guys
were
around
and
I'm
familiar
with
the
strengths
and
I'm
also
familiar
with
like
the
kind
of
tech
debt.
That's
been
there
all
along
and
it's
very
hard
to
remove,
which
is
what
they've
struggled
with
airflow
2.0.
So
yeah.
A
No
actually,
some
of
our
most
ardent
users
are
consulting
firms.
You
know
we
actually
have
a
consulting
firm.
We
work
with
called
thinking
machines
in
southeast
asia
and
they
they're
deploying
us
to
huge
telcos
and
financial
services
across
that
geo,
which
is
super
cool
to
see.
I
think
the
consulting
folks
like
it
because
of
the
infrastructure
flexibility
and
they
can
kind
of
build
a
tool
kit
that
spans.
B
You
know
I'll
tell
you
like
mostly:
we
try
to
stay
away
from
completely
building
systems
for
the
client,
but
we
we
tend
to
do
sort
of
a
collaborative
engagement
model,
but
something
that's
easier
to
teach
and
easier
to
maintain
is
very
attractive.
For
that
reason,
and
we
we
did
run
into
problems.
We
have
run
into
problems
with
their
flow
of
like
upgrades
and
other
issues
along
the
way,
and
that
was
very
painful,
so
yeah.
B
G
G
I
guess
our
main
concerns
about
picking
a
service
like
this
would
be
like
the
the
amount
of
support
for
the
core
library
and
also
managed
services,
because
the
the
less
infrastructure
that
we
have
to
maintain
the
better
so
and
I'm
sorry
I
jumped
in
lights.
I
probably
missed
some
of
the
detail
about
kubernetes.
A
And
in
fact,
you
know
the
system
is
very
pluggable
because,
like
you
can
run
into
your
laptop,
you
can
run
in
a
case
cluster
which
would
a
lot
of
support
for,
but
we
also
have
you
know,
users
who
have
like
deployed
to
their
own
custom
pass.
You
know,
users
who
have
deployed
it
to
ecs.
You
know
all
sorts
of
different,
all
sorts
of
different
configurations,
so
infrastructure-wise,
it's
quite
flexible
in
the
open
source
domain.
The
you
know
we
are.
A
G
B
G
A
Company
yeah
definitely
definitely
on
the
road
man
yeah
all
yeah
effectively.
You
know
if
you
actually,
you
know
we
should
just
ask
people
this
directly.
It's
like
list
all
the
things
you
hate
and
then
see
how
many
of
them
are
operational,
things
that
we
can
offload
up
for
you,
the,
like.
I
hate,
dealing
with
secrets.
I
hate
managing
servers
and
we're
like
we
do
too,
but
we
like
our.
B
Because
I
mean
frankly
when
I
started
using
airflow,
we
had
to
stand
it
up
ourselves
and,
having
you
know,
google
and
astronomer
come
out
with
managed
services
was
a
huge
game
changer
like
before
that
it
was
very
painful
and
then
just
having
someone
else
take
care
of
kubernetes
and
all
those
details
was
big.
Yep.
A
Yep
no,
this
mirror
is
like
feedback
we've
gotten,
which
is
effectively.
You
know
the
game
has
changed
with
these
open
source
projects.
It
used
to
be
that
people
would
deploy
themselves.
It
was
like
pulling
teeth
to
make
them
engage
with
a
commercial
product,
and
now
it's
completely
inverse
where
a
commercial
product
without
support
is
a
prerequisite
for
adoption,
and
so
we
are,
you
know
we
consider
we
should
consider
commercialization,
like
eventually
there's
going
to
be
a
business
model
around
this,
but
you
know
in
the
short
term.
A
E
A
So,
there's
a
lot
of
different
directions.
You
can
go
with
this,
I
think,
there's
you
know
one
aspect,
we're
seeing
and
we've
been
thinking
about
this
a
lot,
because
we
think
we
want
to
kind
of
change
our
positioning
and
messaging
a
bit
to
make
our
value
proposition
more
clear,
and
part
of
that
is
fitting
into
the
current
ecosystem.
A
So
you
know
the
the
term
that
seems
to
be
taking
over
the
most
is
the
modern
data
stack,
which
involves
a
few
things.
I
would
say
one
is
the
commoditization
of
certain
activities
which
can
be
commoditized.
For
instance,
ingest.
That's,
five
trim
and
stitch,
and
some
more
things
and
on
the
other
side,
everyone's
favorite
term,
reverse
etl,
but
tools,
operational,
analytics
tools
like
census
are
also
kind
of
commoditizing
that
so
there's
some
tools
which
are
removing
engineers
and
then
there's
some
tools
which
are
fully
embracing
the
software
engineering
unification
one.
E
A
Know
the
software
engineering
unification
of
data
and
so
the
other
component
that
people
talk
about
the
modern
data
stack
is
dvt
right
which
basically
make
taking
analysts
and
making
them
analytics
engineers,
and
I
think
this
trend
will
continue
of
really
embracing
software
engineering
processes
across
data,
and
we
really
see
ourselves
doing
that
in
the
python
layer
and
really
expand
almost
in
some
ways.
Expanding
the
scope
to
the
modern
data
stack
way
of
approaching
things
to
more
and
more
things
in
the
domain.
A
I
think
the
other
thing
that's
going
to
happen
in
three
to
five
years
is
that
there
has
to
be
a
consolidation,
because
the
number
of
tools
and
the
fragmentation
in
the
space
is
not
tenable
for
the
users
and
builders
we
all
serve.
So
there
is
going
to
be
some
sort
of
consolidating
force.
There's
one
thesis
that
you
know:
that's
the
cloud
data
warehouse
that
you're
gonna
have
the
cloud
data
warehouse
eat.
A
Everything
and
snowflake
does
sequal,
it's
gonna
expand
to
do
you
know:
non-sql
compute,
snowpark,
they're
gonna
have
all
lineage
all
everything
blah
blah
blah
and
snowflake,
I'm
very
skeptical
of
that.
If
nothing
else,
because
snowflake's
too
expensive-
and
they
can't
help
themselves,
they
have
to
you
know
they
got
to
maintain
the
stock
price
and
but
like
people
are
not
going
to
want
to
put
all
their
eggs
in
one
best
in
my
opinion,
and
there
will
be
competitor
systems
that
have
different
the
organization's
more
heterogeneous
than
that.
A
There's,
not
anyone
cloud
data
warehouse
to
rule
the
model
single
org.
So
there's
that
another
thesis
is
that
there'll
be
consolidation
around,
say,
data
catalogs,
because
this
is
like
a
common
layer
where,
like
there
are
there's
like
a
uniform
there's,
a
set
of
data
assets
in
the
org,
and
you
need
one
consolidating
layer
to
track
that.
I'm
skeptical
that
that
will
be
the
inevitable
source
of
truth,
because
it's
very
heavy
way
to
integrate
with
all
of
them
and
there's
really
no
guarantees
that
the
data
in
there
is
correct.
E
I,
when
I,
when
I
posted
a
link
to
this
meetup
today
on
linkedin,
a
friend
of
mine,
wrote
it
still
seems
like
there's
a
lot
of
consolidation
going
on
there's
a
lot
of
tools.
He
he
handles
more.
It's
a
demetrius
brinkman.
He
handles
the
ammo
ops
community
sort
of
it's
interesting
because
you
have
the
data
engineering
community
and
you
have
the
animal
ops
community
over
here
and
I
think
they're
kind
of
two
peas
in
the
pod,
but
he
brought
up
initiate
point.
A
Data
engineers
generally
have
a
different
tool
set,
but
you
know,
in
my
view,
a
lot
of
ml.
Ops.
Companies
are
really
data.
Ops
companies-
and
you
know
it's
just
kind
of
a
reflection
of
the
current
organizational
silos
and
you
know
to
be
honest,
like
a
lot
of
the
the
users
that
gravitate
towards
us
are
the
people
building
ml
training
pipelines
because
they
work
in
python.
A
They
the
whole,
like
model
of
building
computer
functions
in
python
and
then
being
able
to
easily
distribute
that
compute
and
use
that,
for
expression,
makes
a
ton
of
sense
to
them.
They
can
use
their
asset
catalog
to
track
trained
models.
That
makes
a
ton
of
sense,
so
I
think
it's
very
natural
that
we're
viewed
with
overlapping
or
those
more
ml
specific
tools.
A
E
Yeah,
it's
kind
of
weird
to
see
it
almost
like
these
two
on
one
hand,
two
parallel
universes
going
on
there's
like
ml
and
but
then
you
start
looking
at
the
overlap.
You
bring
up
data
apps,
for
example
like
I,
I
can't
see
much
of
a
difference
between
like
observability
and
ml
and
observability
with
data
in
general
right
so
yeah
I
mean
a
matter.
You
know
toying
with
this
idea
too.
E
I
I
I
see,
there's
you
know
just
in
the
book
of
writing,
we're
just
thinking
a
lot
about
this
whole
space
right
now,
and
it's
like
consolidation
is
probably
gonna
happen
in
tools,
but
it's
also
gonna
happen.
I
think
in
sort
of
between
ml
and
and
data
engineering.
I
think
these
are
kind
of
false
dichotomies
in
a
lot
of
ways,
but
I
don't
know
yeah
we're
getting
kind
of
mad
here,
but.
A
A
You
know
we're
one
of
the
things
we're
really
excited
about
is
actually
integrating
with
you
know,
we're
really
thinking
about
how
we
would
integrate
with
kind
of
notebook
based
workflows,
that
ml
folks
use,
because
we
found
people
because
the
code
you
write,
indexers,
fairly
plain
old
python,
and
you
can
also
execute
your
your
orchestration
graphs
locally.
It's
actually
very
natural
to
kind
of
start
coding
in
a
notebook
and
then
kind
of
rip
out
the
code,
one
by
one
and
plop
it
in
a
dag.
As
you
go.
A
One
of
my
colleagues
calls
this
process:
building
roman
roads,
meaning
like
as
you're
exploring
your
space.
You
like
build
rows
as
you
go
along
and
like
find
checkpoints
right,
and
so
the
analogy
there
would
be
like
you
start
exploring
the
notebook
you
find
a
checkpoint.
You
move
the
stuff
over
into
plain
old
python
and
dagster.
Then
you're
kind
of
like
using
both
tools.
A
B
Oh
no
completely
agree,
I
mean
yeah.
The
thing
is
like
in
many
respects
data
scientists
should
be
doing
their
data
exploration
locally.
I
just
think
there's
this
huge
there's
this
huge
wall
between
like
local
data
exploration
and
data
engineering
and
there's,
there's
this
partially.
It's
communication
problem.
Partially,
it's
a
tool
set
problem
partially.
It's
because
maybe
data
engineers
need
to
get
a
bit
better
at
pandas
and
data
scientists
need
to
get
a
bit
better
at
like
some
heavier
duty
tools.
F
A
I'm
not
particularly
familiar
with
the
details
of
those
technologies,
so
I
will
have
to
take
a
pass
on
that
and
get
a
homework
assignment
about
I
mean
I
could
you
know
this.
I
think
that
would
be
an
hour-long
meeting
of
me
asking
you
questions
about
what
the
value
proposition
of
those
particular
tools
are.
The
workflows
are
so
I
you
have.
You
have
touched
the
gap
of
my
knowledge
and
I
won't
pretend
to
know
what
I
don't
know.
A
B
A
F
A
A
F
A
Yeah,
I
mean
to
be
clear
if,
if,
if
you're,
if
your
entire
transform
layer
is
in
sql
dbc's
the
tool
to
use-
and
then
you
know
like
you
know
the
the
dijkstra
comes
to
play,
we
need
to
orchestrate
that
with
other
tools
or
you're
doing
data
processing
directly
in
python.
You
know
one
of
our
yeah
one
of
our
users
said
you
know
what
dbt's
diverse.
What
dbt
did
for
our
sql
dexter
did
for
our
python.
You
know
so,
if
you're,
if
your
whole
team
is
sql
based
highly
recommend,
looking
at
dbt
thanks.
D
A
So
yeah,
you
know
we're
actually
talking
to
we're
actually
talking
in
one
of
the
two
companies
acro
data
that
does
data
hub,
for
example,
that
and
that's
one
of
these
meta
stores.
So
we
certainly
support
you
know
those,
because
the
asset
catalog
that
we
have
we
consider
kind
of
like
an
operational
asset,
catalog
being
very,
very
specifically
about
the
date
observability
and
op
side
of
things
and
those
other
data.
A
A
D
Makes
sense,
and
so
there's
another
integration
point
where
your
tasks
or
solids,
I
guess,
as
we
call
them,
would
invoke
etl
pipelines
right
and
so
do
you.
So,
for
example,
we
use
a
aws
step
functions
for
all
our
for
a
lot
of
our
orchestration
like
glue
jobs
and
stuff
like
that
and
there
they
provide
very
rudimentary
functionality
and
does
really
well
and
being
serverless.
It's
just
it's
awesome
and
they
have
those
integration
points
with
athena
queries
with
glue
jobs
et
cetera
that
are
specific
and
they
handle.
D
A
I
mean,
certainly
you
know
we
have
out
of
the
box
integrations
with
you
know
the
likes
of
bigquery,
and
you
know
lots
of
different
tools
like
like
you
know.
We
have
like
dozens
of
integrations
with
tools
like
that
in
terms
of
step
functions.
We
do
not
have
any
direct
integrations
with
that.
You
know
it
would
so
if
you're,
asking
about
that,
you
know,
step
functions
is
kind
of
its
own
orchestration
layer.
A
We
can
theoretically
actually
fill
this
in
integration,
similar
to
what
we
do
with
airflow,
where
we
effectively
compile
the
extra
dags
into
the
step
function
like
json
format
or
whatever,
but
there's
no
active
plans
to
do
that,
but
certainly
at
lower
level.
The
same
way
that
that
step
functions
interact
with
athena.
We
have
integrations
with
tons
of
other
systems.