►
From YouTube: Dagster Day 2022
Description
Announcing Dagster 1.0 and Dagster Cloud launch
Timestamps:
0:00 Countdown
1:58 Introduction to Dagster Cloud (Nick Schrock)
6:45 Introducing Dagster 1.0 (Sandy Ryza)
8:25 Software-Defined Assets Explainer
10:02 Software-Defined Assets Demo (Owen Kephart)
17:00 Introducing Dagster Cloud (Nick Schrock)
23:03 Community Testimonials (David Wallace, Erik Reppel, Stephen Morton, Emily Hawkins, Zachary Romer, Vinnie Dalpiccol, Stephen Bailey)
27:58 Dagster Cloud Demo (Ben Pankow)
33:20 Dagster Cloud Pricing Explainer (Rex Ledesma)
36:22 Meet the Team Behind Dagster (Yuhan Luo, Sandy Ryza, Owen Kephart, Jamie DeMaria, Pete Hunt, Joe Van Drunen, Josh Braun, Daniel Gibson)
40:35 Live Q&A
Stand-alone videos:
Introducing Dagster 1.0 (Sandy Ryza);: https://www.youtube.com/watch?v=7OHCGU6iy_4
Software-Defined Assets Explainer: https://www.youtube.com/watch?v=In4CUoFKOfY
Software-Defined Assets Demo: https://www.youtube.com/watch?v=lRwpcyd6w8k
Introducing Dagster Cloud: https://www.youtube.com/watch?v=i9jETmOEmX0
Community Testimonials: https://www.youtube.com/watch?v=Zqti1FSR_iM
Dagster Cloud Demo: https://www.youtube.com/watch?v=Cdo2Rrt5pIQ
Dagster Cloud Pricing Explainer: https://www.youtube.com/watch?v=dKz4wDLGDnE
A
A
You'll
also
hear
from
a
few
members
of
the
team,
as
well
as
some
happy
customers
and
users,
then
we'll
wrap
up
with
some
live
q.
A
where
you'll
have
a
chance
to
ask
questions
of
the
daxter
team
so
be
ready
to
jump
in
the
chat
all
right
now
without
further
ado.
Let's
get
into
this,
shall
we
passing
it
on
to
nick.
B
Hi,
my
name
is
nick
schrock
and
I'm
the
ceo
and
founder
of
elemental,
the
company
behind
dagster,
the
next
generation
orchestration
platform
for
the
development
production
and
observation
of
data
assets
today
is
the
culmination
of
years
of
work
in
partnership
with
our
community
and
our
customers
to
everyone
that
has
worked
with
us.
Given
us
feedback
and
contributed.
Thank
you.
B
We
are
making
two
exciting
announcements
today:
the
release
of
dagster10
and
the
launch
of
daxter
cloud.
Let's
start
with
one
o
dagster
is
in
production
at
companies,
large
and
small
powering,
some
of
the
most
sophisticated
data
and
ml
platforms
in
the
world.
Our
1-0
release
is
a
new
milestone
of
maturity.
B
B
B
Every
company
is
becoming
a
software
company
and
then
in
turn
to
understand
itself
and
drive
better
decision
making
and
outcomes.
It
becomes
a
data
company.
That
means
every
company
has
a
data
platform,
a
set
of
tools
and
services
where
data
originating
from
sas
apps
operational
databases
and
the
outside
world
is
integrated,
transformed
and
managed.
B
B
We
see
these
teams
converging
on
a
set
of
principles,
one
they
use
cloud-native
infrastructure
and
manage
services
to
minimize
operational
burden
and
elastically
scale
with
their
business
two.
They
embrace
software
engineering
and
devops
processes
to
manage
complexity
and
three.
They
model
their
platform
in
terms
of
data
products
and
data
assets,
tools
aligned
with
these
principles,
cloud
data,
warehouses,
ingestion
tools,
dbt,
reverse
etl
tools,
are
gaining
wide
adoption
and
delivering
outsized
outcomes.
For
data
teams,
but
orchestration
has
been
largely
left
behind.
B
The
existing
landscape
is
a
menagerie
of
legacy
tools
and
mindsets
that
modern
data
teams
begrudgingly
use,
but
do
not
adore
and
orchestration,
is
critical
without
orchestration.
There
is
no
centralized
control.
Planes
teams
must
schedule
tools
independently
and
hope
and
pray
that
they
run
on
time
and
in
the
right
order.
B
When
things
go
wrong,
they
are
left
to
debug
and
diagnose
in
siloed
tools.
Data
platforms
without
orchestrators
aren't
really
platforms
at
all
they're,
an
operationally
fragile
siloed
collection
of
tools
where
there
is
a
constant
confusion
about
what
ran
what's
supposed
to
run
and
whether
things
ran
in
the
right
order.
B
B
It
provides
operations,
lineage
and
observability
for
your
data
assets
with
these
capabilities,
daxter
cloud
simplifies
and
consolidates
your
stack
of
tools
into
an
operationally
robust,
coherent
platform
that
is
both
easy
to
use
and
a
joy
to
work
in
we're
excited
to
celebrate
these
milestones
at
dexter
day.
I
am
now
going
to
introduce
you
to
sandy
rizza
the
lead
engineer
of
the
dagster
project.
He
is
going
to
discuss
daxter
100
and,
in
particular,
software
defined
assets.
Our
innovative
declarative
approach
to
orchestration.
C
I'm
sandy
and
I
lead
the
open
source
dagster
project,
I'm
extremely
excited
about
the
release
of
daxter
1.0
1.0,
doesn't
include
any
seismic
changes.
Rather
it's
a
major
milestone.
That
indicates.
We
put
the
finishing
touches
on
our
core
abstractions
data
teams
who
want
to
access.
What's
unique
about
dagstar
now
have
a
stable
foundation
to
build
on
top
of
dexter
is
different
from
other
orchestrators.
C
It's
the
first
orchestrator
that's
built
to
be
used
at
every
stage
of
the
data
development,
life
cycle,
local
development
unit
tests,
integration
tests,
staging
environments,
all
the
way
up
to
production,
and
it's
the
first
orchestrator
that
includes
software-defined
assets.
It
frees
up
teams
to
think
about
assets
and
let
the
orchestrator
manage
the
tasks.
C
Software-Defined
assets
are
a
new
concept
and
they
merit
some
explanation.
At
one
level,
dexter,
orchestrates
computations,
you
can
define
operations
that
you
want
to
execute,
specify
dependencies
between
them
and
schedule
them.
Dexter
can
replace
a
task
orchestrator
like
airflow,
without
requiring
significant
shifts
in
your
mental
model,
but
the
primary
job
of
data
teams
is
to
develop,
maintain
and
operate
data
assets.
C
An
asset
is
typically
a
database
table,
a
machine
learning
model.
A
report
any
persistent
object
that
captures
some
understanding
of
the
world
for
that
job.
Orchestration
is
just
a
means
to
an
end.
Software-Defined
assets
are
a
layer
that
can
directly
model
the
relationships
between
computations
and
the
data
assets
that
those
computations
produce.
D
Data
teams
manage
complex
webs
of
data
assets,
even
small
organizations
can
have
hundreds
of
tables
and
machine
learning
models
powered
by
thousands
of
lines
of
code
spread
across
a
variety
of
technologies.
This
easily
devolves
into
chaos.
You
don't
know
why
your
data
is
stale
and
you
end
up
with
conflicting
versions.
D
You
don't
know
what
to
trust
and
making
changes
feels
dangerous.
Much
of
this
chaos
comes
from
managing
data
in
an
imperative
way.
You
may
keep
your
assets
up
to
date,
using
an
orchestrator,
but
most
orchestrators
just
launch
tasks.
They
don't
know
what
an
asset
is.
Daxter
introduces
a
declarative
approach
to
data
orchestration.
D
The
solution,
software-defined
assets,
daxter
lets
you
declare
in
code
an
asset
you
expect
to
exist.
Instead
of
a
task,
you
define
the
function
to
run,
to
update
the
asset
and
the
names
of
the
upstream
assets
that
it
depends
on
now.
Instead
of
asking
whether
your
task
ran,
you
can
ask
whether
your
asset
is
up
to
date.
Instead
of
running
a
task,
you
can
update
an
asset,
you
can
identify
assets
that
don't
have
definitions
and
garbage
collect
them.
D
E
Alright,
so
we
just
saw
the
ideas
behind
software-defined
assets.
Now,
let's
see
how
they
work
in
practice,
this
demo
will
walk
through
how
to
get
started
with
this
programming
model
and
the
powerful
features
that
it
enables
through
it,
we'll
build
out
a
sample
data
platform
showing
how
to
create
our
own
assets
in
python,
as
well
as
how
to
load
them
from
external
tools
such
as
dbt.
E
With
that,
let's
jump
into
the
code.
Let's
start
from
the
very
beginning,
our
team
wants
to
take
some
raw
data
and
store
it
in
a
data.
Warehouse
we'll
start
with
a
few
imports,
then
define
a
function
to
generate
our
raw
data
to
keep
things
simple,
we'll
just
grab
some
data
from
wikipedia
using
pandas,
but
this
is
completely
arbitrary
python
code.
We
can
use
whatever
libraries
we
want
and
load
operate
on
and
return
data
of
any
type
by
annotating
our
function
with
asset
decorator.
It
becomes
an
asset.
E
The
key
or
name
of
the
asset
is
country
population
and
its
contents
are
computed
using
the
function.
We
just
defined
you'll
notice
that
we
haven't
yet
told
dagster
how
and
where
to
store
this
data.
We
could
include
that
logic
in
line,
but
it's
often
useful
to
keep
this
business
logic
separate
from
I
o
concerns
by
default.
Our
assets
will
be
stored
as
pickled
files
on
our
local
file
system,
but
this
behavior
is
completely
customizable
and
dextra
is
built
in
support
for
storing
assets
with
major
cloud
storage
systems
such
as
s3,
adls
and
snowflake.
E
E
We
can
monitor
its
status
directly
from
this
view,
seeing
that
a
run
is
currently
refreshing
it
or
we
can
jump
into
a
live
updating
timeline
to
view
detailed
logs
as
they
come
in
once
the
run
successfully
completes
our
data
is
stored
and
ready
to
be
used
from
here,
it's
natural
to
want
to
take
our
new
data
and
do
something
with
it.
For
example,
we
might
want
to
use
the
raw
population
data
and
aggregate
it
per
continent.
E
E
We
define
a
data
dependency
simply
by
adding
an
argument
to
our
function,
with
the
name
of
the
asset
that
we
want
to
depend
on
once
we
turn
this
into
an
asset.
Dijkstra
will
handle
the
rest
from
creating
a
lineage
link
between
these
two
assets
to
loading
the
contents
of
country
population
as
input
to
this
function
when
it
comes
time
to
run
it.
E
E
While
that's
running
it's
useful
to
take
a
step
back
and
consider
the
benefits
we're
already
getting
out
of
this
declarative
model.
After
just
a
few
lines
of
code.
At
no
point
in
this
process
did
we
need
to
think
about
tasks.
We
just
wrote
the
code
necessary
to
compute
the
contents
of
our
assets
and
the
orchestrator
was
able
to
string
these
definitions
together
to
execute
them
in
the
proper
order.
E
In
addition,
our
orchestrator
has
direct
understanding
of
the
assets
that
it's
responsible
for
giving
us
insight
into
how
they're
computed
and
how
up
to
date,
they
are
now
that
we
have
a
couple
of
assets
working
smoothly.
Let's
fast
forward
a
bit
and
see
how
this
scales
to
keep
everything
organized,
we'll
break
things
out
into
separate
files,
then
combine
our
assets
into
a
single
dagster
repository
first
we'll
load
in
the
population
data
assets
that
we
just
created.
E
E
Finally,
we'll
bring
in
a
machine
learning
team
that
will
use
the
data
processed
by
dbt
to
train
a
machine
learning
model.
These
assets
were
defined
in
python,
just
the
same
as
our
original
assets.
Once
again,
the
code
inside
these
assets
is
completely
arbitrary
and
you
can
use
whatever
tools
or
libraries
you
want
with
all
these
assets.
Added
to
our
repository,
we
now
have
a
number
of
assets
which
are
represented
as
pandas
data
frames
in
python,
then
serialized
this
files
on
localdisk.
E
E
You
might
have
noticed
that
changing
the
storage
location
of
our
assets
didn't
require
modifying
the
definitions
themselves.
This
makes
it
easy
to
write
unit
tests
for
our
assets
as
the
business
logic
stays
decoupled
from
the
external
systems
where
the
asset
will
be
stored.
Each
asset
can
be
assigned
a
different.
I
o
manager
allowing
precise
handling
of
storage
behavior
now
that
we
have
some
more
assets
defined.
Let's
head
back
to
the
ui,
the
first
thing,
we'll
notice
is
that
our
original
population
assets
now
have
some
downstream
dependencies.
E
If
we
go
to
the
global
lineage
graph,
we
get
a
total
view
over
all
of
the
assets
in
our
data
platform,
this
graph
spans
groups,
jobs
and
code
locations,
giving
you
insights
into
your
data
dependencies.
Regardless
of
how
you
choose
to
organize
your
code
and
execution
clicking.
Any
of
these
assets
will
bring
up
a
sidebar
containing
some
high
level
information
about
the
asset.
E
If
we
want
to
learn
more
about
a
specific
asset
such
as
our
population
summary
table,
we
can
search
for
it
by
name
and
get
taken
to
its
asset
details
page
from
here
we
can
see
information
on
every
time.
It's
been
materialized,
its
definition
and
which
assets
it
relates
to.
If
we
want
to
refresh
this
asset,
we
can
do
that
directly
from
this
page.
We
can
kick
off
a
run
that
re-materializes
just
this
asset,
or
we
can
select
its
upstream
dependencies
and
re-materialize
those
as
well
to
make
sure
it's
as
up-to-date
as
possible.
E
This
sort
of
workflow
is
also
easy
to
put
on
a
schedule.
If
we
go
back
to
our
code,
we
can
define
a
new
job
that
targets
a
selection
of
assets
that
are
upstream
of
our
population
summary
table
and
put
it
on
a
schedule
that
will
run
it
once
a
day
back
in
the
ui
we
can
see.
Our
asset
is
now
on
a
schedule
and
the
job
we
created
shows
up
ready
to
be
run.
B
B
B
Today
we
are
launching
two
hosted
options:
serverless
and
hybrid
serverless,
dagster
cloud
frees.
The
user
from
all
operational
burden
hosting
all
computation
spin-up
in
serverless,
is
totally
effortless
and
provides
an
unmatched,
fully
integrated
experience
just
write
a
python
file
and
we
do
the
rest,
no
kubernetes,
no
doctor
files,
no
provisioning,
but
not
all
users
can
offload
computation
to
a
hosted
service.
B
B
You
can
structure
your
code
for
local
development,
but
you
are
still
limited
in
what
you
can
test,
as
data
pipelines
depend
on
secrets,
services
and
data
that
are
not
accessible
or
runnable
on
your
laptop
teams
either
have
to
test
in
production
or
to
set
up
heavyweight,
inflexible.
Staging
environments
that
are
clumsy
to
use
and
costly
to
manage.
B
B
For
most
users,
they
have
a
main
branch
that
contains
their
code
and
a
production
deployment
where
that
code
is
deployed
with
branch
deployments,
you
follow
a
familiar
development
workflow.
You
create
a
new
branch
edit
your
code
and
create
a
pull
request.
Then
our
out-of-the-box
infrastructure
takes
over
on
every
push.
It
creates
a
lightweight
staging
environment
from
that
pr,
where
you
can
run
and
test
your
code.
B
B
Dagster
is
the
single
pane
of
glass
for
your
data
operations.
Dagster
cloud's
run
timeline
provides
a
full
overview
with
real-time
status
on
a
single
screen.
You
can
monitor
all
your
data
runs.
This
is
the
command
center
for
your
data
platform
operations
and
you
can
quickly
diagnose
problems
with
surgical
precision
with
dagster
cloud's
asset
graph.
You
not
only
have
full
asset
lineage
and
observability,
but
you
can
execute
operations
directly
in
the
orchestrator,
so
you
can
tame
the
sprawling
complexity
of
your
data
platform
with
dagster's
launchpad.
You
can
launch,
runs
with
a
powerful
but
easy
to
use.
B
Autocompleting
editor
usable
by
all
stakeholders.
You
can
kick
off,
runs
as
defined
or
easily
experiment
with
their
configurations
and
observe
the
result.
This
barely
scratches
the
surface
of
daxter's
ui
and
operational
capabilities,
debugging
runs
managing
backfills
monitoring
schedules
and
sensors.
Inspecting
your
assets
are
all
done
in
easy
to
use
powerful
uis
that
are
a
joy
to
use.
B
Security
and
access
management
are
built
in
with
soc2
compliance
and
sso
for
all
dagster
cloud.
Users
for
enterprise
users,
octa
and
active
directory
integrations
are
available,
along
with
role-based
access
control
and
the
ability
to
create
multiple
deployments
under
a
single
account
with
daxter.
Your
team
gets
to
regain
full
visibility
and
control
over
your
data
processes.
B
F
G
H
I
J
G
Out
of
the
box,
github
branch
deployments
is
a
brilliant
feature
for
developers
and
not
just
for
the
hardcore
developers,
but
for
the
the
sort
of
like
the
people
who
kind
of
wander
into
our
data
platform
and
want
to
deploy
a
python
process
being
able
to
make
code
changes,
push
them
into
a
pull
request
and
then
go
look
at
the
changes
in
dagster
and
run
a
dummy
job.
It's
a
game
changer.
I.
H
Think
there's
a
lot
of
promise
with
software-defined
assets,
flips
it
from
being
job-oriented
function,
oriented
et
cetera
to
being
asset
oriented.
It
frames
it
in
a
way
that
is
more
intuitive
to
keep
in
mind.
I
think
really
makes
dagster
as
a
framework
sort
of
stand.
Apart
from
the
other
existing
frameworks,
you
know.
L
G
You
a
simplicity
on
the
far
side
of
complexity,
which
is
to
say
that
there's
a
considerable
amount
of
complexity
and
really
great
engineering
work,
that's
gone
into
the
product,
but
in
the
end,
it's
so
that
the
interface
that
developers
have
to
use
and
that
coders
have
to
use
is
as
simple
as
possible,
but
no
simpler.
It.
K
F
L
G
F
J
H
Smartest
collection
of
human
beings,
I've
ever
met
in
my
entire
life,
no
joke
they
just
hustle.
They
just
know
how
to
get
stuff
done.
The
rate
at
which
the
dagster
team
is
shipping
things
is
phenomenal.
They've
definitely
proven
to
me
that
they're
capable
of
shipping
extremely
high
quality
software.
It's.
I
H
M
M
M
M
M
Dexter
cloud
is
built
to
scale
with
the
size
of
your
data
team
and
your
data
workflows.
Let's
take
a
look
at
what
the
development
process
on
a
more
mature
data
platform
looks
like
here.
We've
loaded
the
code
you
saw
owen
build
earlier,
which
utilizes
software-defined
assets
to
fetch
and
process
some
population
data.
M
M
Next,
we'll
add
a
country
gdp
asset,
we'll
pull
some
data
into
a
data
frame,
then
we'll
add
the
per
capita
gdp
asset
we
need
to
depend
on
our
existing
gdp
and
population
assets.
Dagstr's
programming
model
lets
us
specify
them
as
inputs
to
the
asset
function,
we'll
join
the
input
tables
compute
the
per
capita
gdp
and
return
it
we'll
create
a
branch,
commit
our
changes
and
push
up
to
our
github
repo.
M
Branch
deployments
run
against
a
forked
environment
that
we
can
test
against
to
gain
confidence
in
our
work.
Here,
we've
configured
our
branch
deployments
to
create
a
temporary
copy
of
our
cloud
environment.
This
job
creates
a
duplicate
of
our
snowflake
schema.
Our
pipelines
and
assets
are
parameterized
to
write
to
this
copy,
so
we
can
materialize
assets
in
our
branch
deployment.
Without
interfering
with
production,
I
can
now
go
ahead
and
kick
off
a
run
to
ensure
that
my
changes
work
and
a
context
nearly
identical
to
production.
M
Of
course,
these
branch
deployments
scale
with
the
size
of
your
team.
Every
pull
request
spawns
a
new
branch
deployment.
It's
easy
for
me
to
get
context
into
a
team
member's
change
as
a
reviewer
or
another
stakeholder
by
viewing
their
branch
deployment
where
I
can
explore
their
jobs,
their
assets
and
their
runs.
M
M
N
N
We
know
that
pricing
has
to
be
straightforward
and
predictable.
It
should
align
both
the
platform
and
the
customer
with
the
correct
incentives
we
wanted
practitioners
to
work
with
the
framework,
not
against
it.
We
never
want
you
to
compromise
your
engineering
principles,
for
example,
most
commercial
orchestration
solutions
charge
per
task,
but
this
encourages
users
to
consolidate
their
pipelines
into
a
few
large
tasks.
N
For
these
reasons,
we
price
on
the
duration
of
computations,
that
you
run
a
dax
or
cloud
and
naturally,
when
you
optimize
your
computations,
your
usage
also
decreases
you
pay
less.
If
you
run
fewer
computations,
you
pay
less.
If
you
speed
up
computations
and
most
importantly,
you
pay
as
you
go,
you
can
realize
incremental
value
without
committing
to
an
upfront
cost.
N
N
We
have
one
type
of
self-serve
account
which
includes
everything
that
you
need
to
run
single
production
deployment,
which
is
plenty
for
most
teams.
If
you
take
full
advantage
of
branch
deployments-
and
there
are
no
usage
limits
on
any
of
our
plans,
you
can
have
unlimited
users
and
run
any
amount
of
computation
to
meet
your
data.
Platform's
needs.
N
N
If
you
want
full
customization
of
your
data
plane,
we
have
that
option
with
our
hybrid
offering
we
host
and
manage
only
the
control
plane
for
daxxer.
While
you
select
and
manage
the
data
plane
in
your
infrastructure
again,
the
building
is
simple
too.
You
also
pay
a
usage-based
fee
in
either
case.
If
you're
a
larger
enterprise,
there
will
be
some
additional
things,
you'll
need
in
our
enterprise
offerings.
We
include
unlimited
production
deployments.
N
We
also
provide
built-in
security
along
with
single
sign-on
capabilities,
as
well
as
more
granular
user
roles.
Furthermore,
enterprise
plans
have
robust
service
level
agreements
for
those
mission.
Critical
implementations,
you'll
receive
higher
levels
of
support
and
a
dedicated
account
manager
for
your
team.
O
E
C
Spent
a
number
of
years
in
different
data
practitioner
roles
working
as
a
data
scientist
data,
engineer,
machine
learning,
engineer
in
all
of
those
roles
I
felt
like
I
was
fighting
against
the
tooling.
I
ended
up
spending
something
like
50
of
my
time,
not
actually
doing
the
job
that
I
was
hired
to
do,
but
building
tools.
That
would
help
me
and
my
team
do
that.
I.
R
S
C
O
C
T
E
R
P
S
R
A
B
Yeah
I'll
take
that
you
know
we
try
to
meet
our
users
where
they
are
and
kind
of
our
frame
of
mind
is
for
data
practitioners.
You
want
to
offload
as
much
ops
as
possible
who
wants
to
deal
with
it.
So
we
recommend,
if
you're
kind,
of,
have
no
constraints
and
you
can
host
your
you
allow
us
to
host
your
computation.
B
R
A
C
B
I
don't
have
those
stats
off
the
top
of
my
head.
I
don't
know
if
you
do
this
andy,
but
you
know.
I
think
that
the
the
core
of
the
system
is
mostly
maintained
by
elemental
team
members,
and
but
there
are
incredibly
important
contributions
made
by
the
community,
especially
in
the
space
of
integrations.
We
have
entire
integration,
libraries
like
the
microsoft
teams
and
mlflow
and
a
bunch
of
other
really
important
libraries
that
are
completely
community
driven,
yeah
databricks.
Anything
else
add
to
that.
R
Absolutely
so
in
in
hybrid
mode,
you
have
an
agent
in
your
cluster,
that's
responsible
for
any
parts
of
the
of
the
system
that
require
access
to
your
code
or
your
data.
So
what
we
did
for
branch
deployment
is
we
made
the
agent
able
to
track
sort
of
which
of
your
branches
are
currently
active
and
spin
up
compute
to
temporarily
sort
of
dynamically
up
and
down,
based
on
what
branches
are
open
and
serve
code?
R
Based
on
that
also,
when
you
spin
up
code
in
a
branch
deployment,
you
can
sort
of
branch
within
the
code
to
know
whether
or
not
you're
in
a
branch
deployment
and
access
different
resources.
So
the
agent
kind
of
injects
different
environments
for
these
different
branch
deployments
versus
full
deployments
and
sort
of
keeps
track
of
which
ones
are
active,
so
it
doesn't
use
too
much
compute
at
once.
It's
kind
of
a
quick
overview
of
how
it
works.
B
I
also
have
to
say
what
this
team
did
on
branch
deployments
on
hybrid.
Is
an
engineering
marvel
just
incredible
work?
You
know
this
type
of
workflow
is
available
in
front-end
frameworks
like
for
sale,
which
we
use,
but
to
bring
it
to
the
orchestration
layer
for
data
platform
teams
was
a
achievement,
as
you
can
tell
I'm
extremely
excited
about
branch
deployments.
A
B
B
Some
people
think
it
should
only
be
for
ad
hoc
data
exploration
and
analysis,
but
there
are
interesting
projects
like,
for
example,
paper
mill
which
allows
you
to
invoke
a
notebook
like
a
function.
We
actually
do
have
an
integration
for
that
called
dagstermill
and
yeah,
and
I
think,
in
terms
of
future
roadmap,
we
are
thinking
about
how
to
you
know,
bring
in
notebooks
into
our
system
in
a
more
seamless
way,
but
no
promises
there.
C
B
Well,
it
exists,
people
use
it.
I
think
the
most
notable
thing
about
it,
which
I
think
is
cool,
is
that
all
you
do
is
write
functions
which
take
data
frames
and
spit
out
spark
data
frames,
and
it
handles
all
this,
these
ops
for
you,
because
if
you've
ever
used,
emr
or
one
of
these
systems,
you
have
to
figure
out
how
to
copy
code
into
a
cluster
and
do
all
sorts
of
crazy
stuff,
and
this
just
handles
that
for
you,
and
it
also
has
interesting
integration
points.
This
thing
called
step
launcher.
B
C
Yeah
just
there's
a
few
levels
to
our
spark
integration,
so
we
have
a
core
dagster
pi
spark
integration,
library,
which
offers
what
nick
was
talking
about,
and
then
we
have
particular
integrations
built
with
cloud
platforms
that
allow
you
to
run
spark.
So
we
have
a
pretty
mature
and
old
databricks
integration,
as
well
as
a
emr
integration.
B
That
would
be
extremely
useful
because
you
can
have
one
code,
location
to
say
corresponds
to
an
ml
team
which
has
dramatically
different
dependency
needs
than
another
team
which
doesn't
need
those
ml
libraries.
So
we've
really
thought
about
environmental
separation,
both
between
teams,
as
well
as
between
the
user
code
and
the
system
itself.
A
A
B
B
And
daggett
great
question:
currently
those
are
supported
index
for
cloud
almost
certainly
our
back
will
be
exclusively
a
commercial
feature.
You
know,
one
of
the
reasons
is,
actually
it
is
complicated
and
we'll
get.
You
know,
there'll
be
tons
of
demanding
features
and
we
want
to
be
able
to
move
super
quickly
on
that.
A
P
A
B
R
Yeah,
so
that
part
of
the
system
is
is
pluggable
in
serverless
as
well,
and
so
so,
if
you
want
to,
if
you
want
to
start
the
outputs
of
your
assets
or
your
jobs
in
a
an
s3
or
a
cloud
provider
which
you
might
want
to
do,
because
they
can,
they
might
be
relatively
sensitive.
Depending
on
your
use
case,
you
can
absolutely
still
plug
that
into
serverless
and
kind
of
include
the
credentials
for
those
providers
in
the
image.
C
Yeah,
so
we've
seen
a
number
of
users
do
it.
I
think
when
we
talk
about
transitioning
from
ops
and
jobs,
we
wouldn't
necessarily
call
it
that,
because,
when
you're
using
software-defined
assets,
you're
often
using
ops
and
jobs
as
well,
every
software-defined
asset
has
an
op
inside
of
it,
which
is
the
computation,
and
you
can
build
jobs
out
of
software-defined
assets
that
execute
those
or
materialize
those
assets
on
a
fixed
schedule
or
in
response
to
some
sort
of
event.
C
We
have
a
guide
in
our
documentation,
I'd
recommend
taking
a
look
at
it.
It's
called
upgrading
two
software
defined
assets
and
it
has
a
lot
of
the
details
of
what
this
process
looks
like,
but
ultimately
it's
an
incremental
process.
So
if
you
want
to
bring
software-defined
assets
into
the
picture,
you
don't
have
to
suddenly
throw
away
all
your
code.
C
C
The
the
last
thing
I
would
add
is
that
we
don't
we
don't
expect
all
of
our
users
to
transition
from
ops
to
software-defined
assets
for
a
couple
reasons.
One
is
that
there
are
some
situations
that
we
believe
ops
are
best
suited
for
and
another
one
is
also
you
know,
are
here
and
here
to
stay
and
there's
no,
you
know
it
sort
of
comes
down
to
whether
the
value
of
software
defined
assets
is
useful
for
you.
B
Other
said
one
thing
to
that:
you
know
we
don't
we
want
to
make
it
clear.
It's
not
like
this
is
like
a
parallel
two
parallel
system.
It's
a
layered
system,
so
we
had
ops.
We
had
jobs,
which
is
kind
of
manually
constructing
a
graph
of
ops
for
execution.
We
had
a
thing
called
asset
catalog
and
you
can
another
way
to
think
about.
Is
software
defined
assets?
Is
this
layer
which
beautifully
composes
those
components?
So
you
don't
have
to
build
a
dag
on
your
own
acids,
contain
ops.
B
It's
a
great
question,
so
the
github,
the
the
github
integration,
is
kind
of
the
easiest
way
to
go,
because
we
have
a
github
app
and
it's
one
click
and
whatnot.
But
you
know
we
built
it
in
a
layered
way.
That's
just
built
on.
You
know
the
available
cli
tools
we
have,
which
can
be
in
a
relatively
straightforward
way,
integrated
with
git
lab
you
know
or
any
ci
tool.
Really.
You
know
some
of
our
early
access
customers.
You
know
we
we're
pitching
them
branch
deployments
they're
like
the
greatest
thing.
I've
ever
heard.
C
B
B
So
in
that
demo,
for
example,
that
ben
showed-
and
we
have
an
internal
demo
about
this
too-
which
we'll
publicize
it
actually
works
beautifully
with
dbt,
because
every
time
you
push
up
in
a
branch
deployment,
you
can
kick
off
a
job
which
clones
a
snowflake
schema.
And
then
you
can
make
dbt
point
at
that.
So
you
effectively
have
kind
of
an
isolated
environment
for
your
dpt
jobs,
and
then
we,
you
know
either.
C
Yeah
to
add
to
that
in
dbt,
it's
typical
to
configure
a
set
of
targets
which
essentially
correspond
to
different
environments,
that
you
would
run
your
your
models,
so
local,
some
staging
environment
or
ci
environment
production,
and
you
can
basically
use
resources
in
dexter
to
line
up
your
different
deployments
with
those
with
those
targets.
So
you
might
use
a
production,
dbt
target
in
production
and
a
ci
dbt
target
in
your
branch
deployments.
A
B
Yeah
well,
we
actually
really
think
that
branch
deployments
will
eliminate
the
need
for
tons
of
teams
to
support
a
staging
environment,
and
you
know
if
you
need
a
permanent
staging
environment
or
you
have
a
bunch
of
teams
that
each
have
their
production
deployments.
You
know
we
think
it
makes
sense
to
have
that
be
an
enterprise
plan.
B
A
B
Yeah,
for
those
of
you,
I
need
to
know,
don't
know.
Apache
arrow
is
a
really
exciting
technology.
It's
a
standard
for
in-memory
representation
of
data
frames,
and
it's
really
cool
allows
you
to
like
transition
data
frames
from
one
compute,
one
compute
runtime
to
another
without
doing
any
copies,
and
it
has
fast
and
memory
execution.
B
Given
all
that,
I'm
not
sure
if
we've
seen
any
apache
aero
io
managers,
but
the
the
in
which
case
it
would
persist
effect,
I'm
getting
the
weeds
too
much
anyway,
there's
no
the
system
very
pluggable
and
would
work
with
technologies
that
support
apache
aero,
and
you
know,
I
think
you
know,
but
we
have
no
kind
of
explicit
framework
support
for
it.
You
know
the
users
have
to
determine
their
own
storage,
your
own
storage
formats
as
plugable
all
right
now.
I
said
something
that
sandy
needs
to
correct.
C
B
B
You
know
we
think
branch
deployments
are
not
only
an
immediate
game
changer,
for
you
know
the
entire
data
ecosystem,
but
we
think
it's
like
a
really
interesting
point
where
you
can
integrate
with
other
tools
and
implement
all
sorts
of
interesting
features.
I
think
that
we're
also
going
to
be
working
on
making
serverless
more
accessible
and
more
usable
in
more
contexts.
B
So
right
now
you
know
and
discuss
this
later.
Serverless
is
you
know,
early
access
and
we're
rolling
it
out
in
tranches,
because
our
first
you
know
we're
hosting
people's
computation.
We
want
to
make
sure
we're
providing
a
good
service
and
we
think
we'll
be
investing
a
lot
there
and
then
also
more
and
more
enterprise
capabilities.
We
want
dax
to
be
as
accessible
as
possible
in
as
many
contexts
as
possible
and
that's
kind
of
the
focus
here.
Do
you
want
to
talk
about
the
open
source
roadmap.
C
So
yeah
for
open
source.
I
think
there
are
a
few
areas
that
we're
excited
about
and
we'll
have
a
lot
clearer
picture
in
a
couple
weeks
after
the
dust
has
settled
on
1.0,
one
of
them
is
there's
a
set
of
observability
improvements
that
have
been
requested
for
a
long
time
that
we're
excited
to
have
the
chance
to
finally
improve
on
around
surfacing
configuration
and
daggett,
making
it
easier
to
understand
what
sensors
are
doing.
C
A
second
area
is
integrations,
one
that
we're
particularly
excited
about
improving
is
our
airflow
integration.
So
it's
pretty
common
people
are
really
excited
about
using
dagster
but
they're
at
an
organization
that
has
airflow
installation
and
they're,
not
realistically
gonna
move
all
their
pipelines
from
from
airflow
to
dagstr
in
one
fell
swoop.
C
We've
laid
the
framework,
but
I
think
there's
a
ton
of
opportunity
to
make
dexter
is
a
really
powerful
tool,
not
just
for
running
stuff,
but
for
kind
of
this.
This
full
end
to
end
way
of
managing
aspects
of
your
data,
so
having
the
ability
to
build
sensors
and
memorization
that
are
all
asset
focused
is
a
is
a
really
excited,
exciting
direction
for
us.
A
A
B
Thanks
for
all
these
incredibly
high
quality
questions,
it's
always
fun
to
see
the
community
so
engaged
and
asking
such
great
questions.
So,
what's
next
year,
so
you
know
we
have
launched,
you
can
sign
up
for
tax
for
cloud
today.
You
can
sign
up.
You
know,
kick
off
your
own
deployment,
you
know
and
you
have
a
30-day
trial.
So
we're
super
excited
about
that
and
just
a
reminder,
you
know
we're
we're
rolling
out
two
variants
of
this.
One
is
serverless
which
is
totally
hosted
by
us.
B
That
is
early
access,
meaning
that
you
might
be
put
on
a
waitlist
as
we're
rolling
out
the
technology
incrementally.
But
we're
super
excited
about
that
and
then
there's
hybrid,
where
you
bring
your
own
compute
and
there's
no
wait
list
there
at
all
or
anything.
You
know
this
just
beginning.
We
are
really
excited
to
have
this
cloud
product
out
in
public.
B
We're
really
excited
to
get
that
fast
feedback
loop
with
all
our
users
and
I
think
you'll
see
our
innovation
speed
increase
and
you
know
we
just
have
a
tight
end,
a
ton
of
exciting
improvements
ahead.
I
just
want
to
say
thank
you
to
the
elemental
team
who
has
put
their
blood,
sweat
and
tears
into
this
thing.
B
Well,
I
don't
know
if
anyone's
crying,
but
you
know
metaphorically
and
we're
bleeding,
but
the
team
has
been
working
incredibly
hard
and
I
think
that
it
shows
today
both
with
the
features
we're
pushing
out
as
well
as
this.
You
know,
I'm
in
a
tv
studio
right
now.
This
is
crazy,
but
just
an
amazing
event
today,
and
then
you
know,
I
just
want
to
thanks
everyone
for
coming
and
thanks
for
taking
the
time
to
listen
what
we
have
to
say
and
we
hope
you
enjoy
our
product-
we're
incredibly
excited
about
it.
Thanks.