►
Description
Nick Schrock, Founder and CTO at Elementl explains the fundamental challenge for data teams: they are mission-critical to modern organizations and yet face deep problems related to their ability to manage their data platforms and achieve high levels of productivity.
Dagster was designed to help engineers shift from being stuck in a 'scarcity mindset 'to being highly productive and being able to adopt an 'abundance mindset'.
A
Hi
everyone
thanks
for
coming.
My
name
is
Nick
Schrock,
the
CTO
and
founder
of
Elemental,
the
company
behind
dagster
and
welcome
to
airflow
migration
day.
Airflow
is
a
widely
used
technology.
That
was
a
big
step
forward
for
the
data
ecosystem
when
it
was
released,
however,
times
have
changed
and
technological
choices
must
change
with
them.
The
demands
on
data
teams
in
today's
world
are
intense
and
they
have
overwhelmed
airflow's
ability
to
serve
those
teams.
It's
time
to
move
on
to
better
Solutions.
A
The
goal
of
this
event
is
to
demonstrate
why
this
is
true:
why
the
next
generation
orchestrator
is
Daxter
and
to
show
if
you
are
stuck
with
airflow,
how
you
can
migrate
both
with
process
and
with
our
amazing
toolate
you're,
going
to
hear
a
lot
directly
from
our
customers
and
when
you
have
a
product
people
love.
It
makes
putting
this
event
on
this
type
of
event,
really
fun
and
really
effective,
and
it
just
requires
a
simple
strategy:
just
put
our
customers
on
camera
and
let
it
rip
there
are
a
ton
of
great
takeaways
from
those
talks.
A
But
two
facts
stick
out.
The
first
is
that
in
2023,
data
platforms
are
utterly
essential
to
the
success
of
modern
organizations,
and
the
second
point
is
that
the
teams
charged
with
building
and
deploying
those
data
platforms
face
fundamental
and
essential
challenges
executing
on
this
Mission,
especially
when
they
use
airflow.
A
We'll
talk
about
three
foundational
challenges
that
teams
typically
face.
The
first
developer
experience
the
second
fragmentation,
both
across
tools
and
teams,
and
the
third
is
technical
debt.
Let's
start
with
developer
experience,
you'll
hear
all
of
our
customers
talk
about
the
Quantum
Leap
Improvement
in
developer
experience
when
they
adopt
dagster.
When
we
talk
to
most
data
teams,
the
state
of
their
developer
life
cycles
is
pretty
abysmal.
Most
tools
were
not
built
with
the
local
development
life
cycle
in
mind.
They
were
not
built
with
testability
as
a
first
class
concept.
A
This,
in
turn,
frees
the
engineer's
mind
to
think
Bolder
and
bigger
productivity
is
not
just
making
existing
tasks
more
efficient
in
software
engineering,
it
makes
entirely
new
things
possible
productivity.
What
another
way
productivity
transforms
the
mindset
of
an
engineer
from
scarcity
and
fear
to
abundance
and
confidence.
A
This
leads
to
the
next
challenge
facing
data
teams.
Data
platforms
are
typically
extremely
fragmented
across
tools
and
Technologies
platform
teams
integrate
dozens
of
tools
across
teams
in
an
attempt
to
assemble
them
into
a
coherent
data
platform.
In
order
to
synchronize
operations,
they
might
adopt
airflow,
but
they
are
quickly
drowning
under
the
complexity
once
they
adopt
it.
We
are
left
with
a
distorted
data
platform
with
disconnected
tools
and
siled
teams.
A
Developer
experience
does
an
important
thing
and
you'll
hear
this
from
our
customers.
It
actually
incentivizes
teams
across
the
org
to
adopt
the
technology
to
improve
productivity
and
happiness,
but
that
unleashes
new
organizational
Dynamics
you'll
hear
our
customers
use
the
word
collaborative.
You
don't
hear
that
word
often
enough
when
it
comes
to
data
platforms
with
dagster.
The
organization
has
shared
context
on
all
assets
under
management
and
a
single
pane
of
glass
to
debug
technical
issues
that
crosses
organizational
boundaries.
A
A
Today,
you're
going
to
hear
from
customers
who
have
overcome
that
challenge,
Stephen
from
whatnot
is
going
to
explain
how
they
transitioned
their
mission,
critical
multi-personal,
multi-persona
data
platform
to
dagster,
while
incrementally
building
trust
in
the
tool
and
incrementally
delivering
value
at
every
step
along
the
way,
then
we're
going
to
hear
from
goo
from
group
1001
and
he's
going
to
describe
this
magical
experience
of
using
our
airflow
adapter
layer,
which
is
releasing
to
10.
Today
it
is
a
toolkit
where,
with
minimal
work,
you
can
import
your
existing
dags
into
dagster
and
execute
them
natively.