►
From YouTube: Catalog & Cocktails: Modern Data Stack: Technology, Methodology, or both? w/ Nick Schrock
Description
The modern data stack is often defined by the type of technologies that exist within it. Cloud-based, open source, low/no code tools, ELT, and reverse ETL. But surely there’s more to it… isn’t there?
What holds the modern data stack together and makes it the architecture of choice for so many data-driven enterprises? Join Tim, Juan and special guest, Nick Schrock, founder of Elemental and creator of Dagster and GraphQL, to chat about all things MDS.
A
B
Hey
tim,
I'm
juan
cicada,
I'm
the
principal
scientist
here
at
data.world
and
it's
wednesday
middle
of
the
week
end
of
the
day
close
to
end
of
the
day
and
always
a
pleasure.
I
am
so
lucky
that
we
get
to
have
our
pause
and
chat
about
data.
I'm
lucky
that
data.world
lets
us
keep
doing
this
as
part
of
our
day
job.
So
thank
you.
Dated.World
and.
A
B
Data
and
drinks,
and
today
we
have
a
guest,
a
very
special
guest,
who
has,
I
would
say,
impacted
the
lives
of
all
developers
nowadays,
because
some
of
the
stuff
that
he
has
created,
I
hear
it
probably
multiple
times
every
day,
and
that
is
nick
schrock,
who
is
the
founder
of
elemental,
creator
of
baxter
and
also
graphql
nick.
It's
a
pleasure
to
have
you
here.
Thank
you
so
much
for
joining
us
thanks.
C
C
Phase
so
here
we
are
yeah
and
I
guess
I'm
just
toasting
to
the
health
of
my
newborn
son
and
my
wife.
I
you
know
I
used
to
scoff
at
people
who
kind
of
did
this
before
I
had
kids,
but
now
you
know
now
I'm
one
of
those
people,
but
you
know
we
had
a
challenging
pregnancy
and
childbirth.
He
was
born
early,
only
five
pounds,
seven
ounces
and
now
he's
a
big
chunky.
Boy
he's
huge
now.
So
it's
been
quite
the
reversal,
so
I'm
toasting
to
his
health
and
yeah.
B
B
B
A
The
first
whiskey
is
angel's
envy,
so
gonna
give
that
a
try
and
I
wanna
cheers.
First
of
all
to
you
know:
family
health.
That
is
always
key
and
so
really
great
to
hear
nick
that
things
are
going
well
for
you
all
there,
and
I
will
also
actually
cheers
to
dagster,
and
I
have
not
also
been
paid
to
say
that,
but
we're
actually
implementing
daxter
over
a
data.
world,
and
I
know
that
we
love
it
so
far.
There's
a
ton
of
excitement
in
the
community
around
dagster.
B
That's
true:
sales
is
community,
that's
great!
There
we
go.
They
did
that
world
too,
is
also.
We
have
a
big
community
around
it
too.
So
I
I'm
drinking
a
mexican
I'm
calling
it
a
mexican
old
fashioned.
I
had
some
agave
and
there
was
some
mezcal
and
I
had
some
red
bitters
and
that's
why
it's
kind
of
reddish
in
here.
So
it's
actually
really
really
nice
and.
B
Go
with
you,
cheers
on
family
came
out
this
holiday
weekend
spent
time
with
my
whole
family
and
my
brother,
who
has
a
newborn,
and
it
was
just
cool
to
get
everybody
together
at
least
the
same
roof,
so
aries
cheers
cheers.
So
we
got
our
warm
up
question
here,
which
is
describe
your
reaction
upon
glancing
at
matt,
turk's
latest
data
landscape
diagram,
and
I
think
it's
going
to
show
up
right
here.
C
B
I'm
I'm
totally
with
you
like
my
my
reaction
is,
oh,
my
god,
really,
because
I've
been
collecting
these
I've
give.
I
have
some.
I
have
a
talk
that
I
give
where
I'm
showing
how
this
increases
right
he's
been
doing
it
for
five
six
years
now
and
it's
like
well.
The
first
time
it
came
up
was
like.
Oh
that
makes
sense,
and
then
it
just
gets
bigger
and
bigger
and
bigger
and
more
complicated,
more
boxes.
More
logos
are
repeated
and
I'm
like
what
the.
A
C
Kind
of
cool
there's,
so
many
people,
you
know
taking
their
shots
this
week
and
there's
so
much
activity
and
interest
in
the
space-
and
you
know
behind
each
of
these
logos-
there's
a
story
of
some
founder,
some
group
of
people
who's
trying
to
make
their
stake
in
the
world.
So
on
that
on
that
aspect,
I
think
it's
pretty
cool.
A
C
B
B
A
B
Which,
which
is
a
good
segue
to
the
conversation
you
want
to
have
here
today?
So
let's,
let's
dive
into
this
conversation,
though,
and
I'd
like
to
kick
it
off
with
this,
with
an
honest,
the
honest
obs
question,
so
when
we
hear
about
the
modern
data
stack
right,
the
first
thing
that
comes
to
mind
is
a
list
of
technologies,
a
growing
list
of
technologies.
We
just
see
here
right
and
all
cloud-based
and
there's
storage,
there's
open
source,
there's
low
code,
no
code,
there's
e-e-l-e-t-l-e-l-t,
transform
reverse
ct,
analytics
wow
right.
C
I
actually
think
it
started
out
as
a
stack
of
technologies,
but
it
is
a
methodology
and
a
mindset
and
the
way
I
frame
it
in
my
head
is
that
we're
effectively
rebuilding
data
infrastructure
from
the
ground
up
in
the
cloud
era
and
also
what
I'll
call
the
modern
era
modern
being
defined
as
every
enterprise
in
the
universe
has
complex
data
needs,
they're,
ingesting
from
all
sorts
of
sas
services
and
being
able
to
effectively
use
data
as
kind
of
a
base
label,
level,
capability
and
expectation.
So
what
does
that
mean?
C
One?
Is
that
the
cloud
data
warehouse
or
maybe
a
lake
house
we
can
get
to
that,
but
some
sort
of
centralized
store
like
that
is
kind
of
the
center
of
a
company's
data
universe
they
use
and
bias
toward
using
managed
services
and
that
there's
this
software
engineering
mindset
when
it
comes
to
data.
I
think
that's
a
really
interesting
thing
to
dig
into,
because
I
think
there's
a
lot
of
misconceptions
around
that.
So
you
know
I
think
the
short
answer.
C
I
think
the
modern
data
stack
started
out
as
a
like
fairly
narrow
definition,
where
it's
like.
Okay,
you
choose
a
cloud
data
warehouse.
You
have
dbt
an
ingest
tool,
a
bi
tool,
that's
a
modern
data
stack,
but
when
people
encounter
the
reality
of
the
world,
their
needs
expand
and
they
grab
for
more
tools.
And
you
know,
like
recently,
the
modern
dataset
has
expanded
to
include
reverse
etl
right,
for
example,
and
I
think
that
kind
of
expansion
is
going
to
occur,
which
means
it's
not
a
static
set
of
technologies.
B
Okay,
so
let's
dig
into
kind
of
the
the
technology
part
you
started
doing
this,
so
I
want
to
talk
about
technologies.
I
want
to
talk
about
methodologies
when
it
comes
to
technology.
You
said
kind
of
it
all
started
out
with
a
cloud
data
warehouse
dbt
ingesting,
which
I
would
guess,
is
the
the
e
and
the
l
right
and
some
analytics
right
so
yeah
these
four
things
is
kind
of
how
it
all
started,
and
this
is
basically
just
we're
reinventing
ourselves.
B
B
C
C
C
C
B
Okay,
that
that
that's
a
key
one,
okay,
I
want
to
dive
into
that,
but
let's
I
want
to
keep
exciting.
So
we
start
with
these
four
cloud:
data
warehouse,
dbt
and
just
analytics.
How
is
this
extending
so
when
you
say
it's
not
a
static,
it's
so
it's
dynamic,
reverse
utl
is
something
that
we're
seeing
here.
B
Where
else
is
this
extended?
Where
is
this
going
and
part
of
it
is?
How
are?
Is
it
aligned
to
use
cases
so,
for
example,
I
got
this
type
of
use
case.
I'm
going
to
go.
Do
I'm
going
to
go
in
one
direction
versus
this?
Other
use
case
goes
into
this
other
direction,
where
I
would
not
need
some
tool
whatever
how.
C
Are
you
my
the
way
I
see
it
is
that
companies
are
building
up
their
data
infrastructure
from
scratch,
they're
cloud
first
and
they're
they're
answering
the
questions
that
you
answer
in
order,
meaning
that
the
first
thing
you
do
is
that
you
count
things
like
understanding
very
busy
basic
metrics
about
your
company
or
your
enterprise.
You
know
how
many
users
do
we
have?
C
What
is
our
revenue
like
basic
business
metrics
and
that's
in
order
to
answer
that
question?
You
can
just
answer
that
with
ingest
into
a
cloud
data
warehouse
dbt
on
top
bi
and
you're
done
right
in
order
to
do
this
basic
counting,
but
that's
step
one,
not
the
final
stage.
So
then
it's
like
oh
interesting,
we've
ingested
our
data
from
all
our
different
sources.
We
want
to
re-inject
that
into
those
sas
products
so
that
you
can
surface
the
right
information
to
stakeholders
in
their
native
tool.
C
Then
you
have
reverse
etf,
but
then
the
people
who
build
these
data
platforms.
They
actually
want
to
expand
things.
So
maybe
they
want
to
build
ml
and
experimentation
platforms
right,
it's
very
naturally,
adjacent
because,
for
instance,
in
ml
most
of
the
work
is
etl
anyway
right.
Most
of
the
work
is
in
the
data
processing.
So
there's
natural
bleeding
between
those
two
use
cases.
C
You
know
things
just
expand
beyond
the
scope
of
only
sql
computation
in
general,
people
need
to
write
custom
code
to
do
lots
of
things
and
they
need
to
you
know
I
must
like,
among
our
user
base,
I'm
always
kind
of
not
shocked
but
like
it's
continually
interesting,
all
the
different
use
cases
that
people
come
up
with
where
they
scratch
together
data
platforms,
we're
like.
Oh,
we
have
these
contractors.
They
need
to
insert
this
stuff
into
a
google
sheet.
Then
we
need
to
write
some
code
to
do
that.
C
We
need
to
match
that
up
with
our
payroll
system
and
on
and
on
and
on
and
on
so
you
know
to
me.
The
modern
data
stack
is
simply
following
the
natural
evolution
of
kind
of
what
happens
within
a
company
when
you're
starting
to
expand,
and
then
it's
like.
Oh,
we
have
so
much
data
that
we
need
to
catalog
it
right
and
then
you
start
looking
at
cataloging
tools.
Oh
there's
enough
stakeholders
here
and
there's
enough
teams
that
we
need
to
start
doing
data
lineage.
A
So,
even
though
you
know,
we've
started
this
conversation
a
little
bit
more
from
a
technology
perspective
and
I'm
sure
we'll
explore
a
little
bit
more
there.
You
know
you're
talking
about
use
cases
in
a
use
case.
Progression
you're
also
talking
about
sort
of
a
maturity
curve
here
where,
as
you
enter
in
maybe
you're,
starting
off
more
with
descriptive,
analytics
and
then
you're
moving
into
some
more
of
the
prescriptive
things
you're
trying
to
build
data
applications,
you
know
you're
trying,
maybe
you're
now
starting
to
roll
this
out
to
a
bigger
company.
A
So
now
it's
not
just
about
one
group
in
the
company
anymore.
It's
about
how
do
we
sort
of
federate.
You
know
our
stack
and
federate
our
governance
sort
of
elsewhere
in
the
company.
You
know.
Is
it
like
why?
Why
do
we
keep
on
kind
of
coming
back
to
the
like
the
diagram?
The
technologies,
though,
like
is
it?
Does
that
kind
of
become
a
an
anchoring
point
where
we
can
at
least
kind
of
talk
about
like
the
components
as
they
grow,
or
you
know,
I'm
curious
why
methodology
and
and
maturity
haven't
been.
C
Yeah,
I
I
don't
know
if
I
have
any
profound
thoughts
in
that
and
but
engineers
who
are
dealing
with
this.
They
like
talking
about
concrete
things
that
solve
concrete
problems
rather
than
kind
of
you
know
abstract.
You
know,
process
and
methodology
stuff.
It's
kind
of
like
yeah
processing
methodology
stuff
is
something
that
like
mbas,
come
in
and
do
right.
C
They
like
come
in
and
do
a
swot
analysis
right
or
something
like
that
and
you're,
like
your
most
engineer's
relationship
with
the
swot
analysis,
is
like
how
like
guilfoyle
and
the
silicon
valley
show
interact
with
him.
You
know
no,
no,
that
reference,
I
think,
there's
like
some
level
of
skepticism
towards
these
more,
like
high-level,
abstract,
think
pc
ways
of
approaching
things.
But
you
know
okay,
but
in
you
know,
they'll
ask
people
will
ask
like
hey.
C
You
know,
there's
kind
of
like
a
note
when
you
see
it
like
feeling
for
people
and
they're
like
oh,
this
technology
feels
native
in
the
modern
data
stack.
So
I
try
to
listen
to
those
users
and
you
know
kind
of
abstract
away
the
general
or
abstract
way
abstract
out
the
general
principles
that
apply
there.
But
I
think
it's
kind
of
a
dispositional
thing
where
you
know
the
people
who
engage
in
this
space
are
pretty
you
know
by
engineering
in
general,
are
quite
literal,
usually
and
very
valuable
value
and
thing
oriented
right.
A
B
B
But
but
I
feel
that
we
that
we
that
we
have
some
methodology
in
there,
but
but
then
sometimes
we
just
like
just
give
me
the
next.
I
just
want
to
go
solve
this
problem
and
just
get
the
next
tool
about
it
and
then
that's
going
to
kind
of
expand
to
more
more
more
of
these
blocks
that
we're
going
to
see
in
matt's
turk
thing
right
right.
Well,
I
I'm
trying
to
solve
this
very
specific
problem,
so
I'm
going
to
create
this
new
tool
about
it,
but
we're
not
zooming
out
and
realized.
B
C
Yeah,
I
think
your
your
example
is
interesting,
which
is
code,
review
and
source
control,
which
I
think
most
engineers
would
often
describe
primarily
in
terms
of
the
tools
they
use
as
opposed
to
the
process,
because
the
process
and
the
tools
are
interlinked
right.
So
you
know
it's
like,
of
course
we
use
github
or
some
other
tool
for
code
review,
but
they
think
of
it
as
a
tool,
not
a
process.
So
I
think
that
that's
like
an
aspect
of
this.
That's
going
on.
C
B
C
C
Podcast
a
couple
weeks
ago,
alongside
scott
from
brooklyn
data,
and
he
was
describing
a
data
platform
with
without
orchestration,
it's
like
a
bunch
of
kids
in
a
sandbox
and
they're,
not
even
talking
to
each
other
they're
just
doing
their
own
thing.
But
what
you
really
need
to
do
is
coordinate
and
work
together
and
that's
really
where
orchestration
comes
in.
So
in
my
mind
you
know
orchestration
is
you
know,
there's
a
couple
things
you
need
to
do.
One
is
you
want
to
add
operational
robustness
to
your
existing
tools,
so
without
orchestrator?
C
What's
interesting?
Is
that
we've
kind
of
regressed
in
the
modern
data
we're
talking
about
the
monetary
stack,
mostly
if
someone's
using
5tran
dbt
cloud
and
a
reverse
etl
tool,
they're
now
stuck
in
a
world
where
they
have
overlapping
cron
jobs
right
where
you
just
have
to
like
hope
and
pray
that
one
works
after
the
other?
If
something
goes
wrong,
you
have
no
tool
or
you
can
debug
things
across
those
tools.
You
have
three
sandbox
operational
tools
and
you're
like
scratching
through
logs
and
each
of
them
and
figuring
out.
C
C
You
can't.
What
do
you
do
right?
You
have
to
write
some
code,
yeah
yeah.
You
have
to
write
some
code
to
do
any
any
number
of
things,
so
this
is
really
where
orchestration
comes
in.
You
know
I'd
like
to
say
that
the
orchestration
really
comes
in
when
you
need
to
start
assembling
your
modern
data
stack
into
a
platform
where
there's
a
single
kind
of
you
know
operational
plane
of
glass.
B
So
you're
so
with
the
modern
data,
stack,
we're
building
a
platform
and
we
can
make.
We
can
start
really
simple
where
I
mean,
if
I'm
just
counting
how
many
users
and
what
are
their
metrics
and
stuff
like
that,
and
it
doesn't
need
to
be
up
to
date
immediately
in
real
time
like
yeah.
I
can
just
put
this
stuff
together.
That's
fine!
But
the
moment
where
I'm
starting
to
expand
and
the
moment
where
just
there's
just
more
complexity
about
how
things
need
to
go
flow
and
you
need
to
have
more
security
about
likes.
B
C
C
You
know
I
think
it's
a
critical
building
block
and
I
think
not
incorporated
early.
You
set
yourself
up
for
pain
later
to
me,
it's
kind
of
like
oh
we're
not
going
to
like
we're
working
in
a
programming
lesson,
programming
language,
but
we're
not
going
to
use
classes
for
now,
because
it's
like
over
engineering
for
now
like
well.
Actually,
you
should
probably
like
use
the
building
blocks
you're
going
to
use
from
day
one.
C
C
Like
go
into
two
different
tools
to
debug,
something
like
like,
I
think
they,
you
know
with
the
right
combination.
It
should
be
what
you
know.
In
my
opinion,
the
reason
why
orchestrators
haven't
been
used
earlier
in
the
development
life
cycle
of
platforms
in
general
is
that
they've
been
really
difficult
to
spin
up.
They
take
a
lot
of
like
operational
overhead
and
they're.
So
therefore,
like
the
cost-benefit
ratio,
kind
of
like
moved
down
farther
in
the
maturity
cycle,
but
there's
still
benefit
to
adopting
it.
C
So
a
lot
of
what
we've
been
focusing
on
at
dagster
is
kind
of
making
the
spin
up
super
easy,
making.
It
super
lightweight,
just
like
writing
code,
but
building
managed
service,
and
so
that
and
we'll
actually
be
announcing
that
tomorrow,
more
broadly
early
access
to
our
cloud
platform
promise.
I
won't
make
this
whole
thing
a
sales
pitch,
but
you
know,
but
I
think
it's
like
to
summarize.
I
think
it's
like.
C
C
A
Yeah
now
that
makes
a
lot
of
sense,
and
I
I
I
like
your
way
of
approaching
that
that,
like
orchestration,
has
a
ton
of
benefit
and
if
it
was
easier,
if
the,
if
the
bar
to
start
doing
orchestration,
could
be
sort
of
you
know,
hurdled
over
in
an
easier
way
more
quickly,
then
it
actually
makes
sense
to
bring
it
earlier
on.
Your
sort
of
modern
data
stack
journey.
C
C
B
C
Go
around
and
kind
of,
say:
oh
you
know
like
if
you're
using
kind
of
like
maybe
start
with
rest,
and
then
you
know,
move
the
graphql
if
things
get
more
complex
and
we
got
a
ton
of
pushback
from
our
community
on
that,
because
they're
like
listen,
you're,
making
our
job
so
much
harder
like
just
tell
people
to
use
graphql
from
day,
one
like
it
ends
up
with
better
systems.
It's
not
that
much
like
like
you're
underselling,
it's
not
much
much
more
difficult
to
use
or
anything.
In
fact,
I
think
it's
easier.
C
You
know
we
got
a
ton
of
pushback.
That
is
this,
like
just
start
using
what
you're
going
to
use
from
day
one
because
it
has
all
these
implications
around
the
tools
and
processes
built
around
it.
You
know,
and
so
the
cost
of
undoing
it
even
like
a
month
later
is
actually
quite
high
and
I
think
orchestration
properly
conceived
will
be
like
that
or
just
like.
Is
this?
It's
just
what
you
do
right,
because
even
if
you
just
have
two
tools
having
overlapping
crown
makes
no
sense,
things
should
run.
A
Right
yeah
right
and
then
you
don't
have
to
migrate
later,
and
that
makes
a
ton
of
sense,
and
you
know
this
whole
conversation
has
me
thinking
a
little
bit,
though
about,
like
you
know
it's
not
always
as
clear
how
to
get
started
on
this.
Modern
data
stack
journey
to
do
things
right
and
to
get
these
tools
implemented
if
you're,
a
larger
company
or
you
have
a
more
complicated
environment
you
know
is,
is
modern
data
stack
kind
of
just
really
for
the
the
smaller
companies
or
the
younger
companies
that
are
earlier
on
their
journey.
C
I
don't
think
so,
but
it
kind
of
depends
on
if
you're
it
kind
of
goes
back
to
the
original
premise
of
the
discussion
which
is
like.
Is
it
a
methodology
or
a
very
narrowly
prescribed
set
of
technologies?
And
so
that's?
What
I
think
is
interesting.
Like
should
companies
to
use
the
framing
of
like
move
to
the
cloud
move,
to
manage
services
and
apply
software
engineering
mindset
to
their
data
processes?
C
I
would
say:
yes,
you
can
call
it
the
modern
data
stack
or
not,
but
that's
like
an
undeniable
win,
and
then
you
also
see
companies
incrementally
adopting
technologies
in
the
modern
data
stack
within
their
organizations
as
well
so
yeah.
I
think
it
makes
a
lot
of
sense
and
then
the
other
thing
that
you
know
like
like,
for
example,
one
of
the
reasons
why
snowflakes
doing
so
well
is
that
they're
doing
such
a
great
job
of
lifting
and
shifting
workloads
from
on-premises
data
warehouses
to
them.
C
You
know
they
were
kind
of
playing
a
different
game.
The
whole
time
where
I
think
a
lot
of
engineers
in
the
valley
would
be
like,
oh
you're,
going
to
get
people
to
migrate
from
hadoop
to
snowflake.
It's
like
guys
like
people,
99
of
the
world,
is
still
on
their
like
on-premises
data
warehouse
and
have
no
ability
to
use
hadoop
at
all.
Like
they're,
like
jumping
straight.
C
Exactly
so
yeah,
I
think
that
there's
going
to
be
a
similar,
you
know
there's
going
to
be
a
similar
thing,
because,
as
people
adopt
the
cloud
data
warehouses,
then
they
have
the
same
problems.
Everyone
else
they're
going
to
be
grabbing
for
the
next
tool.
So
I
think
that
these
technologies,
the
the
shift
towards
this
style
of
data
infrastructure,
is
inexorable
both
for
green
field
and
existing
companies.
B
B
Yes,
you
can
use
the
modern
data
stack
if
your
goal
is
to
move
to
the
cloud
right,
you
want
to
manage
services
and
you
want
to
be
able
to
apply
software
engineering
principles
to
the
data.
If
that's
what
you're?
That's
that's
what
you
want
to
go.
Do
then
yeah
you
can
get
on
the
modern
data
stack
like
that.
It's
not
just
for
smaller
companies
who
are
growing.
I
think
that's
a
really
good
way
of
thinking
about
that
and
which
leads
me
kind
of.
B
I
think
our
initial
question
here
is
it's
just
a
list
of
technologies
or
there's
a
methodology?
I
think
if
the
answer
is
both
but
the,
but
the
question
here
now
is
what
are
the
do's
and
the
don'ts
like.
So
I
think
there's
two
things
one.
We
already
talked,
if
you
have
a
smaller,
if
you're
a
smaller
company
right,
you're,
just
growing
you're,
probably
going
to
do
the
simplest
thing,
just
count
things
and
you're
gonna.
You
don't
need
all
this
stuff.
B
I
think
the
part
of
that
methodology
is
you
choose,
there's
like
a
a
minimal,
a
simple,
minimal.
Modern
data.
Stack
that
you
that's
that's
the
minimal
thing
you
gotta
do
right.
The
four
things
I
think
right:
the
cloud,
the
the
warehouse,
the
the
dbt,
the
e
and
the
l
and
then
analytics
right,
that's
the
minimal
and
then
part
of
the
methodology
is
depending
on
the
use
cases
you
would
want
to
go,
do
a
reverse
etl
or
want
to
have
more
catalog
and
so
forth.
B
C
Yeah,
I
mean
to
me
the
important
what
I
have
learned
through
my
years,
but
is
that
having
an
incremental
process
in
place
so
that
every
stage
of
a
migration
and
moving
from
one
to
technology
to
another
feels
like
you're
just
hiking
up
a
hill
rather
than
like
jumping
over
a
canyon?
You
know
so
always
construct
these
migration
products
such
that
there's,
always
intermediate
checkpoints.
You
can
stop
assess,
understand.
C
A
Well,
if
it
fits
very
well
with
some
of
the
things
that
that
show
up
on
our
show
a
lot
around,
like
don't
bowl
the
ocean,
take
a
use
case.
First
approach:
you
know:
can
you
iterate
your
way
to
value
here
and
you
know
are
those
some
of
the
key
tenets?
You
would
point
to
here
for
sort
of
a
modern
data,
stack
methodology,
and
is
there
anything
you
would
add
to
that.
C
C
C
Yeah,
so
what's
interesting
is
that
I
I
think
that
his
work
and
his
definition
of
data
ops-
and
you
know
he's
like
been
pushing
the
data,
ops,
manifesto
and
all
this
stuff
and
what
he
said
very
early
on,
resonate
with
me
a
lot
and
he
really
talked
about
the
software
engineering
unification
of
data.
That's
kind
of
like
he's
like
analytics
is
code.
That
was
his
mantra,
and
I
really
think
that
a
lot
of
you
know,
data
ops
has
become
such
a
buzzy
term
that
there's
tons
of
different
definitions
of
it.
C
I
think
it's
like
hard
to
grasp
onto,
but
really
I
think
that
the
ideas
in
data
ops
have
really
been
kind
of
co-opted
in
the
modern
data
stack
like
the
modern
data
stack
kind
of
wave,
so
the
the
entire
software
engineering
mindset.
Let
me
give
you,
let
me
put
some
mean
on
some
phones
there.
So
what
do
I
mean
by
software
engineering
mindset?
C
C
They
are
getting
analysts
who
used
to
write
sql
and
save
them
in
files
on
their
desktop
to
use
git
and
make
their
kind
of
analytics
work
product
part
of
a
software
engineering
process
that
it
is
like,
in
my
view,
like
the
ultimate
data,
ops
technology
and
the
chris
berg
framing
of
the
term.
So
yeah
I
mean
that's
kind
of
that.
That's
kind
of
my
take
on
that.
B
Minute,
32
and
40
seconds:
that
was
a
really
good
analogy.
You
did
right
there
in
that
definition.
I
really
like
this
about
especially
what
dbt
is
such
a
very
popular
thing,
which
kind
of
quick
parentheses
we're
having
a
special
catalog
in
cocktails
next
week.
Doing
the
dpt
coals
happy
hour
edition.
So
this
is
a
good,
so
the
official
happy
hour
is
gonna,
be
a
catalog
in
cocktails
podcast.
B
If
there
is
one
thing
that
we're
going
to
define
that
the
modern
data
stack
changed,
the
world
was
literally
bringing
in
these
software
engineering
practices
into
data,
and
I
think
that's
what's
going
to
make
sure
that
we're
going
to
have
a
better
world
of
data
instead
of
all
the
crap
that
we
have
to
go
through
right
now,
just
because
we
just
didn't
have
good
didn't,
have
any
practices
around
data
anyways.
That's
my
my
my
little
aha
moment
here.
A
Absorbing
it,
you
know
one
thing
this
brings
up
for
me
and
you
know
nick
interesting
in
your
way
in
on
it.
We've
discussed
this
similar
topic
with
a
few
other
of
our
guests.
Is
that
when
you
say
what
you
said
about
dbt
right
about
how
you
know
it's
it's
turning
the
sequel
that
analysts
were
writing
into
something.
That's
like
now.
It's
a
github
project,
you're
wrapping
it
with
ci
cd
you're,
applying
software
best
practices
to
it.
A
A
You
know
how
do
we
start
to
take
like
the
the
every
man
and
the
every
woman
and
turn
them
into
a
an
engineer
right
and
like
to
me
these
things
they
feel
like
they're,
pulling
in
different
directions
and-
and-
and
I
know,
there's
some
overlap
there
right
in
terms
of
how
they
could
work
together
but
like
what
is
your
thought
process
on?
That?
Does
one
of
these
things
win?
Do
they
find
a
way?
Did
we
find
a
way
to
get
oil
and
water
to
kind
of
mix
together.
C
A
And,
and
for
for
me,
it's
the
user
experience
paradigm
right
it,
it
is,
can
you
make
it
so
that
something
so
that
you
don't
have
to
write
the
code
right
and
maybe
I'm
already
implying
what
maybe
the
marriage
can
be
here
right
but
like
if
I
can,
if
I
can
drag
and
drop
and
say,
I
want
this
to
join
with
this,
and
I
want
this
to
be
the
outcome
and
I'm
never
having
to
write,
join
and
where
and
select,
and
things
like
that
like
to
me,
the
no
code
experience
low
code.
Experience
is
like.
B
About
it,
because
I'm
just
afraid
that
we're
gonna
it
works
for
all
the
kind
of
general
cases,
but
then
the
world
is
not
just
one
simple
case:
there's
all
these
corner
cases
and
then
then
you're
not
going
to
be
able
to
go
deal
with
those.
So
then
you're
just
going
to
hack
up
some
stuff
and
then
you're
going
to
not
have
good
practices
around
it.
C
Yeah,
so
that
all
makes
sense,
I
I
think
there
is
definitely
a
space
for
low
code,
no
code
solutions
there's
a
few
requirements.
One
is
they
have
to
be
composable
with
other
tools,
so
I
think
historically,
when
people
think
low
code
no
code,
they
think
of
these
completely
siloed
systems
that
are
not
composable
at
all
and
try
to
reinvent
the
entire
world.
C
C
There's
there's
nothing
to
prevent
you
from
writing
that
tool
and
then
literally,
it
saves
a
file
and
then
the
entire
software
development
process
takes
over
and
it
runs
through
a
cicd
pipeline
and
all
that
stuff,
and
that's
what
I
mean
by
composable,
meaning
that
the
tool
that
you
described,
that
would
just
generate
a
sql
statement,
or
maybe
a
sql
statement
like
in
some
sort
of
context,
but
at
its
core.
That's
what
it's
doing
that
can
be
incorporated
into
a
software
engineering
process
and
be
composed
in
other
tools.
C
C
That
is
just
too
bespoke
and
too
custom
and
there's
no
other
way
to
do
it
by
making
it
composable,
meaning,
like
have
an
engineer,
be
able
to
like
write
that
one
and
then
kind
of
publish
it
to
that
business
user.
So
they
can
plop
that
in
so
you
can
kind
of
how
to
be
part
of
an
integrated
platform
instead
of
this
complete
silo.
That's
way
over
here.
If
that
makes
sense,.
B
I
fully
agree,
but
for
that
basically
the
whole
low
code.
No
code
is
just
a
higher
level
abstraction
and
we
need
to
be
able
to
go
make
sure
that
we
can
compile
that
down
to
something
that
we
can
actually
go
use,
so
so
because
that
that's
something
that
scares
me
too,
is
that
you'll
have
all
these
apps
saying
like
well,
you
did
all
this
work,
but
I
I
I
want
to
go
edit.
It
afterwards
right,
yeah
right
to
go
use
that
work
outside.
B
If
it's
either
I
want
to
go
edit
it
myself
and
then
that
can
go.
I
can
compile
it
or
bring
it
up
to
a
higher
level
abstraction.
So
I
think
that's
something
very
that's
crucial.
That
goes
back
into
your
composability.
I
mean
yep.
I
all
I
mean
query
languages
I
mean
sql
is
a
beautiful
language,
because
it's
all
composable
right
tables
in
and
table
come
out.
I
can
go
use
this
very
easily.
I
think
that's
something
that
when
it
comes
to
declarative
languages,
is
something
that
changed
the
way
how
we
think
about
computation.
B
C
B
C
So
I
think
there
will
be
some
consolidation,
but
not
in
the
way
that,
like
like
databricks
and
snowflake
want
it
I
mean.
Maybe
it
will
end
up
like
that,
but
I
don't
think
that's
a
good
outcome
for
users
where
effectively
one
of
those
two
becomes
a
new
oracle
right,
they're,
both
kind
of
there's
this
titanic
battle,
that's
happening
between
snowflake
and
databricks,
where
they
both
want
to
become
the
oracle
of
the
cloud,
meaning
like
a.
C
The
new
monolith,
so
I
think
there
needs
to
be
simplification
and
consolidation,
but
without
siloed
monolithic
architectures,
where,
like
early
in
your
career,
you
have
to
say,
like
I'm,
a
snowflake
person,
and
I
can't
work
anything
else
like
that
sucks.
You
know
it's
kind
of
going
back
in
time
when,
like
there
was
like
oracle
people
and
microsoft
people-
and
I
don't
think
that's
great,
so
I'm
a
big
fan
of
open
standards
that
kind
of
can
span.
C
Now,
how
call
it
consolidation
call
it
unification,
call
it
simplification,
but
something
has
to
happen
in
order
to
start
to
make
sense
of
this
world
so
to
speak,
and
I
do
think
that
and
we'll
see
how
that
plays
out.
I
have
my
own
theories
about
that,
but
we
you
know,
I
think
that
there
will
be
a
unified
data
management
platform
where
you
can
plug
in
different
solutions
for
different
kind
of
components
of
it,
whether
it's
data
quality.
B
Well,
let
me
let
me
go
follow
up
on
this,
so
kind
of
connecting
a
lot
of
dots
of
previous
conversations.
We
had
andy
palmer
from
tamer.
I
think
last
last
episode
and
he's
totally
there's
the
monolith
is
dead
and
and-
and
I
I
can
imagine
that
you
can
see
a
snowflake
trying
to
consolidate
and
trying
to
make
that
one-stop
shop
right,
the
the
modern
oracle
and
the
cloud
type
of
thing
and
and
some
people
was
like-
I
just
want
to
have
one
thing
and
I
don't
know
deal
with
everything.
B
That's
probably
yeah,
there's
reasons
on
that.
But
then,
if
you
don't,
if
we're
not
going
to
go
down
that
route,
then
we
need
to
have
things
need
to
go.
Work
together,
talk
to
talk
to
each
other,
and
I
think
standardization
is
something
that
is
going
to
be
key
there.
I
think
so.
I
asked
myself
what
is
standardization
around
the
modern
dataset
look
like,
and
this
is
something
I've
seen
talks
and
I've
actually
had
some
conversations
with
bob
moogly.
I
read
the
former
ceo
of
snowflake
and
he
is
so
I
mean
he's.
B
He
is
really
betting
on.
We
need
the
metrics
layer,
which
is
another
thing,
I'd
love
to
go,
talk
about,
and
standardizations
and
and
knowledge.
Graphs
like
those
are
the
three
things
that
he's
focusing
on
and
but
it's
it's
kind
of
easier
said
than
done
that
we
always
think
about
that.
Xkcd
comic
right!
You
got
14
standards.
We
need
one
standard
to
go
rule
them
all.
We
got
15
standards
right,
so
how
much
of
that
is
actually
going
to
happen?
I
mean,
is
it?
B
Is
it
just
going
to
be
like
little
cohorts
of
folks
like
we
work
great
together?
Is
there
really
going
to
be
a
standard
center
standard?
That's
going
to
connect
all
types
of
modern
dataset
tools
through
metadata
or
we're
just
kind
of
we're
just
trying
we're
just
I
don't
know
not
thinking
about
this,
but
it's
really
not
going
to
happen
and
we're
just
going
to
see
informatica
2.0
coming
around
we're
going
to
go,
buy
all
these
companies
and
investors
are
putting
money
all
over
the
places
right
and
they're.
B
C
So
I
have
my
own
theories
around
that.
Obviously,
but
there's
a
lot
of
other,
like
people
can
say
like
hey,
it's
going
to
be
the
data
catalog
that
you
know
you
unify
us,
everything
or
other
people
will
say
like
actually
the
cloud
data
warehouse.
It
actually
makes
sense
for
them
to
build
a
silo
because,
like
that's
where
all
the
data
is
going
and
you
can
know
stuff
there,
metrics
player
is
a
candidate.
So
yeah
we'll
see.
C
B
C
B
C
C
So
in
a
lot
of
these
domains,
like
I
don't
know
if
we
know
the
best
way
to
do,
cataloging
or
lineage
or
orchestration,
you
know
like
we
just
don't
like.
We
have
a
bunch
of
different
people
with
their
own
thesis
around
things,
but
you
know
to
me,
like
the
standards
should
kind
of
you
know,
there's
like
a
proper
process
with
building
open
standards,
and
the
first
thing
is:
you
have
to
prove
the
core
underlying
value
of
it.
C
That
actually
works,
and
you
know-
and
I
think
if
you
try
to
standardize
stuff
too
early-
you
kind
of
hamstring
yourself
and
don't
have
enough
flexibility.
You
know
like
like
to
the
graphql
analogy
right.
We
came
out
with
graphql
as
an
open
standard,
but
we
had
worked
on
it
for
three
years
inside
of
facebook
and
built
the
whole
company's
infrastructure
around
it.
So
we
at
least
had
like
one
data
point
about
yeah.
A
That
makes
sense-
and
I
think
this
has
given
us
a
lot
to
think
about,
because
you
know
we're
thinking
about
how
things
fit
together,
we're
thinking
about
they
how
they
evolve.
This
is
the
beginning
of
lots
of
thinking
and
lots
of
conversations
to
come
about
how
things
should
move
forward
and
hey
nick.
It's
been
exciting
to
chat
about
how
this
can
evolve
together,
because
I
think
there's
a
lot
there's
a
lot
of
moving
pieces
and
also
a
lot
of
exciting
opportunity
here.
So.
C
C
A
Looking
for
new
approaches,
new
tool
sets,
I
mean,
like
you,
look
at
something
like
dbt
right
could
dbt
have
been
as
successful
six
or
seven
years
ago.
You
know,
I
don't
know
right.
It
feels
like
the
conditions
might
actually
be
in
a
better
place
for
something
like
that
to
get
adopted
today.
After
you
know,
the
whole
hadoop
movement
happened
after
some
of
these
things
have
happened
right.
C
Yeah,
I
totally
agree,
I
think,
there's
also
like
an
influx
of
another
dynamic
here.
That's
kind
of
opening
up
the
space
is
that
companies
are
so
desperate
for
data
engineering
data
engineers
that
they're
kind
of
like
moving
folks
laterally
over.
I
saw
this
happen
in
front
end
10
years
ago,
when
people
kind
of
more,
like
mainline,
normal
engineers,
kind
of
moved
into
javascript
and
front
end
to
make
sense
of
that
world,
and
then
all
sorts
of
interesting
innovation
came
out.
C
I
think
it's
similar
things
happening
in
data
right
now,
like
we
talked
to
lots
of
people
who
were
are
looking
for
a
solution
for
orchestration
and
they
kind
of
find
action
like
wow.
This,
like
makes
kind
of
sense
to
me
because
they
came
from
more
traditional
software
engineering
domain,
and
this
is
intuitive
for
them
so,
and
I
think
it's
happening
all
over
the
place
so
yeah.
It's
super
exciting.
B
C
A
C
No,
I
don't
think
so,
because
I
think
there
is
space.
We
see
this
a
lot
in
terms
of
I'll
call.
It
green
field
within
brown
brownfield,
meaning
that
new
projects
with
new
teams
who
are
empowered
to
kind
of
make
their
own
infrastructure
choices
within
the
context
of
a
larger
enterprise.
There's
much
more
flexibility
around
that
now.
A
A
Yeah
sure
so
this
is
the
part
where
we
get
to
tell
you
nick,
if,
if
we
got
all
the
right
takeaways
here
so
first
of
all,
you
know
the
modern
data
stack.
We
really
talked
about
at
its
core
being
sort
of
this.
This,
the
e
of
the
l,
the
data
warehouse,
the
t
and
the
bi,
but
then
there's
all
these
other
aspects
that
are
really
interesting
around
it,
such
as
orchestration.
B
A
As
catalog
such
as
so
on
and
so
forth,
right
machine
learning,
ai
et
cetera,
but
you
sort
of
talked
about
a
process
here
right.
The
first
thing
you
do
is
you're
kind
of
you're,
counting
things
you're,
implementing
your
basic
metrics,
then
maybe
you're,
trying
to
figure
out
how
you
can
take
some
of
these
metrics
and
push
it
back
into
some
other
systems.
A
Right,
maybe
that's
where,
like
things
like,
reverse
etl,
come
to
play,
you're
trying
to
get
smarter
about
your
data,
now
you're
evolving,
from
descriptive,
analytics
to
prescriptive
or
predictive
type
analytics,
and
your
company
is
growing
in
size.
It's
getting
more
complicated.
Your
data
uses
cases
and
quantity
and
complexity
are
getting
more
complicated.
Okay,
you
need,
you
know,
catalog,
you
need
lineage,
you
need
monitoring.
You
need
all
these
different
things
right,
so
there's
a
process
here,
there's
a
methodology
behind
all
these
different
technologies.
A
And
ultimately
you
talked
about
how
the
methodology
has
to
be
incremental.
You
said
you
know
you
want
to
hike
up
a
hill
instead
of
jumping
over
a
cannon,
a
canyon
and
you,
you
said
evolutionary
means
for
revolutionary
ends,
which
I
think
is
a
great
phrase.
There
and
one
I
like
to
to
use-
and
I
actually
have
a
little
backlog
of
t-shirts,
that.
A
A
B
My
takeaways
here
so
first
of
all,
the
modern
data
stack.
It's
a
it's,
a
this
natural
evolution
that
a
company
goes
through
right
and
it's
core
thing
is
about
bringing
software
engineering
practices
into
data,
we're
we.
We
are
in
this
process
of
reinventing
the
data
ecosystems
going
to
the
cloud
and
your
your
warehouse
or
lake
houses
at
the
center.
All
these
key
things
around
it.
I
love
your
def.
Another
definition.
B
Modern
data
stack
is
it's
hipster
data
people
for
the
hipster
data,
people
love
that
and
it's
just
it
started
as
a
stack,
but
it
is
evolving
into
some
methodology.
When
we
talk
about,
is
it
just
for
smaller
companies?
The
answer
is
no.
I
mean
if
you're
moving
to
the
cloud
and
you're
man,
you
want
to
go
to
work
with
managed
services.
B
You're
thinking
about
adding
software
engineering
principles
of
data
doesn't
matter
if
you're,
small
or
large,
whatever
you're
part
of
the
modern
data
stack,
so
those
legacy
companies
can
do
it
too,
and
when
it
comes
to
consolidation-
and
I
like
how
you
said
it
call
it-
consolidation,
unification,
standardization-
whatever
it
does,
it
needs
to
be
simple-
is
it
gonna?
Is
there
gonna
be
a
unified
data
management
platform?
Is
that
gonna
be
a
monolithic
or
a
bunch
of
players?
Working
together
I
mean
it's
still
really
early,
we're
figuring
this
out.
B
C
B
Well,
this
is
this
has
been
a
fascinating
discussion
and
I'm
really
excited
to
to
actually
meet
you
in
person,
hopefully
one
day
and
and
have
keep
having
this
conversation
and
have
this
conversation
live.
So
I
want
to
throw
it
back
to
you.
Finally,
two
questions
advice:
what's
your
advice
about
data
about
life
or
whatever
and
second,
who
should
we
invite
next.
C
So,
guest
suggestions-
I
was
thinking
about
this
before
the
show,
and
you
know
I
think
you.
I
hear
lots
of
vendors
and
tool
authors
on
these
shows
all
the
time
and
I
think
we
should
talk
to
more
power
users
about
the
problems
they're
trying
to
solve
and
an
early
user
dagster,
who
also
was
a
dev
addict
in
his
previous
life.
This
guy
named
david
wallace,
who
works
at
duchy.
C
Yeah,
I
guess
this
is
gonna,
probably
be
cheesy,
but
the
optimize
for
the
people
around
you
in
your
life
both
and
your
personal
life
and
your
work
life
I'm
very
blessed
to
we.
We
literally
had
all
hands
five
minutes
before
it
ended
five
minutes
before
I
hopped
on
this
show,
and
it's
I'm
really
proud
of
the
team
we've
built
and
the
people
on
it,
and
it's
just
a
joy
to
work
with
them
all,
and
it
makes
us
all
this
so
much
more
fun.
C
A
C
B
Thank
you
so
much
appreciate
it
and
just
quickly.
Next
week
we
got
a
lot
of
stuff.
It's
dbt,
coalesce
and
we're
having
the
official
happy
hour
is
going
to
be
a
special
episode
of
cataloging
cocktails
with
claire
look
and
meet
karia
from
the
zebra
and
that's
going
to
be
on
tuesday.
Live
at
2
p.m,
pacific
4
p.m.
Central
I'm
also
going
to
be
at
the
data,
governance
and
information
quality
conference
live
and
we're
going
to
do
some
live
talking
to
folks
at
in
person
conference.