►
From YouTube: Hello Big Complexity: Is Your Modern Data Stack Ready? | Nick Schrock & Scott Breitenother
Description
Tune in to MAD Data Podcast to hear from data engineers, leaders of data teams, and everyone who has a stake in their data quality. Be sure to check out our upcoming guest lineup at: https://databand.ai/mad-data-podcast/
. . .
Nick Schrock, Founder & CEO of Elementl, and Scott Breitenother, Founder of Brooklyn Data Co., discuss the evolution of data from Big Data to Big Complexity – what's next now that the data industry has solved the problem of data storage? While the modern data stack has become embraced as every data team's "must-have" to address 'modern data problems,' Nick and Scott muse on the struggles that continue to plague data teams and the next wave of potential in data infrastructure innovation. With one problem solved, a new era of possibility and complexity is now unleashed.
. . .
Listen to MAD Data on -
Simplecast: https://mad-data.simplecast.com/
Google Podcast: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS9OU3FITE5kZQ?sa=X&ved=0CAMQ4aUDahcKEwjA-qzZoKX0AhUAAAAAHQAAAAAQAQ&hl=en
Amazon Music: https://music.amazon.com/podcasts/2969426c-15f3-49e9-a54a-13edcf6e6768/mad-data-a-data-quality-podcast-by-databand-ai
Spotify: https://open.spotify.com/show/3V0dhexP4cSYIRBlKLf9KV
. . .
View our upcoming guest lineup at: https://databand.ai/mad-data-podcast/
. . .
#dataquality #dataops #datapipelines #dataobservability #datareliability #dataengineering
B
C
Cool
well,
I
mean
first
of
all,
I
find
it
extremely
unfair
that
nick
has
like
this
professional
setup
with
his
microphone
and
like
key
lights.
Like
I
feel
like,
I
have
to
turn
my
lights
up
more
I
mean
I,
you
know,
I
don't
I'm
not
even
working
with
great
raw
material
to
start
with,
so
I
really
got
to
do
whatever
I
can
so
yeah,
I'm
I'm
scott,
I'm.
C
You
know
founder
of
brooklyn
data
and
we
are
a
small
but
rapidly
growing
consultancy
that
focuses
on
building
out
the
modern
data
stack
for
companies
of
all
shapes
and
sizes,
so
we're
a
fully
distributed
team.
You
know
we,
you
know,
we've
been
using
dbt
and
snowflake
and
all
those
tools
since
the
early
days
and
and
I'm
just
really
excited
about
all
that's
going
on
in
the
space.
D
Well,
apparently,
the
most
important
fact
about
me
is
that
I
have
a
new
mic,
which
scott
is
very
jealous
of.
So
thank
you,
scott
for
pointing
that
out.
No,
I
I
will
not
do
asmr
the
whole
time.
No,
so
my
name
is
nick
schrock,
I'm
the
ceo
of
elemental,
which
is
the
company
behind
dagsar,
very
quick
background
on
me.
Bulk
of
my
crew
was
at
facebook.
D
A
Awesome,
I
loved
hearing
all
that
and
really
what
started
this
conversation.
Actually
I
wanted
to
say
maybe
it
was
like
two
three
months
ago
we
were
discussing
the
idea
of
the
modern
data
stack
and
is
it
maybe
not
as
complete
as
it's
been
portrayed
as
and
we
talked
about
why
don't
we
bring
a
few
perspectives
into
the
room,
obviously
nick
from
coming
from
the
orchestration
angle
and
then
scott
you
working
with
analytics
and
also
with
a
variety
of
use
cases
at
brooklyn
data
and
then
harper?
A
Are
you
coming
in
from
the
angle
of
data
observability
and
how
that
fits
into
the
modern
data
stack?
So
I
wanted
to
maybe
talk
really
quickly.
First,
like
history
about
the
evolution
of
the
data
life
cycle
and
really
how
we
got
here,
should
I
maybe
should
we
start
with
you
scott,
like
give
us
your
take
on
how
the
life
cycle
of
data
got
to
where
we
are
today.
C
Okay,
really
starting
with
the
easy
ones
right
the
so
this
that's
a
that's
a
really
great
question
I
mean
I,
I
think
you
know
everybody's
probably
talked
about
the
journey
from
etl
to
elt
a
bunch,
but
I
think
to
summarize
it
it.
You
know
the
big
change
was
when
storage
and
computational
power
got
so
cheap
that
there
was
no
like
when
it
was
no
longer
a
a
scarce
resource.
C
You
can
kind
of
just
save
and
do
whatever
you
want,
like
you
didn't
have
to
worry
about,
compressing
or
or
anything
like
you
could,
just.
I
guess,
optimize
for
I'll,
save
the
data
to
use
it
later
or
flexibility,
not
for
like
saving
pennies
on
some
sort
of
infrastructure
cost
and,
and
so
what
you
end
up
getting
to
a
point
is
like
now,
you've
enabled
this
whole
kind
of
world
of
elt,
where
you
have
all
the
data
in
your
data
warehouse.
C
You
have
everything
at
your
fingertips
and
and
we're
kind
of
in
a
point
where
you
really
can
do
anything
you
want
in
this
unlimited
elastic
cloud
compute
world
as
long
as
you're
willing
to
pay
for
it.
So
we're
at
this
world,
where
everybody
is
migrating
to
cloud
data
infrastructure
you've
got
great
tools
like
snowflake
dbt
5
that
really
led
the
wave
of
this
kind
of
elt.
C
Modern
data
stack
and,
and
and
to
be
honest,
it's
I
felt
like
as
an
early
user
of
the
modern
data
stack
and
even
when
we're
starting
consulting
we'd
have
clients
that
would
come
to
us
and
say
hey.
We
have
a
problem
like
we
need
attribution
or
sales
reporting,
we
would
say
actually
backup.
You
need
a
modern
data
stack
now.
C
People
actually
just
come
to
us
and
say:
hey
scott,
hey
brooklyn
data:
can
you
implement
the
modern
data
stack,
so
I
feel
like
we've
kind
of
gone
mainstream
in
the
sense
that,
like
everybody's
converted
to
this
vision,
but
I
don't
know
nick
you
have
a
hot
take.
What
do
you?
What
do
you
have
on
the
history
of
how
how
we
got
here?
I.
D
D
With
the
the
modern
data
stack
in
my
view,
you
know
there's
kind
of
two
definitions:
one
is
that
it's
one
set
of
technologies:
it's
ingest,
5,
tran
or
meltano,
plus
a
cloud
data
warehouse,
a
transform
layer
on
top
of
that
dbt
plus
reverse
etl
and
a
bi
tool
right
and
those
are
the
you
plop.
You
know
those
four
things,
those
five
things
together.
That's
a
modern
data
stack.
I
have
a
broader
view
and
more
expansive
view
of
the
modern
data
stack
that
it
is
rebuilding
the
data
ecosystem.
D
With
these
new
underlying
realities
in
place,
those
realities
are
cloud
computing.
Those
realities
are
that
the
world
of
data
is
complicated.
It's
only
getting
more
complicated,
meaning
that
any
non-trivial
size
company
has
maybe
a
hundred
different
system
of
records
that
they're
integrating
and
they
are
doing
very
sophisticated
things
with
it.
It's
a
very
challenging
engineering
problem,
so
they're
collecting
tons
of
data
there's
this
move
to
the
cloud
where
you
can
incrementally
adopt
technology
and
then
there's
infinitely
elastic
compute.
D
D
D
So
it's
like
the
embrace
of
cloud
technologies,
the
embrace
of
managed
services
and,
most
importantly,
not
most
importantly,
crit,
as,
as
importantly,
in
my
view,
the
embrace
of
engineering
practices
to
tackle
and
control
that
complexity,
and
I
think
the
first
kind
of
beachhead
there
was
dbt
and
the
entire
point
of
dbt
is
actually
taking
analysts
and
moving
them
to
analytics
engineers
right.
It
doesn't
dumb
down
the
analysts.
B
Absolutely-
and
I
agree
with
you
100
when
it
comes
to
the
the
volume
aspect
of
those
five
vs
kind
of
being
done.
Like
we've
answered
the
storage
problem,
we've
answered
the
ability
for
us
to
be
able
to
like
manage
that
aspect
of
data.
I
think
the
velocity
veracity
variety,
like
the
data
quality
aspects
of
that
those
still
exist
out
there,
because
everyone
has
data
quality
issues.
B
B
B
My
my
definition
of
low
code
is
that
you
have
the
ability
to
in
import
a
library
and
insert
two
to
three
lines
of
code.
That
then
do
does
the
computational
business
logic
that
you
expect
to
give
you
an
output
or
an
artifact
that
you
would
traditionally
have
to
architect
yourself.
So
dbt,
for
example,
is
a
low
code
solution
for
me.
You
can
go
in
there.
You
can
write
some
information,
but
then
it
then
gives
you
the
model
and
the
artifacts
to
then
reuse
that
over
time.
D
See
I
disagree
with
that.
I
do
not
consider
dbt
a
low
code
solution.
Dbt
is
a
gbt
is
code
like
the
it
is,
so
our
typical
users
who
integrate
dexter
with
dbt
are
definitely
kind
of
farther
along
in
their
data
infrastructure
journey.
I
would
say
the
median
is
600
dbt
models
around
hundreds
of
models,
each
of
which
are
like
you
know
dozens
or
hundreds
of
lines
of
sql
code.
You
know
like
dbt
is
a
high
code
and
I
don't
know
if
high
code's
a
word
but
like
what
that,
which
is
not.
B
D
Me
one
of
the
properties
of
low
code
is
that
you're
not
interacting
with
a
version
control
system.
So
typically,
what
I
view
low
code
is
like
web
flow,
for
example,
where
like
is
the
drag
and
drop
tool,
the
whole
notion
that
you're
part
of
an
engineering
process
has
been
abstracted
away
from
you.
I
consider
like
excel
a
low
code
environment,
probably
the
best
one.
C
I
would
I
would
add
that
there
is
some
sort
of
like
syntax
that
allows
you
to
go
slightly
custom,
but
is
like
a
highly
like
humanized
and
prose
written
syntax,
syntax
and
not
code.
Like
you
know,
like
any
of
those
low
code
tools
will
have
some
sort
of
just
like
you
want
to
do
something
a
little
custom.
You
can
use
some
sort
of
like
their
own
proprietary
human,
readable
syntax.
To
do
it
right.
B
B
But
it's
it's
essentially
a
gcp's
drag
and
drop
ui
that
allows
you
to
bring
in
tasks
to
do
data
processing
where
it's
so
it
comes
to
that
low
code.
Where
you're
talking
about
you,
don't
have
the
version
control
that
you
have
to
manage.
You
have
just
the
models
that
are
in
the
pipelines
that
are
on
your
gc
cloud,
but
you
also
have
the
ability
to
get
in
there
and
kind
of
define
your
own
tasks
through
their
sdk
and
api.
B
D
I
think
I
think
there
is
a
distinction
for
sure
you
know
to
me.
No
code
is
like
a
completely
commoditized
tool.
I
think,
like
five
trans
excellent
example
of
a
no
code
tool
that
replaces
a
what
previously
was
a
custom
engineering
process
with
out
of
the
box
component
that
require
no
real
customization,
like
they
require
a
little
bit
of
configuration,
but
I
would
consider
that
like
a
no
code
tool,
I
guess
the
other
question
is
like
the.
D
Maybe
the
dividing
line
is,
if,
like
a
drag
and
drop
graphical
tool
where,
like
informatica,
is
that
no
code,
because
you're
not
typing
text,
is
like
the
barrier
between
no
code
and
low
code
like
a
text
box
where
you
can
type
python
or
some
dsl,
you
know,
I
don't
know,
I'm
sure,
there's
some
taxonomy
out
there
that
some
there's
got
to
be
a
there's
got
to
be
a
vc,
no
load,
no
low
code
landscape
like,
but
we
can.
Why?
Don't?
We
just
pause
the
podcast
and
look
that
up.
B
C
D
D
D
D
D
So
we
also
kind
of
you
know
believe
in
that
or
doubling
down
on
that
as
well,
and
I
think
like
actually
that
that,
like
technical
viewpoint
and
philosophy
is
almost
like
part
of
the
modern
3d
stack
that,
like
you,
should
think
of
like
producing
like
immutable
data
artifacts
and
allow
for
memoization
recomputation
and
all
this
all
this
sort
of
stuff.
It's
like
the
way
data
has
to
be
produced.
So
I
think
the
full
embrace
of
this.
D
B
You
mentioned
that
there's
been
a
shift
that
you've
noticed
in
your
client
base
that,
instead
of
them
coming
to
like
hey,
we
need
to
solve
this
problem.
They
literally
just
come
to
you
saying:
hey.
We
want
you
to
implement
a
modern
data
stack
for
us
when
they
come
to
you
with
this
ask:
do
you
think
they're
talking
about
this
abstraction
there?
B
That
is
handling
the
big
complexity
problem,
or
do
you
think
that
they're
looking
for
something
that's
more
on
the
low
code
side
of
things,
that's
easier
for
them
to
manage
and
easier
than
to
get
started
quickly
and
then
just
get
the
answers
that
they're?
Looking
for?
Where
do
you
think,
like
the
general
perspective,
is
on
modern
data
stack
versus
the
three
of
us
who
have
like
a
fairly
technical
view
that
want
to
have
that
ability
to
interact
with
a
medium
code
environment?
If
you
will.
C
I
even
remember
like
you
know
three
four
years
ago,
when
everybody
needed
to
hire
a
data,
scientist
and
yeah.
I
said
like:
why
do
you
need
to
hire
data
scientists
like
I
need
to
hire
data
scientists,
because
everybody's
hiring
data
scientists
like
like
people,
knew
wasn't
thought
it
was
important,
but
didn't
actually,
you
know,
have
a
specific
thing
they
need
to
solve.
Like.
C
I
now
see
that
a
lot
of
folks
saying
like
they
know
the
modern
data
stack
is
important
and
they
want
to
just
implement
it,
and
what
we
help
to
do
is
help
them
to
really
understand
why
they
need
to
do
it,
and
you
know
that's
powering
marketing
campaigns
using
reverse
etl.
It's
making
data-driven
decisions,
but
I
actually
think
it's
surprising
enough
when
people
are
asking
for
something
they're,
they're
kind
of
had
this
tool
ecosystem
in
mind.
C
That
they'd,
like
like
it's
kind
of
like
everybody's,
accepted
that
you
know,
with
dbt
being
so
successful
with
snowflakes
ipo
going
so
well,
like
it's
kind
of
reached,
a
level
of
trust
that
even
large,
publicly
traded
companies
enterprises
feel
really
comfortable.
Just
this
is
the
stack
of
the
future.
We
just
want
to.
You
know,
build
from
build
our
fee,
one
in
this
stack
or
we
want
to
shift
a
generation
or
two
to
this
latest
stack.
A
C
And
so
like
it's
an
interesting
process
because
before
we
used
to
understand
their
needs
and
then
make
the
tailored
recommendation
and
now
they're
coming
in
and
saying,
we
need
a
modern
data
stack.
How
can
this
help
us
and
it's
it's
like?
We
don't
have
to
educate
the
the
importance
of
the
modern
data
stack.
We
have
to
help
them,
navigate
all
the
solutions
out
there
and
the
different
options
and
how
how
it
will
impact
their
business.
You
know
we
it
it's
it's
you
know
no
longer
like
I
mean
I
remember
a
couple
years
ago.
C
You
know
if
we
were
saying
snowflake
or
dbt
like
I
had
to
actually
like,
educate
and
sell
these
technologies
in.
You
know
like
well
well,
this
dbt
is
a
small
company
based
out
of
philadelphia.
They
do
a
little
bit
of
consulting
and
they've
got
this
really
great
open
source
framework,
and
it's
getting
a
lot
of
momentum.
I
mean
and
then
snowflake
it's
it's
like
rapidly
growing
sas
company.
You
know
series
d
value
and
like
now,
it's
like
these
are
they've
gone,
mainstream
and
people
universally
accept
that
these
are
great
tools.
B
Nick,
do
you
think
there's
like
any
risk
or
downside
to
that
shift
in
that
conversation
where
people
are
coming
to
the
to
scott's
company
and
saying
oh,
this
is
the
prescriptive
data
stack
that
we
need
as
opposed
to
coming
and
saying
like?
Can
you
help
us
solve
creating
a
modern
data
architecture?
I.
D
D
So
for
me,
when
someone
is
like
goes
to
scott
and
is
like,
I
need
the
modern
data
stack,
they
kind
of
are
implicitly
communicating
that
hey
like
we
are
at
level
one
of
like
maslow's
hierarchy
of
data
like
we
can't
really
count
things.
We
don't
understand
kind
of
our
core
business
metrics
and
we
want
to
be
able
to
do
that
and
get
on
the
latest
and
greatest
stuff,
and
I
know
lots
of
other
people
have
been
able
to
successfully
do
that.
Scott.
Is
that,
like
a
fair
representation,
I.
C
So
I
think
it's
like,
I
think
it's
it's
a
spectrum,
so
I
think
you
know,
we've
got
the
small
companies
that
want
to
just
count
things,
and
then
we've
got
the
larger
companies
that
are
building
complex
internal
effort
to
support
complex
internal
data,
use
cases,
or
you
know,
complex
external
data
use
cases.
I
think
the
the
the
one
difference
is
now
people,
no
matter
where
they
on
the
spectrum.
They
already
have
a
good
idea
of
the
tools
they
want
to
use.
C
So,
like
maybe
like
that,
I
want
to
count
things
small
early
stage,
shut
up,
say:
hey,
we
won
this
modern
data
stack,
but
we
have
like
fairly
large.
You
know
complex
use
cases
of
clients
coming
in
and
saying
like
yeah,
you
know,
we've
been
seeing
a
lot
about
snowflake
and
we've
heard
a
lot
about
dbt.
We
want
to
like
kick
the
tires
and
see
if
it
works
for
us
like
I.
I
do
think
it
is
the
the
conversation
is
even
in
the
more
complex
side
of
things.
C
People
are
yes.
Obviously,
it's
more
custom,
but
they're
already
coming
in
with
a
perception
of
these
are
the
tools
that
we
need
to
like
they
come
with
the
short
list
already,
as
opposed
to
coming
with
a
problem
like
it
might
be.
You
know
the
account
things.
Might
you
know,
come
and
say
we
want
this
exact
data
stack
or
the
shortest
the
more
complex
ones
would
like.
You
know.
D
Yeah
speaking
of
that,
how
many
people
come
with
the
short
list
that
includes
a
reverse,
etl
or
operational
analytics
tool.
These
days
like
if
out
of
your
next
50
customers
say
how
many
would
you
estimate
or
come
with
you
or
that
you
end
up,
recommend
recommending
an
operational
analytics
tool,
yeah.
C
D
Interesting
yeah,
because
that
really
does
change.
You
know
I
kind
of
like
was
maybe
a
little
dismissive
about
the
count
things.
What
not,
but
I
think,
adia
reverse
etl,
adding
a
reverse
etl
tool
really
changes
the
fundamental
dynamic,
where
this
is
no
longer
just
a
place
to
count
things
and
produce
dashboards,
but
a
way
to
interconnect
all
your
systems
of
record
in
a
pretty
novel
way.
I
think
it's
a
really
interesting
space.
C
C
From
from
earlier,
I
think
even
the
simplest
stacks
are
actually
getting
more
complex
in
that
you
know.
There's
you've
got
this
like
kind
of
beating
heart
of
the
cloud
data
warehouse
at
the
center
of
the
modern
data
stack.
You
know
before
people
just
used
to
like
put
some
sort
of
monolithic,
bi
tool
on
it,
but
you
know
now
people
might
have
a
primary
bi
tool,
like
you
know,
look
or
tableau,
but
they
might
have
hex
for
kind
of
you
know
notebook
kind
of
visualizations.
C
C
Stacks
people
are
now
getting
more
value
out
of
modern
day
sex.
But
it's
scary
because
you
know,
I
don't
think
you
know.
Companies
are
have
historically
built
the
teams
or
invested
in
the
tooling
and
the
infrastructure
to
run
production
workloads
on
a
modern
data
stack
and
so
you're.
Getting
kind
of
these
scary
situations.
Like
you
know,
people
are
doing
a.
You
know,
pushing
some
code
into
a
dbt
model
that,
like
I
don't
they
might
not
realize
that
is
actually
you
know
going
to
accidentally
push
10
000.
C
D
C
100
and
like
I
would
like
that,
I
mean
that's
like
you
know,
and
that's
where
I
feel
like
tools
like
you
know.
You
folks
are
like,
like
I
I'm
kind
of
struggling
right
now
now,
when
I'm
building
these
modern
data
stacks
to
have
the
tooling
to
orchestrate
from
end
to
end,
to
observe.
What's
going
on
and
to
like
detect
these
things
before
it
happens
and
like
that's
like.
C
A
Nick
while
we
well,
we
have
you
here
about
your
most
recent
talk.
I've
been
wanting
to
ask
you
this.
Your
open
source
data
stack
conference
talk
on
your
core
beliefs
of
what's
happening
with
the
future
of
data,
tying
it
back
to
what
you
mentioned
earlier
about
the
big
complexity.
Can
you
recap
for
us
what
you
shared
in
that
talk.
D
Oh
god,
I
mean,
I
don't
remember
exactly
what
I
said,
but
I
can
probably
approximate
it.
You
know,
I
guess
the
the
it
was
what
I
was
speaking
to
before
about
I
view
the
modern
data
stack
more
as
an
approach
in
a
methodology
than
a
prescribed
set
of
technology
or
technology
categories.
So
just
an
example.
D
Two
years
ago
you
know
what
scott
said
like
an
ingest
tool,
a
bi
tool
transform
tool
like
cloud
data
warehouse
was
a
modern
day
stack
and
now
we've
added
reverse
etl
as
a
new
first
class
citizen,
and
I
think
we'll
be
adding
more
and
more
first-class
citizens
to
the
modern
data
stack.
Because,
what's
going
to
happen,
is
that
someone
like
scott
you?
D
A
B
B
I
love
the
outcomes
that
come
out
of
that
and
being
able
to
predict,
what's
going
to
occur
like
understand
the
the
fabric
that
it
exists
and
that's
the
way
that
I
see
data
observability
kind
of
looking
at
the
modern
data
stack
and
the
retooling
that
nick
and
scott
have
talked
about
here.
We
have
these
different
tools
that
are
coming
into
play
that
are
addressing
the
different
use
cases,
but
as
we
continue
to
see
those
increase
of
downstream
use
cases,
that's
gotten
nick
talked
about
earlier.
B
You
have
more
and
more
stakeholders
that
are
invested
in
the
data
that's
coming
in
and
what
that
data
is
going
to
be
telling
them,
because
if
you
have
a
client
coming
to
you
saying
like
can
you
solve
x
and
then
you
say
cool
here's,
my
solution
to
x
and
then
three
months
later,
they'll
say
actually
x
is
solving
y.
So
what
do
I
do
now?
If
you
don't
have
a
way
to
understand?
B
What's
going
on
there,
it's
difficult
to
do
and
we've
seen
the
orchestration
layer
come
into
play
over
like
we
cron
jobs
evolved
into
airflow.
We've
got
dagger
coming
into
play
here,
that's
doing
for
python.
What
dbt
did
for
sql
right
and
then
also
giving
us
the
ability
to
be
more
data
aware,
as
you
move
through
the
the
process
here
and
connecting
those
tools
that
are
adjusting
those
use
cases
I,
but
the
next
step
here
that
I
see
is
what
happens
in
those
interfaces
between
those
tools.
What
happens
in
the
fabric?
B
That's
covering
those
interfaces
between
those
tools
and
being
able
to
see
what's
happening
to
your
data
in
near
real
time
is
going
to
be
extremely
valuable
when
it
comes
to
resolving
an
issue
whenever
your
solution
for
x
is
now
solving
for
y,
and
you
know
at
databand,
what
we
are
focusing
on
is
the
actual
movement
of
data
in
those
pipelines,
understanding
that
when
data
comes
in,
how
does
it
match
the
profile
that
you
expect?
How
does
it
actually
affect
the
data?
B
That's
already
in
your
lake
or
in
your
warehouse
and
then
understanding
when
those
changes
occur?
How
you
can
resolve
those
right
like
there's
lake
fs
is
an
interesting
part
here,
where
you
get
into
data
versioning
and
being
able
to
roll.
That
back
quickly
is
a
really
interesting
use
case
that
I
think
about
a
lot.
But
that's
where
I
see
data
observability
fitting
into
the
modern
data
stack
is
understanding.
B
It's
that
same
mentality,
that
tristan
talked
about
where
it's
taking
devops
principles
and
finding
ways
to
do
that
in
a
way
that
still
empowers
your
software
engineer
to
own
that.
But
it's
not
something
that's
abstracted
away
from
them
that
they
they
can't
manage
themselves.
If
that
makes
sense
thoughts
on
this
scott
nick,
I
know
y'all
aren't
specifically
in
the
observability
space,
but
it's
such
a
nascent
space
that
it's
fun
to
hear
everyone's
opinion
about.
C
B
C
Related
and
you
know
complementary
to
the
observability
space
yeah,
I
mean-
I
think
it's
it's
really
interesting,
because
you
have
at
least
a
lot
of
the
folks
that
that
I
see
in
our
clients
and
that
we
work
with
in
this
analytics
engineering
space
are
not
historically
like
backend
engineers
that
have
transitioned
to
data
they're
kind
of
analysts
that
have
like
found
this
part
of
them.
And
it's
like
oh
wait.
There's
this
role
called
analytics
engineer.
C
C
I
see
that
all
this
stuff
is
like
universally
applicable
and
helpful
and
when,
when
we
add
observability
into
the
modern
data,
stack
like
and
ci
cd,
like
everybody
kind
of
gets
it
and
it
sees
the
benefit,
but
I
think
the
folks
that
are
kind
of
playing
in
the
mod
like
the
analysts
and
analytics
engineers,
the
modern
day
tech.
Don't
know
this
is
an
option
like
they're
they're.
C
D
I
don't
know
if
I
have
strong
thoughts
on
this.
The
I
guess
the.
I
think
that
the
interesting
tension
is
going
to
be.
I
think
we're
going
to
see
this
too,
because
we
have
some
data,
integrated
data
ability,
capabilities
in
the
orchestration
layer
in
dagster,
because
this
kind
of
makes
sense
to
provide
like
a
minimum
layer,
because
in
our
view,
in
the
end,
the
purpose
of
these
systems
is
to
produce
data
assets.
D
So
you
should
be
aware
of
that,
but
we're
not
gonna,
but
there's
also
going
to
be
a
space
for
cross-cutting
tools,
because
not
everyone's
gonna
move
to
dexter
immediately
and
people
have
specialized
capabilities
in
the
observation
layer.
So
I
kind
of
think
of
it.
You
know,
I
think,
there's
a
struggle
that
continues
to
play
out
in
data
in
terms
of
everyone,
like
people
are
building
best
of
breed
solutions
for
everything,
but
then,
when
they
do
that
they
end
up
having
to
integrate
10
tools
in
order
to
get
a
heartbeat
going
and
that's
rough.
D
But
then
people
also
don't
want
to
be
locked
into
a
monolithic
system.
So
I
kind
of
view
it.
I
commit
from
an
urban
vendor
standpoint
of
I'm
thinking
about
you
know
I
want
to
provide
some
out-of-the-box
observation
capabilities
for
so
that
people
kind
of
have
an
easy
button
for
that,
but
then
also
appropriate,
plugability
layers,
so
that
if
someone
wants
a
more
specialized
solution
or
needs
to
cross
a
bunch
of
systems
that
we
can
plug
into
that
and
figuring
out
the
balance
there,
I
think,
is
kind
of
kind
of
interesting.
C
I
keep
telling
nick.
I
want
my
easy
butt.
It's
in
your
email,
I
know,
there's
literally
an
email
in
my
mailbox.
My
inbox,
that's
from
nick
is
like
here's,
your
easy
button
watch
this
video
and
it's
like
on
my
to-do
list
for
this
afternoon,
but
it
was
seriously
like
because
you
know
both
of
you
hit
the
nail
on
the
head.
It's
like
when
we
set
up
these
clients,
it's
like
five
tools
on
a
cron
nine
o'clock,
9
10,
9,
17,
9
30.,
and
it's
just
like
you
space
them
enough
to
like.
C
Hopefully
you
know
it's
fine,
or
sometimes
I
have
some.
I
have
some
clients
that,
like
the
dvt
model
will
run
you
know
or
like
multiple
times
just
to
catch,
something
just
just
because
it's
it's
it's
funny.
Everybody
suddenly
has
access
to
all
these
great
tools,
but
they
don't
have.
You
know
they're,
not
linking
them
together.
They
are
co-existing,
like
you
know
when,
when
when
you
have
two
young
kids,
they
don't
play
together.
C
C
If
something
breaks
it's
hard
to
replay
it's
hard
to
know
where
it
broke,
like
you
know,
we're
looking
for
smoke
signals
like
dbt,
test
failures
or
or
look
or
dashboards
that
you
know
smell
funny
like
you
know
it
it
that's
the
space
that
I
like,
I'm
I'm
that's
the
space.
I
feel
in
at
the
moment
right.
C
B
Yeah,
I
feel
like
I
feel
like
that's
like
that,
should
be
like
the
north
star
for
all
of
data.
Observability
right
like
eliminate
the
ceo
email
like
eliminate
the
on-call
data
rotation
right
like
because,
if
you
can
do
that,
then
you've
got
a
trillion
dollar
product
at
that
point
in
time.
So
harper.
D
B
I
think
that
there's
a
good
interplay
that
like
orchestration,
should
be
aware
of
the
movement
of
data
that's
going
through,
but
there
is
still
a
need
for
a
vendor
to
address
the
understanding
of
the
why
these
things
have
occurred
and
presenting
that
in
a
way
that
makes
it
actionable
like
that's,
that's
the
ultimate
outcome
of
a
good
data.
Observability
tool,
in
my
opinion,
is
not
just
telling
you
where
it
occurred
or
when
it
occurred,
but
making
it
actionable
for
you
to
address
that
problem.
D
Yeah,
no,
I
think
the
awareness
piece
is
good,
because
I
think
you've
actually
provided
me
a
tool
to
explain
our
relationship
to
the
other
domains
of
data
in
a
way
that
well
accurately
communicates
that
we're
not
trying
to
like
eat
every
other
tool
right.
It's
just
like,
for
example,
we
want
our
tool
to
be
aware
of
kind
of
the
concept
of
data
quality,
just
as
a
different
domain,
so
that
tool
like
great
expectations
or
something
can
like
plug
into
that
and
like
be
like.
D
Oh,
this
is
where
you
put
your
data
quality
test
in
the
orchestration
layer,
but
we're
totally
agnostic
to
like
how
those
data
quality
tests
are
expressed
because,
like
you
know,
great
expectations
has
a
super
opinion
of
dsl
like
we
don't
want
everyone
to
have
to
use
that
they
have
their
own
thing.
That's
cool
in
the
same
way
like
we
want
to
be
like
aware
of
data
observability,
so
that
there's
a
place
for
observability
tools
to
plug
in
and
get
our
metadata
exhaust.
So
they
can
build
interesting
and
novel
things
so
yeah.
B
A
Awesome
well,
this
was
this
was
so
fun
chatting
with
you
guys.
Can
we
we're
coming
up
on
time,
but
I
do
want
to
ask
for
one
maybe
call
to
action
from
all
of
you.
If
we
were
to
educate
our
space
about
this
rebuilding
infrastructure,
what
would
you
offer
as
a
piece
of
advice
and
how
to
best
equip
data
teams
start
with
scott.
B
C
B
Yeah,
absolutely
let
me
pull
out
my
six
dollar
words
here
to
try
and
confuse
people,
so
they
don't
quote
me
on
what
I'm
saying.
No,
I
mean,
I
think,
the
the
last
point
that
we
were
talking
about
the
the
awareness
aspect
here,
where
you're
creating
the
ability
for
these
tools
to
to
work
together
is
going
to
be
the
best
way
for
the
data
industry
as
a
whole
to
address
this
big
complexity
problem.
B
So
if
there's
one
like
call
to
action
that
I
would
give
is
don't
let
capitalism
get
in
the
way
of
innovation,
like
it's
very
important
that
we
find
a
way
to
sorry,
it's
it's
very
important
that
we
find
a
way
to
grow
our
careers
and
grow
our
businesses,
and
that
way
we
are
successful
and
fulfilled.
But
at
the
same
time,
it's
very
important
that
we
keep
in
mind
that
we're
trying
to
serve
a
community
as
a
whole,
and
if
we
lose
sight
of
that,
then
we
won't
actually
help
evolve.
D
A
D
It
yeah
pretty
much
yeah.
I
would
say
that
a
little
a
little
upfront
investment
in
thought
about
how
you're
going
to
structure
things
can
pay
dividends,
not
just
down
the
road
but
like
tomorrow,
meaning,
like
you
know,
if
you
actually
kind
of
think
of
your
system
as
like
your
own
little
platform
and
you
think
about
like
hey,
how
does
a
stakeholder
interact
with
it
and
just
kind
of
do
a
little
upfront
planning
and
think
about
that,
while
still
meaning
agile,
you'll
pay
enormous
enormous
dividends?
You
know.
Yes,
there
is
an
easy
button.
D
D
There's
well
it's
in
your
email
right
right
away,
but
the
the
the
easy
button
before
the
easy
button
you
know
there
is
this
like
prescribed
stack,
but
that
prescribed
stack
is
like
the
result
of
kind
of
a
lot
of
thought
and
care.
So
you
know
I
would
just
like
you
know,
do
like
think
ahead,
like
just
a
little
bit
before
scratching
stuff
together
because,
like
you
might
like
it
might
save
you
a
lot
of
lot
of
pain,
not
just
in
the
long
term,
but
right
away.
C
Yeah
this
is
so
advice
for,
but
I
got
a
few
actually
so
I
mean
I
think
you
know
one
thing
that
I
I
always
kind
of
you
know
challenged
my
team.
Is
that
like
just
just
no,
you
gave
me
plenty
of
time.
I've
had
the
most
time.
So
now
it's
like
the
pressure,
you
know
it's.
We
have
all
these
great
tools
to
handle
all
sorts
of
like
instability
like
data.
C
I
think
yes,
we
want
to
build
robust
fault,
tolerant
data
pipelines,
but
I
think
sometimes
people
actually
forget
that
you
can
go
down
the
hall
knock
on
the
door
of
the
data
of
the
production
engineer
of
the
folks,
building
those
systems
and
actually
give
them
feedback
too
and
say,
like
hey,
you
know,
could
you
change
this?
Or
can
you
like?
You
know
you
we're
doing
some
sort
of
complex
logic
to
it.
You
know
infer
this
relationship,
but
actually
it's
an
existing
relationship.
C
They
just
don't
put
the
foreign
key
in
the
table,
it's
like
hey,
can
you
do
a
one-time,
backfill
and
then
put
the
foreign
key
in
this
table
moving
forward,
and
it's
like
makes
things
much
more
robust.
It's
like
just
you
know,
and
then
I
would
just
abstract
that
to
just
like
just
generally
make
you
validate
that
the
problem
you're
trying
to
solve
is
a
problem
and
the
constraint,
you're,
optimizing
and
operating
around
is
actually
a
constraint.
D
Scott,
you
know
has
a
good
line
about
that
constraint.
Problem
is,
is
elon.
Actually
I
saw
him
interviewed
and
he
was
like
at
at
spacex.
We
never
say
that.
Oh,
this
team
insists
on
having
this
constraint
of
this
process
in
place.
It
always
is
a
human's
name
yeah,
so
that
you
know
that
there's
a
person
to
talk
to
because
then
you
like
realize,
there's
no
human
there.
It
was
actually
some
intern
last.
You
know
two
years
ago
who
put
that
and
then
no
one
can
defend
it
anymore.
You
know
I
mean
he.