►
Description
Join us for a live tech talk and learn about architecting for data quality in the lakehouse with delta Lake and PySpark. After the presentation, we’ll have time for questions. Excited to have you join us!
From null values and duplicate rows to modeling errors and schema changes, data can break for millions of reasons. To combat this, teams are increasingly adopting best practices from DevOps and software engineering to identify, resolve, and even prevent this "data downtime" from happening in the first place. Join Prateek Chawla and Ryan Kearns as they walk through how data and ML engineers can solve for data quality across the data lakehouse by applying data observability techniques. Topics to be discussed include: how to optimize for data reliability across your lakehouse's metadata, storage, and query engine tiers, building your own data observability monitors with PySpark, and the role of tools like Delta Lake to scale this design.
Links:
Exercises: http://github.com/monte-carlo-data/data-downtime-challenge
Jupyter Notebooks: http://github.com/monte-carlo-data/data-observability-in-practice
A
So
rachel
is
just
setting
up
the
live
stream
so
for
joining
us
in
zoom
from
the
the
event
page.
We
also
are
streaming
live
to
youtube
and
linkedin,
which
is
really
exciting.
So
we
just
have
some
things
that
on
our
side,
that
we
need
to
get
set
up
and
and
give
a
few
more
minutes
for
folks
to
join
and
then
then
we'll
go
we'll
get
started
all
right.
Rachel
says
we
already
see
it
on
youtube.
A
A
B
A
D
A
D
A
A
A
B
Meanwhile,
it
looks
like
there's
a
lot
of
folks
that
chiming
in
from
all
over
the
place,
I'm
looking
to
india,
florida,
brazil,
italy,
cleveland,
ohio,
some
other
folks
from
seattle,
texas
yeah.
This
is
a
great
turnout
here.
We've
got
folks
from
new
york,
brighton
uk
over
in
zoom
as
well.
So
I'm
looking
at
linkedin
and
let
me
switch
over
to
youtube
so
yeah.
It's
we've
got
some
really
good
turnout
here
right.
This
is
awesome.
A
A
All
right,
so,
thank
you,
everyone
for
joining
us.
This
is
a
tech
talk
with
our
folks,
our
friends
from
monte,
carlo,
so
architecting
for
data
quality
in
the
lake
house,
with
delta
lake
and
pi
spark,
and
today
we're
joining
joining
us
is
pratik
and
ryan
from
monte,
carlo
and
new.
This
this
time
around.
A
So
we
typically
run
our
meetups
on
a
meetup
group,
the
data
in
ai
online
meetup
group
and
just
recently,
actually,
our
team
launched
a
a
page
on
the
community
linux
foundation
and
I'd
love
for
you
to
join
us
there.
So
it's
a
virtual.
It's
our
delta
lake
chapter
there,
where
we're
gonna
host,
all
of
our
community
office
hours
and
tech
talks
and
all
awesome
content
with
the
linux
foundation.
A
So
there's
the
link
there
to
join
I'll
drop,
that
in
the
chat,
spaces
and
stay
tuned
up
upcoming
delta
lake
specific
talks
and
then
also
too,
we
have
I'd
love
for
you
to
subscribe
on
youtube.
We
have
a
great
tech
talk
and
meetups
playlist
there,
where
we
record
all
of
our
sessions
and
they're
all
available
immediately
after
the
recording
and
then
follow
us
on
linkedin.
We
have
a
delta
lake
and
a
databricks
linkedin
page,
and
I
wanted
to
do
a
quick
call
out
because
we
have
a
conference
coming
up
soon.
A
It's
the
data
nai
summit,
it's
gonna
be
hybrid
this
year,
which
really
excited
in
person
and
online
at
end
of
june.
The
27th
through
30th
lots
of
awesome
sessions
and
the
theme
kind
of
is,
you
know,
building
the
modern
data
stack
on
the
on
the
data
lake
house,
so
hopefully
you'll
join
us
in
person
or
online
and
before
I
pass
it
over
to
jenny,
my
co-host
and
our
speakers.
I
just
want
to
remind
everyone
that
the
session
is
recorded
and
it'll
be
available
on
youtube
directly
after
we're
done
so
I'll
drop.
A
That
link
in
the
chat
and
then
also
too
we'd
love
to
hear
your
questions
and
gather
those
while
we're
going
through
the
presentation.
So,
if
you're
joining
us
on
zoom
q,
a
is
your
best
place
and
then
you
know
the
chat
is
on
for
youtube
and
linkedin
and
denny
myself
will
be
moderating
that
so
we'll
capture
your
questions
and
answer
as
many
as
we
can
at
the
end.
So
with
that
I'll
pass
it
over
to
demi.
B
Thanks
very
much
karen
really
appreciate
it,
so
I'm
not
going
to
go
ahead
and
do
too
much.
I
just
wanted
to
reiterate
two
things.
One
is
that
please
do
ask
her
questions
we'll
do
our
best
to
answer
them
live
in
linkedin,
zoom
or
youtube
all
three.
In
fact,
okay,
we're
not
gonna.
There
will
be
pauses
during
the
session,
so
both
pratik
and
ryan
can
go
ahead
and
also
chime
in
as
well,
but
for
the
some
of
the
other
questions,
we'll
wait
till
the
end
of
the
session.
B
C
Cool
then
thanks
dan
and
karen,
I'm
pretty
chavela,
and
I'm
here
with
my
colleague
ryan
kearns
and
we're
the
founding
technical
team
members
of
monte
carlo.
So
we're
really
excited
to
be
here
today
and
we
really
wanted
to
thank
the
databricks
team
and
the
linux
foundation
for
this
opportunity
and
we're
huge
fans
of
both
organizations
and
the
broader
communities
they're
building
around.
C
I
think
some
of
the
most
innovative
and
important
open
source
frameworks
we'll
be
discussing
today
is
what
it
takes
to
build:
a
build
and
design
a
more
reliable
data,
lake
and
lake
house
using
systems
like
data
breaks
delta
like
pi,
spark
indian
observability,
but
before
we
get
started.
Let's
just
do
some
quick
introductions
so,
like
I
said,
I'm
prathik,
I'm
the
founding
engineer
and
technical
lead
at
monte,
carlo,
where
I
help
drive
the
technical
strategy
for
our
data
observability
platform.
C
Previously,
I
served
as
a
technical
lead
at
barracuda,
where
I
worked
on:
email
fraud,
preventive
technologies
and
I
graduated
summa
laude,
with
a
bs
in
computer
science
from
uc
santa
cruz
and,
of
course,
in
my
free
time,
as
you
can
see
from
the
slides
I
enjoy
watching,
broadway
shows
flying
airplanes,
reading
and
exploring
new
places
I'll.
Let
ryan
introduce
himself
as
well.
B
D
D
So
I
took
a
pretty
beat
up
minivan
across
the
country
with
some
friends
of
mine
when
I
was
in
high
school
and
remains
one
of
my
coolest
memories.
So
definitely
a
big
fan
of
that
car,
but
I'll
pass
it
back
to
pratik
who's,
going
to
get
us
jumping
in
right
to
the
the
technical
meet
and
details
of
the
presentation.
D
C
C
Some
of
these
principles
and
technological
approaches
to
lakes
and
lake
houses
using
delta
and
all
the
other
tools
we
have
at
our
disposal
after
that,
ryan
will
walk
through
anomaly
detection
by
doing
a
demo
using
pi
spark
and
ml
flow
and
and
like
karen
and
danny's
iterated.
If
you
ever
have
any
questions
feel
free
to
put
them
in
the
chat
and
we'll
try
our
best
to
answer
them
all
right.
So,
let's
get
started
so
imagine.
You've
joined
a
company
as
a
lead
data,
architect
and
you're,
inheriting
a
data-like
environment.
C
This
one's
built
on
spark
and
data
bricks,
which
is
I'd,
say
pretty
exciting
you.
You
can't
wait
to
dive
in
and
solve
the
new
challenges
and
like
work
through
all
the
thorniest
data
problems
the
company
has,
but
the
first
one
on
your
roadmap
really
is
data
quality,
and
this
only
came
to
light
because
wonky
dashboard
was
shared
with
your
ceo
and
was
projecting
next
year's
financials,
but
it
actually
used
last
year's
data
and
of
course
that's
not
what
you
want
to
do.
C
So
the
question
is:
how
are
we
actually
going
to
solve
this
before
we
dive
in
on
how
to
like
actually
fix
the
problem
and
it
sure
doesn't
happen
again.
It
kind
of
helps
to
understand
the
implications
of
bad
data
for
stakeholders
across
the
company.
It's
like
the
gift
that
keeps
on
giving
so
to
speak
and
works
as
you
move
downstream,
and
these
broken
pipelines
can
manifest
in
any
number
of
ways
originating
in
buggy
code,
missing
data
or
even
operational
issues.
C
So
and
then,
even
if
even
no
matter
how
you
test,
like
the
best
testing,
there's
literally
no
way
to
catch
all
of
them,
so
the
key
really
is
to
monitor
and
alert
on
these
data
quality
issues
as
soon
as
they
happen
and
before
they
affect
your
stakeholders,
whether
those
stakeholders
are
internal
analysts
or
maybe
your
external
customers.
So
here
we
put
together
this,
what
we've
deemed
the
data
quality
cone
of
anxiety-
and
I
think
it
highlights
a
lot
of
these
different
levels
of
impact
as
data
moves
downstream
beyond
your
lake
house
altogether.
C
So
you
can
see
bad
data
can
have
serious
ramifications
like
it.
You
can
lead
to
lost
revenue,
it
can
lead
to
maybe
erosion
of
trust.
You
can
obviously
waste
your
resources,
there's
just
so
many
things
that
bad
data
can
lead
towards.
It
affects,
and
it
affects
all
different
types
of
organizations
like,
for
example,
with
nasa
they
had
their.
C
They
had
their
mars
climate
orbiter.
It
was.
This
is
a
multi-million
dollar
satellite,
but
it
was
lost
because
the
software
used
to
build
this
technology
used
a
different
measurement
that
the
astronauts
used
to
program
where
it
should
land.
I
believe
it
was
pound
seconds
versus
metric
units,
they
lost
about
125
million
dollars
and
years
and
years
of
progress
and
there's
lots
of
other
examples
of
this.
C
Like,
for
instance,
there's
a
there
was
a
617
billion
dollar
so-and-so
quote-unquote
fat
finger
incident
where
trading
error
on
the
japanese
stock
market
happened
where
a
broker
just
entered
the
wrong
number
and
it
caused
it.
A
huge
stock
fluctuation
and
the
japanese
stock
market
was
sent
to
rattle
for
a
few
hours.
It
was
just
a
simple
order
for
toyota
motor
stock,
but
it
was
worth
more
than
the
sizes
like
sweden's
economy,
so
there's
like
that
list
goes
on
and
you
might
necessarily
be
risking.
You
know
617
billion
dollars
due
to
bad
data
quality.
C
You
can
understand,
like
I
think,
how
the
implications
grow
as
your
bad
data
flows
downstream.
So
now
the
question
that
begs
the
question:
how
do
we
address
this
bad
data?
So
we
believe
that
you
can
learn
a
thing
or
two
from
our
friends
in
software
engineering.
The
problem
of
stopping
bad
data
in
its
tracks
can
be
tackled
by
relying
on
some
of
the
best
practices
of
like
our
friends.
C
In
software
engineering,
software
engineers
leverage
these
principles
of
site
reliability,
they
use
observability
and
they
use
all
this
stuff
to
ensure
that
their
applications
are
performing
as
expected,
and
that
up
time
is
high,
while
downtime
is
low,
so
as
organizations
grow
and
the
underlying
tech
stack
powering
them
becomes
more
complicated.
For
instance,
let's
say
you're
moving
from
a
monolith
to
a
microservice
architecture.
C
I
think
it's
very
important
for
devops
teams
to
maintain
a
constant
pulse
on
the
health
of
their
systems
and,
more
specifically
at
least
we
believe
observability
speaks
this
need
and
refers
to
the
monitoring,
tracking
and
triaging
of
incidents
to
prevent
downtime
and,
as
a
result
of
this,
like
we're
kind
of
shifting
as
an
industry
to
more
distributed
systems.
So
observability
engineering
is
really
becoming
really
really
critical.
So
now,
if
you
look
at
it
from
from
our
friends
in
software
engineering,
observability
engineering
really
is
broken
down
into
three
major
pillars.
C
As
you
can
see
on
the
slide,
you
have
your
metrics,
you
have
your
traces
and
you
have
your
logs
now.
What
are
these
three
things?
So
metrics
refer
to
like
a
numeric
representation
of
data
measured
over
time.
Traces
are
like
the
casually
causally
related
events
in
a
distributed
environment
and
logs
are
like
a
the
source
of
record
for
an
event
that
took
place
to
have
like
a
time
stamp.
C
C
So
now
what
does
this
look
like
in
practice?
How
can
you
actually
actually
follow
these
footsteps
of
these
engineering
friends
that
I've
been
talking
about
and
apply
these
principles
to
your
data
pipelines
in
the
next
few
slides
I'm
going
to
outline
a
few
of
these
best
practices
and
hope?
My
my
hope
is
that
you
can
take
that
away
from
this
presentation
and
apply
it
to
your
own
systems
and
just
like
how
you
have
the
three
pillars
of
observability
for
software
reliability.
Every
data
team
has
these
pillars
of
data
reliability.
C
I've
I've
broken
them
down
into
these
five
here
and
we
believe
they're
strong
indicators
of
whether
or
not
something
is
broken
or
gone
wrong.
So
these
five
pillars
are
really
freshness
distribution,
volume
schema
and
lineage,
I'm
going
to
walk
through
each
one
of
them,
so
the
first
one
is
freshness,
freshness
kind
of
seeks
to
understand
how
up-to-date
your
data
tables
are,
as
well
as
the
cadence
at
which
your
tables
are
updated.
It's
particularly
important
when
it
comes
to
decision
making.
As
we
all
know,
stale
data
is
synonymous
with
wasted
time
and
money.
C
That
was
our
first
example
last
year's
data.
The
next
step
on
our
our
next
pillar
up,
basically,
is
distribution,
and
it's
really-
or
you
can
say
it's
more
like
a
function
of
your
data
as
possible
values.
It
tells
you
whether
your
data
is
in
an
accepted
range
data.
Distribution
gives
you
insight,
I
think,
into
whether
or
not
the
tables
in
your
your
the
tables
in
your
data
can
be
trusted.
Based
on
what
you
expect
from
your
data,
the
next
next
pillar
is
volume.
C
C
The
fourth
of
these
is
schema,
so
schema
really
is
changes
in
the
organization
of
your
data,
in
other
words,
schema
can
indicate
broken
data
and
monitoring
who
makes
these
changes
to
these
tables
is
foundational
to
understanding
the
health
of
your
data
ecosystem,
not
to
mention
preventing.
I
think,
some
of
these
things
from
occurring
again
and
finally,
we
have
lineage
when
data
breaks.
C
I
think
the
first
question
people
always
ask
is
where
data
lineage,
I
think,
provides
this
answer
by
telling
you
which
upstream
sources
and
downstream
integrators
were
impacted
as
well
as
which
teams
are
generating
the
data
and
accessing
it.
Essentially,
I
think
lineage
acts
as
their
context
and
it
kind
of
serves
as
a
singular
source
of
truth
for
all
your
data,
consumers-
and
you
can
see
from
my
slide
here.
This
is
a
dlt
which
is
from
databricks
and
shameless
plug,
so
tracking
these
five
pillars
can
give
you
a
good
sense
of
how
healthy
your
data
is.
C
C
The
devops
lifecycle
incorporates
a
bunch
of
distinctious,
continuous
stages
from
like
planning
to
coding,
to
development,
to
monitoring
the
cycle
repeats
itself,
and
while
many
of
our
technologies
and
frameworks
have
adapted
to
meet
the
standard
and
best
practices
we
see
in
software
engineering,
I
think
we
tend
to
handle
data
a
little
bit
more
reactively
and
it's
prevented
us
from
driving
these
changes
in
like
a
scalable
way.
So
much
in
the
same
way,
devops
apply
the
continuous
feedback
loop
to
improving
software.
It's
we
believe.
C
It's
time
you
can
leverage
the
same
blanket
of
diligence
for
data,
so
the
data
reliability
life
cycle
is
kind
of
an
organization-wide
approach
simultaneously
and
proactively.
Improving
your
data
health.
It
eliminates
data
downtime
by
applying
some
of
the
best
practices
from
devops
to
your
to
your
data
pipelines.
So
we
hope
we
think
this
framework
will
kind
of
allow
you
to
be
to
hit
a
bunch
of
different
key
factors.
One
of
them
is,
you
can
be
the
first
to
know
about
data
quality
issues
in
productions.
C
Then
you
can
fully
understand
the
impact
of
the
issue
and
then
fully
understand
where
your
data
broke,
but
now,
let's
actually
dive
into
the
environment.
So
if
you
recall,
we
inherited
a
lake
house
and
to
see
we
can
set
up
data
pipelines
for
success
when
it
comes
to
preventing
like
these
billion
billion
dollar
data
incidents.
I
think
the
first
thing
we
need
to
discuss
here,
though,
is
what
is
really
the
difference
between
a
warehouse
and
lake.
C
There's,
a
bunch
of
different
factors
and
a
bunch
of
people
have
opinions
on
this,
but
I
think
it
really
boils
down
to
the
way
data
is
stored
and
structured,
as
well
as
the
various
entry
points
you
have
in
a
lake
that
many
warehouses
don't
so
I've
kind
of
broken
down
the
data
lake
into
four
components
or
layers.
The
first
of
them
being
metadata,
then
you
have
your
storage
layer.
Then
you
have
your
query
engine
layer
and
then
you
have
your
query
logs
there.
As
you
can
see
from
this
slide.
C
There
are
various
technologies
you
can
use
to
manage
this,
and
this
is
not
even
complete
close
to
all
of
them
and
but
for
the
purposes
of
this
presentation,
let's
dive
a
little
bit
into
building
a
reliable
freshness
detector
with
spark
and
data
bricks.
But
before
we
get
to
that
and
I
hand
it
off
to
ryan,
I
wanted
to
briefly
discuss
all
of
these
different
layers.
So
the
first
one
is
your
metadata
layer.
So
leica
warehouses
have
an
information
schema
that
provides
like
an
extensive
amount
of
metadata
information
about
objects
like
tables
and
views.
C
Many
lakes
have
what
is
called
a
metastore
and
acts
as
a
central
repository
for
metadata,
and
there
are
many
flavors
of
this
mana
store,
but
I'm
going
to
talk
about
the
three
that
work
with
data
bricks.
So
you
have
your
central
hype,
metastore
and
every
databricks
deployment
has
this
and
it's
accessible
by
all
of
the
clusters
you
use.
You
can
also
provision
an
external
hive,
mysql
meta
store
or
you
can
leverage
the
aws
blue
catalog.
C
This
ladder
is
only
of
course
available
if
you're
using
the
aws
provider
and
then
for
those
of
you
who
are
using
delta
tables,
you
can
actually
access
the
whole
transaction
log,
which
is
the
real
source
of
truth
and
contains
like
an
ordered
record
of
every
transaction.
That's
ever
been
formed
on
a
delta
lake
table
and
databricks
is
doing
some
great
work,
so
I
think,
unify
a
lot
of
this
with
their
upcoming
unity,
catalog,
which
you
can
sign
up
for
the
waitlist
today.
It's
okay!
So
that's
great!
C
That's
like
I
just
introduced
a
bunch
of
different
technologies.
What
does
that
mean?
And
why
do
they
actually
matter?
And
why
do
you
want
to
crawl
and
track
any
of
these?
So
I
think
doing
so,
we'll
let
you
cover
actually
three
of
the
pillars
that
we
talked
about
by
just
by
just
tracking
metadata.
You
can
get
schema
changes,
you
can
get
size
and
you
can
get
freshness
well
to
be
fair.
C
C
This
can
be
done
in
one
of
two
ways:
kind
of
regardless
of
what
blob
storage
you're
using
so
the
first
one-
and
I
think
the
most
obvious
one
is:
you
can
crawl
the
paths
that
correspond
to
a
table,
and
this
allows
us
to
compute
like
the
total
size
by
just
you
know,
summing
up
the
files
and
the
one
or
more
directories
that
correspond
to
the
table.
Obviously,
this
is
like
this
for
very
large
table.
This
can
be
very
slow,
but
you
could
there
are
a
couple
different
optimizations
that
you
can
make
like.
C
For
instance,
you
can
cache
older
partitions
if
you
know
that
only
new
partitions
are
written
to
and,
of
course,
like
all
resource
problems,
you
can
throw
compute
at
it
until
of
course,
the
date
you
can't
and
then
for
freshness.
If
a
file
is
encountered
in
the
crawl,
you
know
there's
no
freshness
event,
but
if
you
haven't
encountered
one,
you
know
something
is
up
or
you
can
do.
What
I
personally
believe
is
a
more
is
a
more
scalable
and
better
approach,
and
you
can
just
track
files
as
they
change
and
map
these
to
a
table.
C
It's
obvious,
I
think
it's
more
scalable,
because
it's
push
based,
so
you
can
bake
this
into
your
etl
tool
or
use
a
service
like
s3
events,
if
you're
using
that
provider
like
on
the
left
here,
I
have
an
example
of
s3
events
through
event
bridge
and
you
can
basically
use
this
to
track,
puts
and
deletes,
and
obviously
this
can
only
track
new
updates.
But
if
you
need
to
get
a
history,
you
can
always
bootstrap
and
create
the
baseline
as
needed.
C
C
Let's
say
if
your
queries
are
degrading
over
time
or
let's
say
you
have
tables
that
don't
get
frequently
run,
queries
or
specific
fields
that
don't
get
run
frequently.
You
can
either
clean
them
up
or
move
them
to
a
slower,
yet
cheaper
tier
of
storage,
if
you're
using
a
cloud
provider
and
the
last
layer
we
have
here
is
the
query
engine.
So
if
you
have
the
query
engine,
I
think
you
can
execute
more
specific
or
custom
data
checks.
This
includes
things
like
running
field
health,
which
covers
like
you,
can
get
retinol
race,
unique
grade
percent,
none.
C
So
many
different
versions
of
that
you
can
also
do
distribution,
which
covers
like
values
with
low
cardinality
tables,
to
like
determine
if
something's
significantly
changed
like.
If
you
have
something
tracking
different
clients
like
iphone
or
android.
If
a
new
client
shows
up,
maybe
that
wasn't
expected
and
then
obviously,
with
the
query
engine,
you
can
kind
of
write
any
sql
queries
you
can
validate
whatever
type
of
business
logic
you
want.
You
have
some
custom
thresholds
that
you
know
that
have
to
be
hit.
C
You
can
use
all
this
information
context.
You
have
to
triage
it
and
just
an
important
thing
about
after
the
fact
you
just
you
need
the
necessary
tools
to
rca
and
determine
impact,
but
you
can
use
all
these
things
to
kind
of
build
like
a
holistic
picture
of.
What's
going
on
in
your
data
environment,
okay,
so
now
I've
kind
of
walked
through
some
of
these
components.
D
B
No,
no,
no
current
questions
right
now.
We've
actually
answered
most
of
them.
I
did
want
to
call
out
that
for
any
folks
that
are
asking
data
brick,
specific
questions,
let's
go
ahead
and
leave
them
in
the
community
form
the
database
community
forum.
Community.Datebricks.Com,
that's
the
great
place
for
that.
We
definitely
want
to
discuss
much
more
about
the
about
the
topic
that
we're
here
today
to
talk
about:
okay,
so
about
architecting
data
quality.
So
let's
go
ahead
and
definitely
focus
on
that
here,
but
pl,
please!
B
D
D
At
the
very
end,
we'll
give
a
brief
kind
of
indication
of
how
you
would
do
some
sort
of
model,
tracking
experiment,
tracking
and
general
kind
of
parameter,
search
grid,
search
with
ml
flow.
But
I'll
have
some
important
points
about
the
limitations
of
that
technology.
In
this
case
and
kind
of
invite
you
to
think
about
some,
some
future
directions
that
you
could
take.
D
So
what
I'll
do
is
I'll
get
myself
over
to
data
bricks
here,
making
sure
pretty
you
guys
can
check
out
my
yeah,
my
presentation,
yeah
so
architecting
for
data
quality
in
the
lake
house,
I'm
now
in
a
notebook
environment.
I'll,
be
writing
some
python
and
just
sort
of
importing
some
some
base.
Libraries
for
for
handling
this
task.
What
I'm
going
to
be
using
is
I'll,
be
leaning
heavily
into
this
new
command
that
delta
lake
tables
support,
which
is
called
describe
detail.
D
So,
if
I
can
just
sort
of
post-process
on
this
describe
detail
command,
you'll
notice,
I
can
get
some
last
modified
timestamp
and
some
measured
timestamp,
which
is
just
the
current
time,
and
what
I
would
do
if
I
was
architecting
for
an
observability
platform
here
is
I'd,
have
some
you
know,
update
times
table
and
I
would
use
the
pi
spark
data
frame
format
to
write
into
that
table.
A
batch
of
these
last
modified
timestamps.
D
This
table
hasn't
been
updated
since
2020,
it
is
a
demo
table
in
the
databricks
environment.
So
I
sort
of
cheated
and
I
created
some
synthetic
data
that
would
respect
you
know
this
workflow
had
we
been
recording
data
for
about
the
last
two
months
of
last
60
days,
and
so,
if
I
run
on
this
update
times
table
you'll
notice,
I've
got
four
tables.
Now,
as
I
said,
we
could
kind
of
scale
this
up.
D
I've
got
diamonds,
but
also
emerald,
sapphires
and
rubies,
and
then,
since
you
know
mid
to
late
february,
I've
got
when
we
modified
those
and
when
we
measured
them
and
the
measurement
is,
is
sort
of
just
an
augmented
feature.
It's
not
strictly
necessary
for
understanding
how
tables
update,
but
it
does
give
you
a
sense
of
how
up
to
date,
your
own
orchestration
system
is
running.
So
if
you.
D
So
now
I'll
do
a
bit
of
of
sql
kind
of
wrangling
on
this
table.
You'll
notice
that,
in
order
to
plot
this,
I'm
just
going
to
kind
of
catch
a
ones
column
as
my
value
and
I'm
going
to
get
the
measured
time
stamp
and
the
modified
timestamp
I'll.
Take
the
first
400
points:
this
table
is
actually
around
6
000
rows,
but
for
the
purposes
of
this
demo,
we're
going
to
stick
with
a
kind
of
constrained
group.
D
B
D
D
So
if
we're
using
this
table
for
dashboarding
purposes
or
we're
feeding
the
results
into
an
ml
model
that
runs
live
in
production,
this
may
be
fine.
You
know.
Maybe
we
can
tolerate
a
delay
of
roundabout
20
hours.
Maybe
one
day,
two
full
days
seems
to
be
quite
a
lot
and
we
may
want
to
identify
this
anomaly
here
this
anomaly
and
the
tables
update,
cadence
and
I'll
sort
of
show
how
we
can
do
some
really
naive
anomaly,
detection
to
to
surface
that
result.
D
So
what
I'm
gonna
do
is
is
a
form
of
a
bit
of
feature
engineering
and,
of
course,
for
those
in
the
in
the
machine
learning
world
feature.
Engineering
can
get
a
whole
lot
more
nuanced
than
this,
but
I'm
just
going
to
kind
of
give
you
the
minimal,
minimal
augmentations,
and
then
you
can
sort
of
infer
all
of
the
fun
directions.
You
would
take
this
on
your
own,
so
what
I
did
is
I
just
I
added
a
lambda
function.
D
I
assigned
a
new
column
that
just
takes
the
last
modified
timestamp
and
subtracts
it
by
the
shifted
timestamp.
So
this
intuitively
gives
you
the
delay
between
incremental
updates
and
you'll
notice.
If
I
sort
of
I
can
scroll
between
these
two
and
you'll
see
that
dependent
on
the
size
of
the
gap
between
updates,
the
corresponding
bar
is
larger,
because
this
records
the
number
of
seconds
since
the
table
was
last
updated.
D
So
we've
got
like
kind
of
a
little
regularity
here.
It
doesn't
seem
like
we
eclipse
something
like
80
000
seconds
all
too
often,
and
we've
got
the
spike
at
around
150
000
seconds.
So
visualize
like
this,
it's
not
quite
so
clear
that
the
anomaly
is
all
that
significant.
But
if
we're
looking
in
terms
of
its
delay,
time
can
sort
of
get
a
more
nuanced
understanding,
more
obvious
understanding.
D
D
So
what
I'm
using
is
a
pretty
out
of
the
box
detector
from
scikit-learn
called
a
local
outlier
factor,
lof
detection.
If
you
may
be
familiar,
this
basically
sort
of
computes
the
density
of
neighboring
points
in
a
time
series
and
will
check
their
deviation
relative
to
their
to
their
neighborhood,
and
if
the
deviation
is
significant
enough,
we
get
a
higher
score
for
an
outlier
factor
and
then
we're
more
likely
to
tag
those
data
points
as
anomalous.
D
So,
if
you
know
the
kind
of
machine
learning
breakdown
of
tasks,
this
is
an
unsupervised
learning
approach.
We
don't
have
gold
labels
as
to
what
constitutes
real
anomaly
or
not,
and
this
is
a
kind
of
key-
I
don't
want
to
say
setback,
but
a
consideration
you
need
to
make
when
you're
doing
anomaly.
Detection
is
the
fact
that
you
know
most
of
the
most
of
the
tasks
that
you
can
configure
at
least
at
the
beginning
are
going
to
be
unsupervised.
D
D
So
I
just
fit
to
my
data
with
this
local
outlier
factor,
I'm
using
10
neighbors
on
a
on
a
train
set
of
around
50
data
points,
and
if
I
can
just
highlight,
there's
one
outlier
factor
that
stands
out
the
threshold.
If
you
don't
change
the
kind
of
vanilla
scikit-learn
configuration
is
a
1.5
absolute
value
1.5
for
detection,
so
we've
got
one
point
eclipsing
that
and
actually,
if
I
use
some
some
plotly
to
add
a
vertical
line
where
that
integer
is
located,
the
one
that
that
returns
positive
for
the
predictions
well
negative.
D
D
D
You
know
setting
up
some
notification
routing
to
dump
this
anomaly
into
another
table
or
some
event
log
somewhere,
where
we
could
maybe
send
an
email
or
push
a
slack
notification
or
something
to
alert
our
engineers
to
the
fact
that
the
table
is
out
of
date
at
this
point
in
time
and
what
I
want
to
get
into
I'm
going
to
I'm
going
to
show
the
ml
flow
stuff
in
just
a
second,
but
I
also
want
to
indicate
kind
of
the
setbacks.
That
of
using
this
such
a
naive
attempt.
D
Lof
is,
I
mean
it's
good
in
massive
data
settings
for
certain
certain
tasks,
because
it
actually
scales
quite
nicely.
It
has
good
complexity,
but
it's
not
the
right
detection
algorithm
for
our
particular
case
here.
For
one
you
know
the
threshold
for
detection
is
in
it's
bidirectional,
which
would
mean
that
if
one
of
these
small
updates
down
here
was
too
small
we'd
be
issuing
an
alert
for
that.
For
that
point,
and
that's
obviously
not
what
we
want,
we
don't
really
care
if
our
table
updates
so
close
together
in
time.
D
Unless
you
have
like
a
particular
use
case
that
would
make
such
a
such
an
instance
relevant,
also
we're
only
passing
in
the
delay
information.
We
have
sort
of
a
kind
of
a
singular,
one-dimensional
vector
as
our
feature
set
I've
ignored
all
of
the
fancy
and
interesting
stuff.
You
can
do
around
seasonality,
so
there's
there's
no
auto
regression,
no
rolling
average
exponential,
smoothing
kind
of
seasonality
prediction
any
of
the
stuff
that
you
might
have
heard
about.
D
If
you
know
time
series,
anomaly,
detection,
whole
winters,
and
what
have
you
you
can
do
that
stuff
here?
If
you,
if
you'd
like
to
and
there's
a
lot
of
features
of
your
disposal,
even
from
just
pulling
the
the
measurement
timestamp
and
the
kind
of
splitting
the
date
time
of
the
the
update
time
intelligibly.
D
But
just
in
my
sort
of
last
couple
of
last
couple
minutes
in
the
technical
demo,
I
want
to
show
what
ml
flow
had
been
doing
after
I
had
configured.
My
auto
log
here
and
the
representation
is
a
bit
a
bit
lacking,
given
that
the
task
is
unsupervised
and
I'll
talk
about
that
in
just
a
moment,
but
here's
my
latest
experiment
run.
You
can
see
at
9,
35,
pacific
time.
If
I
wanted
to,
I
could
restore
the
notebook
to
the
state
where
that
model
had
been
run.
D
I'm
not
going
to
do
that
because
it's
identical,
but
I
can
look
at
what
the
experiment
looks
like.
It
took
three
seven
3.7
seconds
to
train
it's
finished
and
then
I
can
actually
look
at
the
parameters
I
used
for
this
model
instance,
so
I
can
check
that
my
lof
leaf
size
was
30.
I
used
10,
neighbors
and
p
equals
two
means.
I
was
using
a
euclidean
distance
metric
instead
of
manhattan.
D
Actually,
I
think
if
I
go
back
to
my
code,
we
may
surface
a
warning
at
some
point
about
the
fact
that
yeah,
because
we're
running
unsupervised,
we
don't
have
training
labels
and,
as
a
result,
all
of
the
kind
of
nice
out
of
the
box,
training,
metric
tracking
and
accuracy
tracking
that
mlflow
provides
is
not
available
for
us
and
I'll
actually
go
back
to
the
slides
to
explain
why
that's
an
interesting
thing,
so
I'm
gonna
go
back
into
the
presentation
here.
But
do
people
have
any
questions
in
the
meantime.
B
Bunch
of
questions
that
actually
have
popped
up
since
there,
so
let
me
go
ahead
and
try
to
chime
in
on
some
of
them.
In
that
case,
okay
sounds
good.
For
starters,
from
logistically
will
the
notebooks
and
the
slides
be
made
available.
We've
already
responded
to
various
forms
that
we
will
prop
them
up
to
the
youtube
channel.
There
is
a
request
to
us
email
it
out
as
well.
B
I'm
not
I'm
a
little
concerned
that
we
might
be
spamming
folks,
so
we
might
not
do
that,
but
I'll
talk
to
rachel
and
karen
to
figure
out
what
the
right
approach
is
for
something
like
this,
but
we
will
definitely
be
propping
it
up
to
the
youtube
channel.
So
that
way
you
can
see
it
from
there.
Okay,
all
right,
let's
see!
Oh
there
is
an
earlier
question
concerning
the
data
lineage
graph.
B
C
Sure
so
the
very
cool
thing
is:
if
you're
using
delta
live
tables,
it's
created
for
you,
you
don't
really
have
to
do
anything.
It's
just
built
out
of
the
box
from
data
breaks
and
you
can
just
get
it
through
there.
If
you're,
not
using
data
bricks,
there's
a
lot
of
open
source
options
and
plug
monte,
carlo
as
well
a
bunch
of
other
different
ways
to
build
lineage
out
there.
B
Yeah,
no,
I
think
it
does
yes
and
for
everybody,
I'm
looking
at
linkedin
youtube
and
zoom,
so
I'm
trying
to
compile
the
questions
together.
So
do
me
a
favor
if
it's
not,
if
we
didn't
answer
it's
not
on
purpose,
just
chime
in
again
and
we'll
we'll
continue
to
follow
up
okay.
So
that's
the
first
question
next
question:
okay,
let's
see
also,
would
you
be
able
to
throw
some
light
into
how
you
see
the
changes
in
the
query?
B
That
actually
is
directly,
I
believe
during
earlier
in
the
demo
that
ryan
you
had
shown
that
was
actually
from
a
delta
table,
so
the
delta
table,
when
you
saw
you
described
table,
actually
contains
a
snap
snapshot,
history
of
all
the
transactions
that
were
committed
to
the
table,
and
so
because,
that's
all
there,
that's
actually
how
you
see
any
of
those
changes,
and
then
there
was
a
subsequent
question.
I
believe
from
malcolm
that
asked:
oh
well,
why
don't
you
just
use
delta
cdf?
That's
actually
a
good
call
out
delta
change.
B
Data
feed
is
a
cool
ability
to
go
ahead
and
see
all
the
changes,
but
at
the
individual
data
level.
So
every
road
change
every
insert
every
update
every
modification
shows
up
in
the
change
data
feed
itself.
But
if
you
want
a
summary
of
the
transactions
that
occur,
which
obviously
will
often
will
occur
with
multiple
rows,
that's
what
the
describe
table
will
end
up
showing
you,
especially
when
you
show
the
history
of
that.
So
hopefully
that
helps
answer
that
question.
I
think
that's
it
for
now.
B
D
Yes,
yeah,
not
only
can
you,
but
we
actually
have
implementations
of
of
pretty
much
that
exact
thing,
using
a
kind
of
lightweight
package
of
sql
lite
to
spin
up
kind
of
a
database
file
and
doing
some
scikit-learn
stuff
and
some
pandas
to
kind
of
manipulate
the
data
frames
so
yeah.
You
can
run
this
out
of
the
box
on
some
like
local
jupiter
server
and
there's
some
demos
for
that.
D
Also,
that
that
describe
detail
command
happens
to
be
a
particularity
of
the
kind
of
delta
table
environment,
but
you
can
do
very
similar
stuff
and
the
point
in
using
that
command
was
to
showcase
that
yeah.
There
are
kind
of
there's
a
natural
way
to
surface
freshness
data
from
tables
that
need
not
be
specific
to
the
database
table
format.
So
yeah
a
lot
of
extensibility.
A
D
I'm
gonna
give
some
talking
points
about
some,
some
more
kind
of
like
machine
learning,
considerations
running
this
at
scale,
building
out
kind
of
an
experiment,
laboratory
and
and
whatnot.
But
why
don't
we
just
hop
in
here
and
then
we
can?
We
can
pick
it
back
up
for
questions
in
just
a
few
minutes,
so
we
kind
of
did
the
demo.
D
I
I
sort
of
just
mentioned
two
minutes
ago
that
I
think
there
are
some
setbacks
to
unsupervised
learning.
Obviously,
in
the
big
data
world
you
hear
unsupervised
learning
gaining
a
lot
of
traction,
because
when
you
have
like
a
massive
amount
of
unstructured
data,
running
clustering,
algorithms
and
doing
sort
of
feature,
exploration
and
discovery
is
is
quite
a
powerful
tool.
D
The
problem
is,
obviously
you
have
a
it's
not
straightforward
to
compare
the
results
of
two
unsupervised
classifiers
to
each
other.
So
two
anomaly
detectors
might
disagree
on
what
constitutes
anomalous
and,
if
you
don't
have
a
good
kind
of
like
north
star
signal
as
to
what
you're
trying
to
pick
up
it's
hard
to
know
how
to
improve
on
our
naive
baseline
that
we
just
implemented,
and
that
prompts
me
to
talk
a
little
bit
about
two
concepts
that
are
sort
of
interrelated
bootstrapping
and
meta
learning.
D
So,
if
you're
familiar
with
this
statistical
learning,
literature
you
might
know,
bootstrapping
is
like
a
form
of
model
bagging
or
ensemble
learning,
I'm
using
it
in
kind
of
a
more
colloquial
sense
to
refer
to
bootstrapping
a
process
so
specifically
bootstrapping
a
supervised
learning
process.
So
unsupervised.
Learning
at
a
high
level
is
basically
taking
data
that
doesn't
have
a
gold
label.
So
no
correct
answer
with
the
data
and
clustering
that
data
into
different
sets
that
share
common
features
and
that's
what
we
just
did.
D
D
Supervised
learning
is
different
in
that
you
actually
have
a
source
of
truth.
You
know
what
the
real
answer
to
the
question
is
in
my
little
diagram
over
here
you
can
see
the
the
points
are
colored
differently,
based
on
what
class
they
actually
belong
to,
and
so
that's
not
something
that's
up
to
the
model
to
decide.
That's
a
gold
feature,
a
sort
of
gold
standard
and
a
supervised
classifiers
allow
for
kind
of
much
more
precise
accuracy.
Metrics.
D
In
fact,
I
think,
if
I
just
show
my
slide
on
tracking
monitor
performance
in
this
line
graph
over
here,
you're
looking
at
the
mean
squared
error
of
a
supervised
classifier
over
time
and
noticing
that
as
it
as
it
runs,
it's
improving.
So
it's
it
seems
to
be
converging
on
a
good
strategy
for
for
this
detection
task,
as
indicated
by
the
the
low
error
rate
so
back
to
bootstrapping.
D
What
I'm
referring
to
with
bootstrapping
is
basically,
if
you
have
anomalies
in
your
environment,
and
you
have
some
ability
to
capture
those
anomalies-
represent
real
incidents
in
the
data
pipeline.
You
know
a
real
event
where
some
etl
schedule
went
offline
or
some
schema
change
broke.
Something
upstream,
consider
recording
that
data
and
consider
using
it
as
a
labeled
feature
when
you're
going
about
training
anomaly
detection
models.
D
I
know
that's
kind
of
a
tantalizing
suggestion,
because
I
could
go
into
a
whole
nother
hour
of
presentation
on
how
you
would
actually
go
about
doing
that
and
sort
of
tracking
the
feedback.
From
your
end,
users
like
data
engineers
who
actually
go
about
investigating
incidents,
it's
a
whole
other
ballpark,
but
something
to
consider
and
especially
to
consider
if
you're,
using
something
like
ml
flow,
and
you
have
interest
in
building
out
some
sort
of
scaled
sorry,
some
sort
of
scaled
approach
towards
anomaly
detection.
D
D
Houses
can
be
tricky
like
pratik
mentioned,
the
flexibility
that
you
get
through
all
the
different
endpoints
and
data
formats
that
are
able
to
be
manipulated
comes
at
the
cost
of
a
lot
of
messiness.
If
you
don't
govern
your
system
correctly,
so
we
provide
the
five
pillars
of
data
observability
to
give
a
holistic
view,
a
sort
of
end-to-end
understanding
of
data
health
and
to
provide
kind
of
a
natural
transition
towards
knowing
what
actions
to
take
when
data
goes
down
dependent
on
you
know
the
different,
the
different
signals
from
your
data
observability
pipeline.
D
So
we've
talked
about
a
lot
I
as
I've
kind
of
mentioned
multiple
times.
Unfortunately,
we've
got
way
more
content
to
discuss
that
I
can
possibly
introduce
in
this
short
time
we
have
together,
but
I
will
say
if
you're
interested
in
kind
of
hearing
some
of
this
in
in
longer
form,
we
do
have
some
interesting
events
coming
up.
So
the
impact
tour
is
a
three
city.
Stop
in-person
events
on
on
date,
observability
concepts
our
next
one.
D
D
I
recommend
taking
a
look
and
in
even
longer
form,
if
you
have,
the
attention
span,
we're
putting
together
a
book
on
these
concepts,
called
data
quality
fundamentals
which
will
be
going
through
not
just
the
data
lake,
but
the
data
warehouse
environments,
everything
from
elt
to
dashboard
applications,
business,
intelligence,
all
of
the
types
of
observability
concerns
and
data
quality
concerns.
You'd
want
at
different
places
in
that
pipeline.
D
So
do
take
a
look.
We've
got
some
chapters
for
early
pre-release
with
new
ones
coming
pretty
soon,
and
we
just
wanted
to
say
thanks.
It's
been
a
real
pleasure
to
work
with
databricks
and
linux
foundation
on
this.
We
hope
that
it's
for
you
guys
at
the
beginning
of
learning
a
whole
lot
more
about
data
quality
and,
of
course,
we're
super
accessible
for
asking
questions
feel
free
to
send
us
emails
here
and
yeah.
I
think
I
do
have
some
appendices
if
people
are
interested
in
seeing
some
specific
stuff,
but.
B
Absolutely
actually
one
one
of
the
questions
that
I
wanted
to
make
sure
that
people
that
you
got
a
chance
to
answer
is
like
you've
shown
a
great
demo
of
running
of
showing
of
showcasing
data
quality
within
databricks,
and
we've
already
answered
the
question
of
like
forsake
argument
like
that:
oh
that
we're
going
to
be
attaching
open
links
to
notebooks
to
do
this
all
in
open
source.
There
is
a
question
that
was
like
hey
what
what
does
monte
carlo
data.
Do,
though,
like
what
do
you?
B
C
Yeah,
I
can
feel
this
one.
So
basically,
what
monte
carlo
does
is
a
lot
of
what
you
saw
was
us
kind
of
manually
walking
through
the
different
components
and
a
lot
of
the
complexities
that
are
associated
around
it.
Monte
carlo's
aim
is
to
automate
this
for
you
and
make
this
kind
of
automagical
like
the
rest
of
your
different
tools,
so
kind
of
it's
like
a
data
observability
platform.
C
I
need
a
discovery
platform,
so
it
kind
of
does
all
of
what
we
were
talking
about
and
I
hopefully
a
lot
more
just
out
of
the
box
and
it
works
on
a
bunch
of
different
platforms.
You
can
connect
it
to
a
bunch
of
different
warehouses
and
lakes
and
different
components,
and
so
you
don't
have
to
go
through
the
collecting
this
data.
C
Carlo
is
a
platform
that
kind
of
tries
to
do
all
of
that
for
you
and
give
you
the
tooling,
to
investigate
and
understand
your
data
better
by
providing
you
different
programmatic
methods
that
you
can
use
in
sdk,
you
can
use
our
airflow
provider
and
you
can
do
kind
of
it'll
it
just
basically
it's
like
if
you've
ever
used
a
if
you've
ever
used
new
relic
or
if
you've
ever
used
the
data
dogs.
It's
intended
to
be
that
for
data.
C
I
hope
that
answers
that
I
can
go
into
more
details
like
on
a
specific
component.
If
you
guys
want
to
learn
like
a
specific
warehouse
or
more
about
like
some
of
these
specific
layers
and
like
the
different
components
of
them,
and
how
they're
monitored
and
instrumented
out
of
the
box
happy
to
do
it,
but
the
intent
is
basically
it's
like
a
no
code
solution.
You
don't
have
to
run,
describe
detail,
you
don't
have
to
store
it
in
a
table.
You
don't
have
to
monitor
this
and
kind
of
build
your
own
framework
on
it.
C
D
I
was
just
going
to
say
briefly,
you
know
from
my
wheelhouse
on
the
anomaly
detection
side.
We
what
I
just
showed
was
sort
of
a
naive
freshness
detector.
We
have
a
whole
family
of
models
that
we
maintain
that
run
freshness,
detection,
size,
detection,
different
field,
health
metrics.
So
we
track
things
like
the
null
rate
and
all
of
those
models
are
sort
of
purpose
built
to
scale
to
the
whole
environment.
D
So
you
can
sort
of
run
out
of
the
box
anomaly,
detection
that
warms
up
in
a
couple
days
and
then
you've
got
observability
across
basically
the
entire
warehouse,
for
example.
So
that's
that's
my
end
of
the
pipeline,
but
petite
can
talk
a
lot
about
the
different
storage
solutions
that
we
haven't
went
with
because
yeah
it's
it's
quite
a
cool
tool.
I
think.
C
Yeah
I'll
actually
try
them
in
one
other
point,
sorry
for
plugging
more,
but
basically
I
think
a
lot
of
the
complexities
with
lakes
comes
to
scalability.
We
didn't
get
an
opportunity
to
really
dive
into
that,
but
that's
I
think
that
can
be
the
entire
deck,
because
the
nice
thing
about
warehouses-
and
maybe
relational
databases-
is
they're
relatively
small
in
scale.
So
you
can
do
a
lot
of
these
operational
things.
C
When
you
talk
about
lakes,
you're
talking
about
terabytes
and
petabytes
of
data
right,
it's
just
it's
a
lot
to
manage,
so
you
have
to
do
a
lot
of
things
to
handle
that
scale.
I
think
there
was
actually
a
question
in
the
chat
about
how
does
how
does
delta
handle
this
so
like,
for
example,
when
you're
writing
the
describe
details
command,
it
actually
isn't
pinging
your
metastore,
it's
pinging,
the
transaction
log,
which
is
wherever
your
blob
storage.
Is
it's
actually
leveraging
the
scalability
of
these
like
provider.
C
B
D
Yeah,
it's
a
good
question,
because
you're
tempted
to
kind
of
meld
them
together
right
because
feature
engineering
is
like.
Is
it
it's
a
complicated
thing
model
tracking
and
experiment
tracking
is
another
complicated
thing.
If
you
want
to
understand
why
your
model
is
performing
poorly,
it
could
be
the
modeling.
It
could
also
be.
D
The
data
you
know
garbage
and
garbage
out
is
one
of
the
most
overused
phrases
in
like
modern
machine
learning,
but
it's
it's
real
that
part
of
why
this
data
quality
concept
exists
is
actually
because,
if
you're
running
a
complex
production
pipeline-
and
you
have
really
terrible
data
coming
in
that's
constantly
out
of
date
or
has
high
levels
of
nulls
like
you
can't
fit
well
to
that
collection
of
data.
But
on
the
modeling
side
I
will
say
you
know
what
I
just
showed
was
local
outlier
factor
detection.
It's.
D
It
can
run
out
of
the
box
and
scikit
in
like
two
lines
in
a
real
kind
of
machine
learning.
Context
you'll
be
running,
really
complicated
models
you
might
be
like
implementing
neural
stuff
with
tensorflow
or
pytorch.
You
might
have
like
multiple
modules
of
hundreds
of
lines
of
code
to
like.
Actually
build
those
models
in
in
kind
of
the
specific
time
series
anomaly
detection
case:
you'll
have
probably
a
lot
of
seasonality
like
if
you're
doing
this
for
real.
D
You
would
be
looking
at
things
like
monthly,
weekly,
daily
trends
and
trying
to
smooth
those
in
order
to
get
anomaly.
Detection
that
respects
the
existing
trends
of
the
data
and
all
of
those
things
I
mentioned
are
hyper
parameters
that
can
go
into
a
model,
and
so,
if
you
are
building
a
model
to
kind
of
solve
for
that,
you'll
want
to
explore
that
space
a
bit
and
say
what
does
it
look
like
if
on
this
table?
Like
I
try
a
much
more
aggressive
seasonality,
smoothing
approach,
maybe
I
implement
monthly
smoothing?
Maybe
I
don't.
D
Maybe
my
training
window
is
smaller
or
larger.
All
of
those
things
are
going
to
impact.
How
well
your
ml
can
perform
in
production
and
you'll
want
an
experiment
tracking
tool
like
mlflow,
to
kind
of
look
at
your
whole
corpus
of
available
detectors
and
say:
okay,
like
we
just
made
this
change,
the
training
window
is
five
days
shorter,
but
now
accuracy
on
this
benchmark
data
set,
we
had
is
down
20.
So
clearly,
we're
missing
some
key
feature
by
changing
that
hyper
parameter.
D
Sorry,
that's
kind
of
a
a
wild
answer,
but
the
space
is
like
pretty
huge
and
and
yeah
you're
right
to
point
out
that
it's
hard
to
sometimes
decouple
those
but
yeah
you
do
want
both.
I
think
you
can't
do
robust
production
machine
learning
without
both
kind
of
a
good
understanding
of
how
your
data
is
coming
in
and
also
a
good
understanding
of
how
your
model
is
performing
to
that
data.
So,
thanks
for
the
question,
oh.
B
A
A
Tech
talk
with
us
and
thank
you
everyone
for
joining,
so
I
dropped
the
youtube
link
of
the
recording
in
the
chat,
and
I
know
there
were
some
links
on
the
slides
so
I'll
make
sure
I
get
the
youtube
copy
updated
with
those
links
and
then,
if
you're
joining
us
on
zoom,
we'll
just
send
a
quick
email
with
the
recording
and
the
links
too.
I'm
since
we're
certain,
since
we're
able
to
do
that
with
that
thanks
so
much-
and
I
hope
everyone
has
a
great
rest
of
your
day.
Thank.