►
From YouTube: Canary Deploys with Kubernetes, Istio and Envoy - Jason Yee, Datadog (Beginner Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Canary Deploys with Kubernetes, Istio and Envoy - Jason Yee, Datadog (Beginner Skill Level)
Working with complex systems means that even well written code can behave in unexpected ways, leading to outages and critical issues. Kubernetes allows you to easily rollout new code and quickly revert to safe, stable releases when issues are identified. But often the damage done in the short period between deploy and rollback is significant. In this session, I’ll show you how to limit the effect of unforeseen issues using canary deploys with Istio. I’ll also cover ways to better monitor your applications in Kubernetes to help you spot potential problems before they affect your entire customer base.
"About Jason
Jason is a technical evangelist at Datadog, where he works to inspire developers and ops engineers with the power of metrics and monitoring. Previously, he was the community manager for DevOps & Performance at O’Reilly Media and a software engineer at MongoDB. He’s currently exploring the world while living as a nomad and would love to hear about the part of the world that you call home."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right
so
I
think
we're
right
at
about
time
right
now
to
start,
if
you're
still
coming
in
there's
a
few
seats
up
here
in
the
front-
and
it
looks
like
there
might
be
a
couple
over
on
the
front
here,
but
welcome
so
you're
here
to
talk
about
Canary
deploys.
So
let's
talk
about
deployments
right.
What
could
possibly
go
wrong?
A
The
answer
is
obviously
a
lot.
How
many
people
have
experienced
a
deployment
failure
within
the
past
week,
yeah
painful?
Hopefully,
none
of
you
are
on
call
right
now,
and
hopefully
that
doesn't
happen
right
now,
but
I
love
this
meme
right
when
I
deploy
to
or
when
I
do
blank
I
do
it
in
production,
the
most
interesting
man
in
the
world-
and
we
can
fill
this
in
with
everything
from
you
know,
just
crazy
stuff
that
happens
when
we're
when
we're
doing
operations
or
when
we're
doing
software
development.
A
But
the
problem
that
we
have
is
this:
when
I
deployed
fraud
I,
do
it
in
production
right
as
much
as
we
would
like
to
match
our
staging
environments,
our
you
know:
development
environments.
Whatever
we
have
to
production
at
the
end
of
the
day,
you
can
only
do
so.
Many
dry
runs
and
things
just
get
missed.
A
So
when
you
deploy
to
production,
it
obviously
happens
in
real
life
and
there's
no
way
to
get
around
that
so
I
think
that's
why
most
of
us
are
here
and
we
want
to
talk
about
canary
deploys
before
you
do
that
a
little
bit
about
me
quickly.
I'm
Jason,
I
work
as
a
Technical
Evangelist
at
data,
dawg
I
call
that
doing
docks
and
talks,
so
I
write
technical
documentation,
which
often
requires
me
to
read
through
our
source
code,
to
understand.
What's
going
on
and
then
I
come
to.
A
Events
like
this
and
I
speak
and
I
try
to
share
my
knowledge
and
the
knowledge
that
we've
gathered
as
an
organization
I'm.
Also
a
nomad,
so
I
travel
around
I
don't
live
anywhere
except
for
out
of
my
suitcase.
So
if
you
like
to
travel,
come
talk
to
me
about
that
later,
I
also
love
whisky
and
I
play
a
lot
of
Pokemon
go.
So
if
you
see
me
wandering
around
later
and
I,
look
lost,
I'm
not
lost
I'm,
probably
just
trying
to
find
a
Pokemon.
That
said,
data
dog
as
I
mentioned,
is
my
employer.
A
Data
dog
is
a
SAS
based
monitoring
platform.
We
do
infrastructure
out
metrics,
we
do
tracing.
We
now
do
logs.
We
just
try
to
make
it
easier
for
you
to
monitor
your
systems,
to
give
you
an
idea
of
the
scale
of
what
we
handle.
We
do
trillions
of
data
points
every
single
day,
hundreds
of
millions
of
data
points
every
second
and
we
are
hiring
so
if
you're
looking
for
new
challenges,
interesting
work
that
you
want
to
do
if
you
want
to
help
other
engineers
monitor
their
systems.
A
A
Obviously
we
want
everybody
to
try
to
use
this
do
play
with
it,
explore
it.
There's
there's
no
other
way
that
we're
gonna
get
feedback
and
have
the
community
keep
contributing
to
it.
But
if
you
just
throw
it
in
production,
bad
things
will
happen,
it's
still
alpha
and
and
very
new.
So
how
are
we
doing
deployments
today?
How
many
people
are
doing
Bluegreen
deployments?
A
Ok,
fair
number
of
people
I'm,
assuming
that
all
of
you
understand
what
Bluegreen
is,
but
if
you
don't
it's
really
simple
idea:
you've
got
a
load.
Balancer
that
sits
out
front
points
to
version
one
or
blue
of
whatever
your
application
is,
and
it's
pretty
simple.
You
just
make
a
switch
right.
We
do
this
on
load
balancers.
We
can
do
this
in
software.
This
is
essentially
a
software
version.
A
We
can
think
of
things
like
Capistrano,
where
you're
just
really
just
directing
a
symlink
to
new
code,
and
so
this
is
traditionally
the
way
that
a
lot
of
us
have
have
done.
Software
deployments,
and
obviously
the
pros
of
this
are
that
you
know,
hopefully,
it's
zero
or
near
zero
downtime
right.
It's
a
configuration,
change,
you're,
just
redirecting
traffic,
not
too
hard,
and
then
you
get
easy
roll
backs
right.
The
idea
that,
if
something
goes
wrong,
you
just
flip
that
switch
back.
It's
like
just
running
an
undo.
A
How
many
people,
though,
have
had
this
experience
right
when
I
say
easy?
Roll
backs,
I
should
say
easy
roll
backs,
because
we've
all
experienced
this.
When
we
do
blue
grain,
you
flip
a
switch
and
whether
that's
in
code
or
whether
that's
with
your
infrastructure,
there's
some
change.
That's
made
that
isn't
reversible,
whether
that's
things
like
updating
tables
in
a
database
and
suddenly
you
try
to
flip
back,
and
your
data
is
all
screwed
up
things
like
that.
How
many
people
are
doing
rolling
deploys.
This
is
we
are
at
cube
con.
A
So
this
is
the
way
that
kubernetes
by
default
does
deployments
pretty
easy
to
understand.
Similarly,
you've
got
load
balancer
in
this
case.
Kubernetes
sits
out
front,
you've
got
a
bunch
of
software,
and
essentially
it
rolls
down.
One
container
puts
up
the
new
one
and
you
keep
running
through
this
until
suddenly
everything's
switched
over,
and
this
is
great
again
very
zero.
A
You
know
little
downtime
and
one
of
the
nice
things
is
that
because
you're
maintaining
both
sets
of
software,
ideally,
the
roll
backs
are
easier
right
because
you
hopefully
have
designed
your
software
to
be
able
to
accommodate
both
versions.
So
so
what
does
canaries
deploys
do
for
us
like?
What's
the
main
difference?
Well,
it
really
is
this
right.
We
understand
that,
what's
your
audience,
look
like
I'm
prone
to
saying
really
really
stupid
things
and,
as
my
job
is
to
get
up
on
stage
and
say
these
things.
A
A
It
looks
really
similar
to
rolling
deploys,
so
you
flip
over
one,
but
the
main
difference
is
we
stop
stop
and
monitor
whatever's
going
on
and
then
you
continue
on
and
maybe
you
double
check
and
then
you
keep
going,
maybe
just
one
more
time
and
then
at
the
end
you
know
you
party
so
canary,
deploys
again
the
benefits.
Are
you
get
a
lot
of
the
similar
benefits
of
rolling
employees,
but
really
the
bigger
benefits
that
you
have
now?
A
Are
you
have
a
small
scope
right,
you're
only
deploying
delimited
people,
and
that
means
your
ramifications-
are
much
more
constrained
easier.
Roll
backs
again.
This
also
happens
with
rolling
deploys
one
of
the
other
things
that
a
lot
of
people
miss
out
on
Canaries
because
of
the
way
they
do.
It
and
I'll
talk
about
strategies
a
little
later,
but
there's
a
lot
of
ability
to
become
more
load
tolerant.
If
you
monitor
the
right
things,
if
you're,
if
you
see
more
resource
use
usage
with
your
Canaries,
you
have
that
time
to
scale
up.
A
If
that's
something
you
want
and
then
concurrency
concurrency
is
another
thing
that
I
think
a
lot
of
people
miss
with
Canaries.
Is
you
can
start
to
essentially
run
your
software
teams
in
in
parallel
and
start
to
work
on
on
various
things
at
the
same
time?
So
what
is
this
any
of
this
have
to
do
with
kubernetes.
Well,
a
lot
of
people,
think
of
kubernetes
as
a
container
Orchestrator
right
when
we
talk
about
rolling,
deploys
and
I
had
that
that
diagram,
you're
thinking.
Okay,
these
are
my
containers.
A
Canari
kubernetes
is
actually
a
service
Orchestrator
and
we
have
to
start
thinking
of
things
in
terms
of
services,
or
maybe
it's
a
squirrel's
Orchestrator.
If
you
want
a
sticker.
I've
got
some
hit
me
up
later,
but
when
we
talk
about
deployments
like
if
we're
thinking
and
in
terms
of
blue
green
for
those
that
are
still
doing,
that,
it
doesn't
really
make
sense
if
we're
thinking
in
terms
of
services
right,
because
you
don't
want
to
mirror
your
entire
service
architecture
right.
This
doesn't
make
sense.
A
A
So
why
do
you
need
a
service
mesh?
Then?
What
does
this
have
to
do?
Well,
kubernetes
is
fantastic
at
rolling
deploys
right.
We
all
know
this
if
you've
ever
done
any
sort
of
upgrades
with
kubernetes,
but
not
so
much
with
canary
deploys
right.
So
if
you've
ever
tried
to
do
a
canary
deploy
with
kubernetes
it's
a
little
tricky
because
it
doesn't
have
that
notion
really
of
pausing
and
waiting
on
a
rolling
deploy.
You
can
pause
a
rolling
deploy,
but
it
happens
quickly.
A
So
you're
not
gonna,
be
sitting
at
your
keyboard
and
telling
kubernetes
to
update
and
then
quickly,
pausing
it
and
then
monitoring
and
then
trying
to
do
that
right,
that's
silly!
So
the
nice
thing
about
service
meshes
like
sto.
Is
they
really
provide
us
that
control
with
with
routing
and
load
balancing?
And
that
really
is
the
crux
of
what
canary
deploys
is,
but
more
than
that
they
provide
a
bunch
of
other
great
features.
Service
discovery
is
another
one,
because,
obviously,
if
we're
doing
routing,
we
want
to
know
where
we're
routing
things
to
timeouts
and
retries.
A
Obviously,
if
all
of
our
traffic
is
running
through
our
service
mesh,
it
gives
us
a
really
great
spot
that
we
can
actually
monitor
all
of
that
so
how's
this
done.
You
start
with
basic
kubernetes.
You
have
all
of
your
workloads,
you
push
them
aside,
you
sidecar!
This
is
what's
called
the
data
plane
and
this
is
actually
envoy.
So
I
know
that
we've
been
talking
a
lot
about
sto
and
I
know.
A
lot
of
sessions
have
been
talking
about
sto
under
the
hood.
For
that
data
plane.
A
A
The
second
is
quota
management
and
that's
essentially
rate
limiting.
So
we
can
say,
for
example,
the
service
can,
you
know,
only
receive
5,000
requests
per
second,
so
it
gives
us
control
over
that
and
then,
as
mentioned,
there's
a
lot
of
telemetry
reporting,
because
it's
really
easy
to
get
that
data
with
the
mixer.
You
have
adapters
currently
the
adapters
that
exist
for
the
precondition.
Checking
it's
really.
These
are
the
ones
that
exist
right
now.
A
There's
the
open
policy
agent
I
think
I
saw
someone
down
here
wearing
the
the
Oh
pleasure,
which
is
awesome,
I
love
that
logo
and
then
there's
our
back
for
the
quota
management.
There's
really
only
two
right
now,
which
is
interesting.
There's
the
in-memory
quota
management,
which
is
the
default
and
then
there's
the
the
Redis
fact
one.
So
just
essentially
two
places
that
you
can
store
those
and
as
I
mentioned
telemetry
is
huge,
so
SEO
started
with
with
Prometheus.
A
Obviously,
and
there
were
stats,
D
they've
added
dog
stats
D,
which
is
what
deta
dog
uses
they've
added
a
ton
of
others,
essentially
whatever
monitoring
product
you're
using
there's,
probably
an
ability
to
get
an
adapter
for
it.
There's
probably
one
already
there,
but
there
there
are
a
bunch
of
them.
You
can
view
them
within
the
the
github
repo.
A
So
that
was
a
lot
of
time
talking
about
the
basics
and
and
hopefully
I
think
you
know
we're
in
day
three
of
cube
con
I.
Think,
most
of
that
you
should
have
known
hopefully,
but
let's
talk
a
little
bit
about
strategies,
because
I
think
this
is
the
most
important
when
it
comes
to
doing
canary
deploys
with
is
do
the
SEO
part
is
easy
and
we'll
cover
that
later,
but
the
hard
part
is
the
strategies,
so
how
many
people
are
doing
canary
deploys?
A
Okay,
just
just
a
few
out
there,
those
people
that
are
doing
canary
deploys
if
you're,
like
a
lot
of
organizations,
most
organizations
that
I
talk
to
are
doing
canary
deploys
pretty
much
at
random,
picking
some
percentage
of
traffic
to
go
somewhere,
and
this
is
this
is
totally
fine
if
you're
talking
about
back-end
services,
but
if
you're
talking
about
canary
deploying
to
your
customers
to
your
end-users,
we
all
know
that
random,
especially
with
computers,
is
generally
not
so
great.
It's
often
not
very
random.
A
There
are
a
lot
of
things
that
we
other
things
that
we
need
to
take
into
account.
So
when
it
comes
to
your
canary
strategies,
you
really
need
to
start
thinking
of
how
they
can
be
representative
and
when
I
say
representative,
you
have
to
think
of
what
your
users
are
like.
You
can
think
of
things
like
geography
and,
if
I'm
rolling
out
a
test
to
some
of
my
users.
A
Well,
if
I'm
rolling
it
out
in
the
middle
of
the
day
am
I
thinking
of
my
users
that
are
currently
in
the
same
country
in
the
same
region
that
I'm
in
or
if
I'm
a
global
company
I
have
to
take
into
account.
What
are
my
users
in
Japan?
How
do
they
use
use
my
product
versus
my
users
here
in
Denmark?
So
that's
one
thing
to
consider.
The
next
is
time.
A
Canary
plays
are
great
because
you
can
deploy
and
then
you
can
monitor,
but
if
you're
deploying
at
a
time
when
a
lot
of
your
users
aren't
actually
using
your
product,
how
does
that
affect
your
canary
plays?
Are
you
getting
misinformation
based
off
of
that,
and
then
you
have
to
think
about
usage
patterns
as
well.
So
what
times
do
your
users
typically
use
your
product?
How
do
they
use
it
in
what
ways?
A
The
other
thing
that
you
have
to
consider
when
you're,
when
you're
doing
a
canary
to
deploy,
is
the
granularity.
So
a
lot
of
people
again
if
they're
doing
just
a
percentage.
That's
easy
to
scale
up
right.
You
can
say,
let's
start
with
deploying
to
five
percent
of
my
users
and
then
go
up
to
10
percent
and
continue
building
up.
But
if
we're
thinking
of
being
representative,
if
we're
thinking
of
how
are
my
end-users
using
this
product,
what
do
I
actually
want
to
measure?
A
Then
how
do
you
do
the
granularity?
So
there
are
some
interesting
ways
that
you
can
think
about
this.
For
example,
a
GrubHub
which
is
one
of
our
customers
they
canary,
deploy
to
small
cities
first,
so
that's
their
granularity
they'll,
deploy
to
a
small
city
measure
that
deploy
to
a
larger
city
or
to
a
province
deploy
to
countries
deployed
globally,
so
think
about
how
you're
gonna
your
scale,
your
your
canary
sets
and
then
the
final
one
is
resource.
Mapping
and
I
think
this
is
really
particularly
interesting,
because
if
your
canary.
A
Encapsulating
packed,
and
as
mentioned
before,
the
great
thing
about
using
Canaries
is
that
you
can
now
start
to
scale.
So
if
you
roll
something
out-
and
you
see
that
it's
hammering
a
database,
that
may
not
be
a
good
thing,
but
potentially,
if
it's
a
significant
change,
a
small
increase
in
in
access
to
a
data
store
might
be
okay,
and
so
you
can
start
to
accommodate
for
that
right,
as
you
see
that
metrics
start
to
spike
up
or
or
trend
upward.
A
So
let's
talk
a
little
bit
about
what
your
monitoring
strategy
should
be
right.
The
whole
point
about
Canaries
is
to
deploy
a
little
and
monitor
it.
So
the
first
thing
that
we
need,
we
absolutely
need
is
tags
or,
if
you're,
using
Prometheus
using
labels.
You
have
to
understand
where
your
data
is
coming
from
what
it
actually
means.
A
So
if
you
think
of
a
data
point
or
a
metric
as
a
thing
and
a
number
and
when
it
happened
well,
that's
great,
but
you'll
never
be
able
to
find
that
data
unless
you
actually
tagged
it
or
label
it,
especially
if
you're
starting
to
do
concurrent
canari
deploys.
The
next
thing
that
we
need
to
monitor
is
starting
to
look
at
those
extremes
as
we
as
we
deploy
smaller
pieces.
A
If
we're
looking
only
at
the
the
p90,
the
90th
percentile,
we're
gonna
miss
a
lot
if
you're
looking
at
the
P
95,
depending
on
your
canary
size,
you're
also
gonna
miss
a
lot
so
start
to
think
about
pushing
out.
What's
your
P
99
look
like
because
that
that
single
user,
or
that
handful
of
users
that
are
experiencing
problems
with
your
canary
deploy
as
you
continue
to
roll
out
that
grows
really
rapidly
start
thinking
about
outliers
outliers
I
like
to
say,
is
one
of
these
things
is
not
like
the
other.
A
It's
pretty
pretty
understandable
right
if
we're
thinking
of
canary
deploys
what
does
my
canary
deploy
look
like
versus
my
production?
Really
simple.
So
if
you're
monitoring
tool
has
outlier
detection
make
use
of
that
the
others
anomalies,
so
the
difference
between
outliers
and
anomalies
outliers
is
one
of
these
things
doesn't
look
like
the
other
anomalies.
Is
this
thing
doesn't
look
like
it
has
in
the
past,
so
enable
anomaly
detection
if
your
monitoring
tool
has
it,
there
are
also
open
source
projects
out
there.
A
A
So
what
signal
should
we
watch
I,
particularly
like
the
the
four
golden
signals
from
the
Google
SR
eBook
I,
think
they
really
match
when
it
comes
to
canary
deploys,
and
the
first
is
that
you
should
look
for
our
changes
in
latency
right.
This
is
pretty
obvious
if
you're
making
a
canary
deploy
and
suddenly
things
are
slower
for
those
people.
That's
that's
going
to
be
a
problem
so
monitor
for
that
latency.
A
The
next
of
the
signals
is
errors,
and
so
a
lot
of
people
when
they
think
of
errors
that
they're
looking
at
error
rates.
One
thing,
though,
that
I
find
particularly
helpful
is
not
to
look
at
the
error
rates,
but
look
at
the
types
of
errors.
When
you're
rolling
out
new
code,
you
might
get
errors,
but
they'll
often
not
be
the
same
errors
that
you
saw
before
right,
because
you're
doing
new
things
so
check
for
different
errors.
A
So,
let's
talk
a
little
bit
about
sto,
let's
jump
into
it
installing
sto,
if
you
haven't
done
it
because
I,
you
know
there
weren't
a
lot
of
hands
when
I
said
or
when
I
was
asking
about
that,
but
download
it
unpack.
It
add
it
to
your
path
and
really
just
apply
it.
So
what's
this
actually
do
well,
when
you
apply
it,
it
sets
up
a
few
things
and
the
first
is
sto
pilot
sto
pilot
is
essentially
what
handles
the
the
Envoy
API.
It's
also
the
interface
between
your
platform
between
kubernetes
and
sto.
A
A
So
how
do
you
actually
run
this
well
really
really
simple.
We're
all
familiar
with
with
cube
CTL
cube,
cuddle,
cube
control.
However,
you
like
to
say
it
really.
You
just
pass
your
EML
files
through
the
ISTE,
OCT
l
and
it
injects
things
and,
and
then
it
just
like
magically,
does
its
thing
right.
I
always
often
compare
this
to
when,
when
people
tell
you
to
just
like
curl
and
pipe
that
to
bash,
and
just
like
trust
me,
so
what's
that
actually
do
well,
it
is
pretty
benign.
It
adds
really.
A
The
cool
thing,
though,
that
I'm
pretty
excited
about
is
Auto
sidecar
injection
and
this
is
fairly
new,
and
so
it
starts
with
mutating
web
controllers,
so
mutating
webhook
admission
controllers.
What
that
really
does
is
it
sits
in
between
the
authentication
for
kubernetes
and
before
that
object
is
persisted.
It
allows
things
to
modify
what
goes
into
kubernetes.
A
So
this
is
beta
with
kubernetes.
1.9
is
when
this
came
out,
and
so
it's
still
being
worked
on,
but
it's
actually
surprisingly,
fairly
stable
from
from
what
I've
been
using
it
for
and
just
playing
around
with
it.
So
canary
deploying
with
this
do
how's
this
done.
Well,
you
start
off
with
a
service.
The
service
doesn't
actually
get
changed.
This
is
all
standard
kubernetes,
the
deployments.
So
you
you
set
up
two
deployments
right
and
there's
not
a
whole
lot.
You
just
have
to
create
version.
One
version
two.
A
This
is
actually
how
you
would
do
carry
deploys
with
straight-up
kubernetes.
The
only
difference
is
with,
if
you're
trying
to
do
this
with
standard,
kubernetes
and
canary
deploy.
Well,
you
would
have
to
manage
both
of
these
right
and
you
would
set
your
replicas
and
you
just
have
to
manually
say
that
I
want.
You
know
ten
replicas
of
version
one
and
one
version
two
and
then
you'd
have
to
Manly
manually
tweak
that
down
to
nine-
and
you
know
three
or
you
know
constantly
shift
that.
A
So
that's
annoying
and
that's
what's
great
about
Sto,
because
with
this,
do
you
have
route
rules
and
makes
it
easy?
So
you
start
off
with
one
route.
Rule
sends
a
hundred
percent
of
your
traffic
to
version
one,
and
it
really
is
that
easy,
adding
another
route
rule
sending
a
percentage
of
traffic
to
version
two.
A
The
other
really
cool
thing
about
SEO,
though,
is
as
mentioned.
You
don't
want
canary
ploys
to
just
be
a
percentage,
so
you
can
really
do
really
cool
things
with
this.
Do
like
having
more
complex
route
rules
set
a
header
there.
There
are
great
libraries
out
there
to
detect
where
your
users
are,
where
they
are
from
their
browser
and
set
a
header
for
that
start,
redirecting
your
users
based
on
geography
or
based
on
IP,
based
on
loads
of
other
things.
A
So
what
else
can
this
you
do?
It
can
do
a
ton
of
stuff
really,
as
mentioned
before,
get
out
there
and
play
with
it.
So
that's
the
the
link
to
the
docs
go
play
with
it,
see
what
you
can
do
so
just
a
quick
recap
service
meshes.
Why
do
you
want
them?
Well,
they
give
you
more
control,
particularly
when
it
comes
to
canari
deploys.
So
when
you're
doing
your
Canario
deploys
as
you're
getting
that
control.
A
Think
about
how
you
can
be
more
representative
with
your
users,
how
you
select
that
canary
deploy
set
when
it
comes
to
the
monitoring
and
the
monitoring
is
the
hardest
part
of
canary
deployments
really
is
to
figure
out
how
you're
gonna
monitor
things
and
start
with
those
tags
or
those
labels.
If
you're
using
from
Atheists,
then
thinking
about
those
outliers
and
anomalies
so
find
those
tools
that
will
get
you
that
algorithmic
detection
or
that
machine
learning,
detection
and
then.
Finally,
what
to
watch
for
take
a
look
at
that
latency.
A
Take
a
look
at
the
types
of
errors
that
you're
receiving
the
traffic
that
you're
seeing
and
the
saturation
behind
those
resources.
All
that
said
it
is,
may
the
fourth
may,
the
fourth
be
with
you
and
I'll.
Do
some
questions
and
as
I
do
that
so
I've
been
told,
use
the
mics
or,
if
you
can't
get
to
a
mic,
because
it's
pretty
full
in
here
yell
it
out
and
I'll
repeat
it,
and
for
those
of
you
that
we're
taking
photos
of
the
slides
there's
the
link
to
download
them
Thanks.