►
From YouTube: Webinar: Developing an Effective Observability Strategy for Kubernetes + Service Mesh Environments
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes has emerged as the de-facto standard for orchestrating containerized applications. While it abstracts away many complexities, it also introduces new operational and monitoring challenges. In fact, the latest CNCF survey cites monitoring as one of the biggest challenges in Kubernetes adoption.
In this webinar, Arijit Mukherji and Maxime Petazzoni will walk through the changes in requirements for monitoring Kubernetes and service mesh environments, typical challenges that enterprises face on their journey to cloud-native and how to navigate around those challenges leveraging CNCF projects such as Prometheus, OpenTracing, and Envoy.
You will also see a demo showing the best practices in monitoring Kubernetes environments using metrics, events, and traces.
A
Okay,
I
I
think
we
should
go
ahead
and
get
started
in
hello
and
welcome
to
today's
CN
CF
webinar,
developing
an
effective
observability
strategy
for
your
kubernetes
and
service
mesh
environments.
My
name
is
barks
at
the
Gorske
I
am
the
head
of
developer
relations
with
Jeff
frog
and
a
very
proud
CN
CF
ambassador
and
I'd
like
to
thank
everyone
who
is
joining
us
today.
We
would
like
to
welcome
our
presenters
today,
a
Arijit.
B
C
A
No
worries
before
we
start
just
a
couple
of
housekeeping
items
and,
as
you
already
noticed,
the
whole
of
you
are
on
mute
and
that's
to
ensure
audio
quality.
It
doesn't
mean
that
we
don't
want
to
hear
from
you.
The
opposite
is
right.
We
do
want
from
hear
from
you
and
there
are
two
channels
for
that.
There
is
in
Q&A
a
button
in
your
zoom
panel
and
that's
for
the
questions
that
we
will
try
to
answer
as
much
as
possible
live
and
whatever
we
won't
have
time
to
answer
wife.
A
We
will
answer
afterwards
and
a
follow-up
email.
So
that's
kind
of
about
the
questions
of
the
topic
of
the
webinar
and
there
is
a
chat
window
or
button
for
everything
else,
like
audio
call
do
quality
or
there
is
no
sound
or
anything
like
that.
So
QA
for
questions
and
answers
towards
the
end
and
chat
for
everybody
else.
A
B
You
Baruch
and
hello.
Everyone
from
signal
effects
I'm
here
with
my
co-workers,
Max
and
yang,
so
single
effect,
is
a
company
that
provides
monitoring
at
a
fast
service
for
this
kind
of
environment
that
we're
gonna
talk
about
still,
let's
talk
about
in
an
effective
observability
strategy
for
these
end
markets.
What
should
you
be
thinking
about?
So
we
have
a
few
things
to
cover
we're
going
to
talk
about
thing
of
why
observability
sort
of
frame
the
problem
a
little
bit
and
what
makes
it
challenging
for
cloud
native
environments.
How
does
kubernetes
allow
itself
to
be
monitored?
B
How
should
you
be
thinking
about
that
and
then
lately
distributed
tracing
is
becoming
more
and
more
of
an
important
aspect
for
observability
strategy.
So
max
is
going
to
talk
about
that
and
then
we're
going
to
end
with.
You
know
how
the
service
mesh
might
help
us,
but
it's
not
at
all.
When
we
talk
we're
going
to
have
a
yan,
do
a
demo
at
the
end,
to
talk
about
some
of
the
principles
and
best
practices
that
we're
going
to
fit
on
to
show
you
what
that
means.
So
hope
that
makes
sense.
B
So
why
observability
it
really
provides,
are
sort
of
these
three
things
in
general.
It
allows
us
to
understand
how
my
infrastructure,
my
applications,
are
behaving
and
why
it
allows
us
to
do
data-driven
decisions
right.
So
it's
not
just
for
monitoring,
and
you
know
that
keeping
the
lights
on
it
has
many
other
purposes
which
we
will
go
into
it,
but
we're
not
always
looking
at
charts
and
outputs
on
the
screen,
so
alerting
allows
us
to
be
notified
when
things
are
not
right.
B
So,
for
example,
if
my
service
level
objective
is
being
breached
by
alerting
on
my
SLI
is
I'm
able
to
notice
when
that
happens,
and
that's
very
important,
because
unless
I
know
something
is
wrong,
how
do
I,
how
can
I
go
in
and
fix
them?
So
not
only
do
we
need
high
quality
alerting,
it
needs
to
be
tiny,
because
otherwise
I
mean
this
world
of
you
know
SAP
services
and
whatnot,
where
everybody
is
providing
SLA.
B
Is
you
need
your
mean
time
to
resolution
to
be
as
low
as
possible,
right
speaking
of
mean
time
to
resolution
when
you
do
have
a
problem?
How
do
you
know
where
the
problem
was
or
what
the
problem
was
so
again?
Observability
helps
us
get
clues
into
that,
so
it
is
pretty
broad
and
it
gives
us
a
lot
of
different
aspects
that
we
are
with
that
really
while
to
run
our
our
infrastructure
and
applications.
Now,
if
you
are
thinking
about
an
absolutely
strategy,
you
have
to
have
all
these
different
aspects
of
it
in
place.
B
You
must
have
logs,
you
must
have
metrics
and
you
must
have
some
way
of
transaction
profiling
and
distributed
traces
along
with
your
monitoring
system.
You
typically
won't
be
able
to
really
be
effective
if
you
had
just
one
or
two
of
these,
for
example,
now
there's
a
little
bit
of
confusion
in
the
industry,
because
you
know
logs
can
produce
metrics
or
places
can
be
considered
as
logs
and
and
so
on,
but
in
general,
these
three
technologies-
these
are
the
pillars
and
they
do
provide
three
different
aspects
of
the
information
that
you
need.
B
So
when
you're
setting
up
your
strategy,
you
must
think
about
all
of
them.
So
how
do
we
think
about
them
right?
We
think
of
metrics
as
the
sort
of
the
canary
in
the
coal
mine.
This
is
the
thing.
That's
telling
you
from
a
high
level
how
things
are
doing
and
it's
allowing
you
to
get
alerted
when
things
are
wrong,
so
they
are
sort
of
anonymized
in
the
sense
that
they
don't
give
you
the
specific
details,
but
they
tell
you
when
something
is
off.
That's
metric
do
I
have
a
problem,
but
then,
where
is
the
problem?
B
Our
environments
are
very,
very
complicated.
Now,
like
we
have
so
many
different
micro
services
and
complex
dependencies,
so
understanding
where
the
problem
is,
that's
where
this
routed
tracing
helps
us,
but
then,
even
if
even
if
I
knew
that
minimum
micro
service
bla
had
a
problem,
what
was
the
specific
problem?
What
was
that
inner?
What
was
the
exception
and
so
on?
B
So
that's
where
logs
coming
to
give
us
the
sort
of
the
forensics
that
we
need
to
understand
what's
going
on,
and
obviously
our
monitoring
system
sits
on
on
top
of
these
and
it
provides
us
the
monitoring
capabilities
on
top
of
the
data.
From
these,
as
I
mentioned,
observability
has
many
many
different
use
cases.
It's
not
just
about
the
inner
monitoring
and
response
so
on
it
can
help
you
with
testing.
So,
for
example,
if
you
doing
AV
testing,
you
want
to
know
how,
in
a
version
it
compares
against
version
B.
B
Well,
how
do
you
do
that
is
by
absorbing
both
these
releases
and
by
seeing
how
they
compare
with
each
other
capacity
planning
is
also
interesting.
Typically,
when
we
think
of
observability,
you
know
we
may
think
about
in
a
short
time
period.
You
know
what
happened
in
the
last
hour
or
day
or
so
on,
but
is
actually
far
more
important.
Other
use
cases
too,
for
example,
with
capacity
planning.
B
If
I
had
a
trend
of
how
your
infrastructure
and
applications
have
been
behaving
over
time,
then
you
know
that
will
based
on
the
trend,
maybe
I'm
going
to
have
so
many
X
more
requests,
or
you
know
my
my
resources
are
getting
used
up
so
I'm.
How
do
you
capacity
plan?
How
do
you
decide
how
much
infrastructure
to
deploy?
So
that's
you
know
observability
can
help
you
with
that
developers
as
they're.
B
Developing
new
features
again
would
use
all
these
systems
to
understand
whether
the
code
that
they're
writing
is
perhaps
performant
or
whether
it's
accurate
and
so
on.
So
overall
observability
is
not
just
for
the
day-to-day
operation.
It
is
for
far
more
right,
so
but
you're
probably
not
doing
all
of
them
right
now
or
maybe
some
of
you
are-
and
maybe
some
of
you
are
not
so
what's
the
observability
maturity
curve,
what
does
good
look
like
in
this
space
right?
So
typically,
these
are
the
curves
that
we
got
from
water
and
I.
B
Think
I
agree
with
this
in
the
sense
that
it
starts
off
with
the
basics.
It
starts
off
with
infrastructure.
You
know
and
starts
off
with
monitoring
your
cloud
services
at
that
point
more
as
black
boxes,
but
then
at
some
point
you
realize
that
goodbye
infrastructure
is
interesting,
but
my
own
application
is
more
interesting,
so
I
need
to
be
start
monitoring
that
so
that's
when
you
might
say
less,
they
use
a
traditional
ATM
tool
which
is
instrumenting
your
code
and
giving
you
insight
into
that.
B
But
as
an
organization
matures,
you
will
find
that
you
know
this
startup
adopting
divorce
methodologies
and
that's
where
the
developers
start.
Having
a
same
this,
they
start
instrumenting
their
application
to
produce
the
things
that
produce
the
telemetry,
that
they
then
depend
on
to
make
sure
that
they
are
able
to
operate
it
well,
because
our
environments
are
large
and
complicated,
because
we
have
way
more
containers
than
we
used
to
have
servers
back
in
the
day.
Without
automation,
it's
not
something
that
you
can
scale.
B
B
Ops
or
artificial,
intelligent,
driven
ops,
the
idea
there,
if
you
want
to
you
data
science
and
machine
learning,
to
have
your
observability
system
even
make
sense
of
your
environment,
because
even
if
I
say
well,
I'm
gonna
monitor
service
X.
What
threshold
should
I
set?
Even
that
can
be
complicated
in
highly
dynamic
environment,
so
with
AI
ops,
you're
able
to
have
the
system,
learning
behavior
and
then
do
the
right
thing
and
set
the
right
order
for
you
right
and
then
obviously
the
last
stage
is
business.
B
Where
I
mean
we
are
all
in
a
various
organizations,
they
have
a
purpose.
They
have
a
business
purpose.
How
do
we
tie
that
back
into
all
of
these
stuff?
That
I
talked
about,
for
example,
how
much?
How
many
products
did
I
sell?
How
many
failures
in
the
in
the
in
the
cart
happen
and
how
do
I
tie
that
into
what
I
am
monitoring?
So
that's
one
of
the
maturity
curve.
You
obviously
cannot
go
from
this
stage.
You
want
to
the
stage
end
right
away.
B
There
has
to
be
some
level
of
a
you
need
to
follow
the
steps
so
to
speak
either
a
walk
before
you
run
I
feel
that
an
organization
needs
to
be
in
one
of
these
stages
for
a
little
while
to
sort
of
stew
in
it
for
to
get
the
engineers
used
to
it
before
they
realize
the
importance
and
how
to
do
the
next
steps.
Well,
so
in
terms
of
where
you
would
want
to
go,
what
good
looks
like
this
is
a
is
a
pretty
good
indication
of
that.
B
So,
let's
talk
about
metrics,
so
in
general
you
know
I
mentioned
logs
metrics
and
traces
in
this
talk,
we're
going
to
primarily
focus
on
metrics
and
tracers.
So
if
I
stick
to
say
that
blogs
are
also
very
important
and
you
must
have
a
good
logging
infrastructure
set
up,
but
that's
not
the
I'm
part
of
the
discussion
today.
So
when.
D
B
Unlucky
because
you
got
to
collect
them,
you
know
all
your
stuff
is
running
on
posts
and
virtual
machines.
So
you
gotta
monitor
those
those
things
are
running
containers
you
gotta,
monitor
those.
The
containers
are
running
my
application
or
third-party
OSS.
Like
you
know,
carpi
lactic
searching
you
gotta
monitor
those
they're
running
in
a
cloud
environment,
so
you
gotta
money
to
the
cloud
infrastructure
from
what
the
you
know.
It
gives
you
the
business
metrics
we
talked
about
of
course,
but
wait.
That's
not
all
I
mean
if
you
have
an
APM
system.
B
If
you
have
logs,
you
can
create
metrics
out
of
those
too.
So
really
when
you
are
thinking
about
a
metric
strategy,
you
need
to
think
about
this
holistically.
You
need
to
be
able
to
collect
all
of
this
together,
because
otherwise
effective
observability
may
not
happen.
So
then
it
may
feel
like
a
tall
talk.
So
how
would
you
then
collect
metrics?
What
metrics
would
you
collect
from
these
places?
So
here
are
some
best
practices
and
methodologies
for
how
to
collect
metrics
from
these
different
systems.
B
If
it's
the
resource,
for
example,
like
a
disk
or
a
host,
you
might
use
the
use
method,
utilization,
saturation
and
errors
when
it
tells
you
know
how
much
a
resource
is
used,
how
much
backlog
is
still
has
to
work
through
and
how
much
errors
are
happening.
Another
good
one
for
your
own
applications,
which
are
not
sort
of
resources,
perhaps
like
things
like
API,
is,
is
the
read
method
or
rates
errors
and
duration.
B
How
many
calls
am
I
having
how
many
of
them
are
failing
and
how
long
are
those
calls
taking,
because
that
way
it
a
lot
of
issues
may
not
have
to
do
with
errors
but
more
with
latency.
So
this
captures
it
very
well.
Google
is
sorry.
You
know
follows
the
four
golden
signals,
as
they
call
it,
which
is
latency
traffic
errors
and
saturation,
which
is
kind
of
similar
to
the
other
one,
and
then,
of
course,
there's
business
metrics.
B
So
the
important
thing
here
is:
you
must
have
a
standardized
strategy
more
than
you
know
which
specific
methodology
you
use
engineer's
need
to
frequently
look
at
other
systems
or
adjacent
systems.
Unless
you
have
a
standard,
then
you
may
not
maintain
a
base
level
of
quality
and
at
the
same
time
they
may
not
know
what
to
expect
when
they're
looking
somewhere
else
having
a
standard
makes
a
whole
lot
of
sense.
B
When
environments
are
not
simple
and
if
you
look
at
the
large,
your
enterprises
that
number
actually
grows
up
and
the
reason
is
I
mean
larger
enterprises
have
to
be
with
way
more
other
complexity.
They
have
perhaps
deployed
in
many
regions
in
many
clouds,
they
are
probably
many
more
micro
services.
They
are
using
many
more
languages,
there
are
more
many
more
users,
so
this
gets
harder
and
harder
right
and
the
reason
monitoring,
hakuna
the
environment
by
the
community
gives
us
a
lot
of
good
things
out
of
the
box.
B
It's
also
a
more
complex
thing
to
to
monitor,
because
we
have
to
monitor
every
single
layer
that
I
talked
about.
You
gotta
monitor
the
micro
service
with
your
applications
at
the
top
they
running
on
the
containers.
You
gotta,
monitor
those
you
gotta
monitor
the
Commodities
platform
itself
and
then
obviously
the
substrate,
the
infrastructure
on
which
it
is
running
on.
So
all
of
this
needs
to
be
done
and
this
to
be
done.
Well,
so
let's
look
at
it
in
the
kubernetes
Orchestrator.
You
still
look
at
how
many
different
sub
components.
B
You
not
don't
have
to
be
an
expert
in
all
of
these
because,
frankly,
it's
the
job
of
the
orchestrator
to
take
care
of
much
of
this,
for
you
and
abstract
it
for
you,
but
there
are
still
certain
basic
primitives
that
you
must
be
aware
of
that.
You
should
be
able
to
do
monitor.
For
example,
you
know,
there's
a
master,
that's
doing
that
doing
all
this
orchestration
is
deploying
services,
as
deployments
you
gotta
be
able
to
have
a
control
on
those.
The
deployments
are
happening
in
multiple
ponds.
The
pods
contain
containers.
A
B
Monitor
and
get
metrics
from
if
you're
talking
about
metrics
now
lucky
will
be
the
coolest
project
he
says,
has
taught
about
observability
from
the
ground
up
and
all
of
these
subsystems
they
make
available
telemetry
and
metrics
for
you,
and
they
also
do
it
in
a
standardized
fashion.
So
at
least
much
of
the
is
the
job
of
creating
those
metrics
etcetera
is
done
for
you,
you
have
to
collect
them
and
they
make
them
available
in
a
standardized
format.
B
The
promises
exhibition,
format
and
Prometheus
is
actually
the
second
project
that
graduated
out
of
the
CN
CF
after
communities
itself,
which
tells
you
how
important
observability
is
to
that
ecosystem.
Prometheus
is
a
time
series
monitoring
system
any
good
qualities
associated
with
it.
It's
multi-dimensional,
which
means
that
you
can
pull
in
all
these
label.
That
kubernetes
applies
to
the
various
resources
and
use
them
to
color.
The
metrics
that
you
are
looking
at.
B
It
has
deep
qualities,
integration
as
I
said
every
single
component
that
makes
Prometheus
readable
metrics
available
for
you,
and
it
also
supports
all
the
multiple
layers.
You
can
go
and
observe
all
the
different
layers
that
I
just
talked
about
in
the
past.
So
what
are
what
are
some
of
the
ways
to
collect
metrics
from
these
different
layers?
So
if
you
were
to
start
off
with
monitoring
metrics
on
a
kubernetes
environment,
you
obviously
typically
row
meteors
in
order
to
get
the
data
from
the
nodes
of
them.
Steps
like
you,
know,
CPU
memory
etc
from
both
them.
B
So
there's
a
process
called
the
Nordics
Porter,
which
you
can
you
know,
run
as
a
daemon
set
on
all
the
nodes,
and
that's
going
to
get
you
those
metrics.
If
you
are
running
third-party
OSS
apps,
there
are
various
other
promises,
exporter
that
run
as
a
sidecar,
along
with
them,
for
example,
caca
and
which
will
make
the
matrix
from
those
systems
available
to
you
in
a
primitive
readable
format
on
the
containers
themselves.
The
cubelet
process
that's
running
on
the
node.
It
has
a
C
advisor
built-in
which
will
give
you
information
about
the
containers
in
there.
B
Then
there
the
the
orchestrator
itself
makes
many
other
what
are
called
cube
state
matrix
available
to
you.
So
these
are
metrics
that
are
created
by
rocking
the
state
of
the
kubernetes
Orchestrator
principles,
how
many
pods,
how
many
nodes,
how
many
containers
in
a
pod,
how
many
pods
in
a
deployment
and
so
on,
so
it
parses
that
metadata
and
makes
metrics
out
of
it.
B
So
you
need
to
be
able
to
get
all
of
this
into
your
system
and
then,
if
you
look
at
your
own
application
code,
each
has
libraries
in
all
the
major
languages
so
that
you
can
instrument
your
code
to
report
counters
and
gauges
and
histograms
and
timers.
And
what
have
you?
So
it's
a
pretty
well-defined
and
complete
ecosystem
in
context.
What
the
data
is
in
there.
You
are
able
to
process
them.
You
are
able
to
analyze
them
kun
it.
B
So
if
Prometheus
has
a
language
called
prompt,
Euler,
Prometheus,
query
language
and,
as
the
examples
here
shows,
you
can
do.
Various
things
like
transformations
and
mathematical
operations
can
combine
metrics
and
do
various
other
interesting
things.
So
this
will
allow
you
to
have
higher
level
of
the
ability
and
a
lot,
but
there
are
some
challenges
with
it.
It's
obviously
you
know
it
doesn't
have
any
visualization,
so
you
probably
going
to
use
crapola.
It
does
not
have
security
and
authentication
etc.
Built-In,
so
you
know,
you'll
have
to
figure
that
out.
B
If
that's
important
to
you
in
terms
of
long
term
storage,
it
starts
getting
interesting
because
it
uses
the
local
file
system.
So
typically,
you
may
not
store
a
very,
very
long
amount
of
data
in
it
and
then
it's
also
not
a
full-fledged
monitoring
system.
It's
just
for
matrix.
So
it's
not
going
to
tell
you
things
about
the
dependencies
among
your
micro
services
and
so
on
right.
B
So,
if
that
happens,
you
can
obviously
decide
to
build
some
of
these
functionalities
on
your
own
or
you
can
use
sort
of
other
sort
of
vendor
solutions
and
so
on,
which
there's
a
pretty
healthy
ecosystem
around
there.
For
example,
signal
FX
has
also
a
pretty
good
Prometheus
and
kubernetes
situation
built
in
at
this
point,
I'm
going
to
hand
it
over
to
max
who's
going
to
talk
about
another
aspect,
which
is
that
the
district
tracing
and
transaction
manager
thank.
C
You
so
this
has
metrics
or
a
very
important
contributor
to
your
observability
strategy.
There
is
either
fast
flexible
and
they
can
give
you
a
lot
of
visibility
into
the
activity
that
behavior
and
performance
of
your
applications,
but
what
it
really
is
providing
you
that
end-to-end
view
of
application
performance
that
it
goes
across
all
the
backend
visited
systems
that
are
involved
in
answering
user
requests.
There.
C
C
Take
on
the
challenge
of
monitoring
and
alerting
micro
services
at
scale,
as
we
start
to
break
out
we're
all
in
one
with
the
complications.
It's
more
flexible
and
lasting
architectures
or
services
functions,
etcetera.
It
becomes
really
difficult
to
make
sense
of
traditional
mentoring
system,
black
metrics
and
locks.
They
like
contact
and
they're,
often
not
sufficient
to
triage
and
city
serving,
would
cause
us
to
risk
countries.
Mister
drifting.
C
On
the
other
hand,
captors
application
performance
information
across
all
the
services
involved
in
a
transaction,
preserving
that
context
and
in
a
lot
of
service
defenses,
so
why
they
should
be
trading
back
to
a
fictional
company
octa-core.
Their
back-end
is
made
up
of
some
of
micro
services
and
systems
that
communicate
to
each
other.
For
example,
the
from
any
web
service
called
in
to
their
application
of
the
recommendation
service,
which
in
turn
is
called
into
account
service.
That
looks
count
information
into
a
profile
database.
C
If
that
database
starts
having
issues
and
all
the
upstream
services
infection
will
certainly
experience
that
limitation,
multiple
adders
might
fire
and
the
end
users
will
start
to
deteriorate
in
user
experience.
So
accompaning
establishes
is
not
anything.
Any
furtive
relies
solely
on
metrics
would
require
a
lot
of
tribal
knowledge,
but
all
of
those
services
interact
each
other
to
understand
that
the
database
invitation
is
a
real
issue
here.
C
System
tracing,
on
the
other
hand,
by
capturing
application,
performance
of
commission
and
the
transactional
level
gives
you
an
end-to-end
view
of
how
a
particular
request
was
handled
all
those
services.
Where
does
that
was
spent?
What
was
the
cold
chain
weathers,
timeout
and
error
as
the
next
era,
even
without
any
architectural
knowledge,
I
didn't
find
the
root
cause
of
an
inference.
Regulation
becomes.
D
C
Raiser
hold
spans
and
they
can
be
represented
in
poultry
that
the
one
should
here
on
the
route,
but
when
facing
is
instead
project
that
defines
the
semantics
and
crowning
interfaces
for
just
tracing.
It
also
offers
a
broad
collection
of
instrumentations
for
commonly
used
open
source
frameworks
and
libraries
that
you
may
be
using
in
your
applications,
but
stops
we
improve
the
coverage
and
detail
at
all
of
the
traces
of
the
capture
from
your
systems.
Additionally,
open
tracing
is
helping
to
define
the
standard
for
context.
Propagation
opt
context.
C
C
What
good
looks
like
for
those
challenges
and
what
others
I
mean
in
practice
in
the
demo
that
we
have
at
the
end
of
the
webinar?
The
first
one
is
about
sampling
trays.
It
is
much
more
complex
in
detail
than
metrics,
so
this
is
typically
higher
cost
to
storing
a
large
volume
traces
depending
on
the
volume
of
requests
ended
by
your
application.
You
might
only
want
to
store
a
fraction
of
the
traces
using
Ken's
base
sampling
with
the
decision
to
record
and
keep
a
trace
is
made
at
the
beginning
of
the
transaction.
C
A
C
No
knowledge
of
the
relationship
between
services
will
typically
need
to
alert
storms
caused
by
the
kept
getting
picture
of
theory
of
this
decision
tree
having
and
with
those
isolation
or
the
top
priority
is
in
his
name
cells.
You
need
to
make
sure
that
you're
tracing
solution
actively
helps
you
reach
those
goals,
instead
of
making
them
harder,
leaving
you
in
a
middle
in
the
hence
action
areas
that
fall
on
your
GTR.
C
So
what
a
good
look
like,
what
are
the
key
contributors
to
ana
visibility,
strategies
that
deliver
compounds
value
for
your?
The
first
thing
is
tell
based
dentling
for
traces
by
recording
and
analyzing
a
home
percent
of
the
traces
and
making
a
small
sampling
incision
with
a
trace
is
a
symbol.
You
can
dramatically
improve
the
quality
of
the
tracing
data
that
you
have
at
your
disposal
without
increasing
your
storage
costs,
which
tell
based
on
playing,
you
can
intelligently
select
interesting
traces
that
are
worth
keeping.
C
Those
that
represent
response
time
at
higher
is
doesn't
contain
errors
exeter
next,
is
by
connecting
and
when
you
drink
and
loading
systems
with
to
reach
dependency,
knowledge
that
is
inherently
provided
by
your
district
tracing
data.
This
allows
you
to
build
smart
of
alerting
but
medically
mute
alerts.
Ownership
services
when
the
root
cause
is
really
with
that
database
at
the
bottom
of
the
call
stack
while
giving
you
a
detailed
context
and
impact
information
along
with
the
alert
spot
that
database.
C
That
will
significantly
reduce
the
times
of
clue,
and,
ultimately,
your
entity
are
finally
in
a
single
pane
of
glass
that
collect
all
of
your
burning
data
and
help
you
navigate
both
or
install
dependencies
between
your
services,
but
also
vertical
dependencies
through
all
of
the
layers
of
your
application,
containers
or
conception,
platforms,
infrastructure.
Again,.
A
C
So
it's
better
illustrate
the
difference
between
his
base
sampling
until
they
sampling
here
the
results
of
an
experiment
that
we
did
in
which
we
sent
the
same
data
set
of
traces
through
a
random
and
a
simpler
on
the
left
and
so
and
intelligent.
Tell
me
something:
either
ism
on
the
right
was
the
same
target
betaine
boy.
The
first
thing
to
notice
at
the
top
that
n
base
sampling
was
virtually
unable
to
capture
any
of
the
traces
of
human
hairs,
some
surprising,
since
the
sampling
decision
is
remnant,
but
it
critically
than
helpful
traces
are
contained.
C
Valuable
in
troubleshooting
system
issues,
especially
one
of
their
interment
and
at
the
bottom
we
can
see
that
cell
based
sampling
is
it
was
like
traces
is
much
further
into
the
long
tail
of
response
times.
What
interesting
opinion
about
race
is
hot,
getting
started
with
district
racing
is
probably
the
biggest
challenge,
as
we've
seen.
B
So
I
touch
about
service
measures
a
little
bit.
This
is
an
area,
that's
getting
a
lot
of
interest.
Lately,
there's
been
a
lot
of
momentum
behind
them.
Obviously
we
have
you
know
Castillo
and
Envoy
and
whatnot
each
other
or
powering
some
of
this
out
there
in
the
wild
and
from
a
monitoring
and
observability
perspective.
They
actually
bring
a
lot
to
the
table.
So
you
know,
if
you're
an
organization
starting
off
with
you
know,
setting
up
new
instructure
writing
your
applications.
B
We
feel
that
you
should
consider
service
mesh,
especially
as
it
comes
to
observability
and
monitoring
the
first
thing
that
it
helps
us
with
which
it
makes
our
life
easier
in
some
sense,
which
is
kind
of
why
doing
the
same
this
bug
in
general,
because
the
service
mesh
is
deployed
in
throughout
your
micro
services,
and
it
is
looking
at
interaction
between
every
single
micro
service
to
another.
It
is
out
there
and
it
can
go
and
monitor
the
detraction
foreign
without
you
having
to
do
a
lot
of
stuff.
B
For
example,
it
can
know
that
in
any
time,
a
service
a
cause,
the
service
be
well.
The
service
mesh
itself
can
then
create
metrics
and
telemetry
from
that.
So
you
get
what
we
call
a
breadth
of
coverage
which
just
comes
for
free
as
a
result
of
this
point,
a
mesh
like
the
other
thing
that
it
brings
is
that
standardization
we
talked
about
in
the
past,
of
having
a
standard
methodology
for
collecting
these
well.
The
mesh
can
do
that.
B
So
what
it
could
do
is,
for
example,
for
every
you
know:
API
call
going
between
services,
it
can
capture
any
metrics
and
it
can
report
them
into
your
monitoring
system,
which
is
also
great
because
now
I
do
not
only
have
read
tough
coverage.
I
also
have
a
minimum
bar
on
the
kind
of
telemetry
that
I
am
collecting,
so,
for
example,
to
make
that
more
apparent.
B
Are
able
to
be
an
aggregator
for
much
of
this
telemetry
and
then
forward
them
through
your
different
metrics
and
observability
systems
for
him?
So
we
feel
that
you
know
if
you
want
to
have
high
quality
and
widespread
collection
of
observability
data.
Having
a
service
mission
place
definitely
helps
make
that
job
easier
and.
D
B
Now
it
could
be
as
simple
as
I
want
the
horse
to
be
up,
and
if
the
horse
is
down,
I'm
gonna
go
and
restart
it
or
it
could
be
something
far
more
higher
level,
but
having
a
solid
observability
system
in
place
with
some
standards
with
some
breadth
of
coverage
really
enables
us
to
go
to
these
higher
levels
and
that's
kind
of
when
we
as
organizations
are
going
to
become
far
more
efficient
and
fast-moving
and
just
better
each
other.
So
we're
gonna
have
now
yeah
who's
gonna.
B
D
So
here,
what
we're
doing
with
signal
effects
that
we've
deployed
our
signal
effects
smart
asian
across
about
1,500
hosts
in
our
infrastructure
here,
and
we're
now
able
to
see
individual
metrics
for
each
of
these.
In
this
case,
we're
looking
at
the
CPU
metrics
for
every
single
in
these
hosts.
We
could
also
be
looking
at
memory,
disk
or
network
and
additionally
down
here,
we're
actually
aggregating
this
and
giving
you
a
percentile
distributions
and
totals
for
the
memory
used
disk
network
memory.
D
When
looking
at
something
like
this
in
aggregate,
it's
not
always
helpful
to
see
kind
of
everything
all
grouped
together.
So
one
things
we
can
do
is
we
automatically
pull
down
a
lot
of
the
metadata
from
AWS
and
other
orchestration
services
that
we're
using,
and
we
can
use
this
either
filter
down
the
data
further
or
to
group
it
up
into
ways
that
make
sense
to
us.
D
So,
for
example,
we
can
take
this
AWS
availability
zone
tag,
use
as
a
grouping
tool
and
apply
that,
and
now
we
can
see
how
our
hosts
are
distributed
among
different
availability
zones,
where
we
are
deployed
in
this
case,
primarily
the
US
East.
We
can
also
pull
custom
tags
and
labels
from
things
like
kubernetes
or
AWS,
and
in
that
way
we
can
actually
look
at
something
say,
like
a
service
tag
that
we
use
to
actually
understand
the
individual
micro
services
within
single
FX's
infrastructure.
D
D
So
as
you
spin
it
more
hosts
or
as
hosts
go
away,
those
along
might
be
included
or
excluded
from
this
alert.
The
recipients
for
this
alert
can
be
meet
some
email,
some
team
or
one
of
our
service
integrations
such
as
HipChat
ager
duties,
ServiceNow
slack,
big
drops
or
everything
about
automation,
something
like
a
web
hook
right
as
Arjun
mentioned
before
so
in
assuming
you
did
receive
an
alert.
The
question
would
be
what
you
might
want
to
do.
D
Next,
in
the
event
that
we
achieve
an
alert,
we
can
actually
dig
into
the
specific
node
that
triggered
that
alert
and
get
even
more
detail,
we'll
be
able
to
see
these
system
metrics.
For
that
particular
note,
but
also
we
understand
the
properties
of
that
node,
as
collected
from
AWS
from
the
agent
itself
and
any
other
properties
that
we
added
after
the
fact,
combining
that
with
the
processes
running
on
that
note
as
well.
D
This
helps
an
engineer
understand
what
the
problem
might
be
without
having
actually
SSH
into
the
box
to
actually
access
the
box
in
any
way.
Furthermore,
our
agent,
because
it
has
all
this
information
available
to
it-
it's
able
to
automatically
instrument
itself
to
collect
metrics
for
the
services
running
on
the
node,
for
example.
In
this
case,
we
see
that
this
notices
docker
host,
so
we
can
generate
a
docker
dashboard
based
on
that
information
and
similarly
it's
an
elastic
search
node
as
well.
So
you
can
see
the
elastic
circular,
dashboard
specific
to
this
particular
node.
D
To
give
you
a
view
more
like
this,
where
now,
what
we
can
see
is
the
different
services
in
architecture
and
how
they
relate
to
one
another
we
can
see
which
Stream,
which
services
are
upstream
or
downstream,
and
we
can
actually
see
the
request
rates,
error
rates
and
durations
of
the
request
to
each
of
these
services.
The
red
metrics
is
Arjun
mentioned
before.
D
On
top
of
that,
we
can
actually
dig
into
a
specific
service
as
well
and
say,
see
the
different
endpoints
within
that
service
and
for
additional
detail.
We
can
go
to
an
out-of-the-box
service,
dashboard
bat-signal
effects
generates
and
see
those
metrics
overlaid
with
infrastructure
metrics
here
in
this
service
dashboard.
For
my
API
service,
I
can
see
the
request
rate
latency,
just
air
raid,
distributed
over
time,
so
I
can
see
as
I'm
trending
towards
some
problem
or
things
getting
better
and
as
I
scroll
down
further
I'm,
actually
able
to
combine
that
with
my
host
metrics
as
well.
D
So
if
I
am
seeing
some
sort
of
problem,
I
can
try
and
understand
whether
or
not
that
relates
to
you,
the
host
CPU
host
memory
usage,
maybe
I'm
doing
some
sort
of
garbage
collection
here
that
would
be
represented
in
those
metrics
by
having
this
as
kind
of
a
primary
dashboard
for
understanding
your
services.
This
helps
you
identify
quickly,
whether
where
the
problem
might
be
in
addition
to
these,
in
addition
to
these
standards,
dashboards
that
we
provide,
you
can
also
have
custom
dashboards
as
well.
D
This
catalog
service
here
handles
a
lot
of
requests
for
this
party
commerce
application
and
handles
those
through
these
three
different
end
points,
and
so
for
us
to
get
a
better
view
on
what
this
service
does.
We've
created
this
custom
dashboard
here
with
high
resolution
metrics
that
helps
us
better
understand.
D
What's
going
on
here,
what
we're
looking
at
as
the
number
of
requests
per
container,
that's
been
cataloged
by
the
being
processed
by
the
catalog,
we're
also
looking
at
the
request
latency
that
each
customer
is
experiencing
and,
as
you
can
see,
these
metrics
are
coming
in
at
a
one-second
resolution.
So
every
second
we're
seeing
an
update,
what's
happening
here.
D
As
an
engineer,
this
is
valuable
because
if
what
I
want
to
do
is
to
go
ahead
and
make
some
sort
of
Canaria
deployment,
I
can
see
the
effects
that
deployment
immediately
so
hopping
over
here
to
my
console,
I
have
a
deploy,
script
ready
to
go
and
what
it
does
is.
This
goes
ahead
and
talks
to
Jenkins
and
tells
thinkin's
deploy
the
next
ready
version
of
my
canary
to
this
particular
environment,
clicking
that
here,
what
we'll
see
happen
here
in
just
a
second.
Is
that
we'll
see
a
custom
event
from
that
deploy
script
from
Jenkins?
D
That
lets
me
know
that
I
have
deployed
a
new
canary
and
then
at
once,
that's
done
walked
we're
actually
seeing
now.
The
new
container
getting
spun
up
is
new.
This
new
canary
and
the
old
container
being
spun
down.
At
the
same
time,
we
can
see
that
this
new
container
seems
to
have
kind
of
high
latency,
which
seems
to
be
a
problem
and,
as
noted,
it's
it's
happening
at
the
same
time
as
we
push
that
canary.
D
So
what
we're
seeing
is
that
this
outlier
is
beyond
what
we
would
expect
and
when
that
alert
triggers,
we
automatically
initiate
an
automated
rollback.
We
signal
effects
alert,
uses
its
Web
book
to
talk
to
Jenkins
and
to
say
rollback,
this
canary
and
put
backup
standard
production
notes.
We
do
not
affect
the
user
experience
too
much,
as
you
can
see
here.
After
this
event,
we've
spun
down
that
canary
and
now
we're
looking
at
a
standard
production
deployment.
D
Again
is
this
kind
of
automation
that
our
Joe
is
mentioning
before
the
helps
engineers
have
more
impact
in
the
organization
and
to
do
more
with
less
once
you've
resolved
was
kind
of
alert.
The
next
thing
you
might
want
to
do
is
to
understand
you
know
what
caused
that
problem.
Is
there
something
I
can
do
to
prevent
this
in
the
future?
One
things
we
allow
you
to
do
when.
D
Enable
is
with
our
data
links
tooling.
We
can
actually
take
a
look
at
the
container.
Id
here
understand,
that's
canary
and
to
issue
a
query
into
some
to
some
logging
platform
using
that
information
in
that
time
frame.
This
is
a
good
way
of
looking
back
understand.
You
know
what
were
the
logs
at
point
in
time.
What
does
that
tell
me?
D
If
you
see
this
latency
distribution
chart,
which
is
telling
us
which
is
showing
us
all
the
trace
on
the
trace
distributions
from
one
point-
eight
milliseconds
down
here
or
two
long
tail
of
up
to
ten
seconds
on
the
on
the
other
end,
we
can
also
see
the
the
rate
of
requests
that
received
in
this
period
of
time,
as
well
as
the
errors
in
order
to
help
sort
through
all
this.
One
things
we
might
do
is
to,
rather
than
we'll
get
every
neutral
one
of
these
traces.
D
We
can
actually
filter
these
down
further
based
on
the
information
that
we
already
have
from
before.
In
this
case,
what
we
might
do
is
to
go
ahead
and
use
our
deployment
filter
and
look
specifically
for
the
Canary
deployment
traces.
This
helps
us
narrow
understand,
especially
what
traces
were
going
through
that
canary
deployment
and
to
get
further
information.
We
can
actually
click
this
analyze
outliers
button.
This
uses
our
statistical
analysis
and
shows
us
specifically
which
operations
and
which
services
are
contributing
most
of
the
duration
of
traces
in
this
time
period
clicking.
D
That's
exhibiting
the
behavior
that
we're
trying
to
solve
so
clicking
this
one
here
now
we're
looking
at
the
the
full
map
of
a
given
trace
with
all
the
stands
that
are
within
it
zooming
in
here
a
little
bit
we
can
see
which
services
were
looking
at
and
which,
which
endpoints
within
those
services
that
were
hitting
and
which
operations
are
running-
and
we
can
see
here
that
there's
kind
of
some
long
long
spans
here,
but
we
don't
necessarily
understand-
is-
is
this
normal
or
is
this
some
error?
That's
happening!
D
One
things
we
do
the
single
effect
is.
We
can
actually
take
a
look
at
each
of
these
fans
and
color
them
by
how
they're
performing
in
this
particular
instance
against
the
historical
duration.
This
will
immediate
ilight
to
us
immunity
to
highlight
if
we're
seeing
deviations
from
standard
behavior
within
this
trace-
and
we
include
identify
that
indeed,
this
long-running
trace
here
is
running
much
longer
than
we
ought
to
thought
to
be
getting
additional
information
here.
D
B
Those
are
the
kind
of
qualities
that,
as
you
are,
building
your
observability
infrastructure
that
you
should
keep
in
mind
about,
because
these
are
the
things
that
will
make
them
more
and
more
effective.
We
also
shall
have
showed
how
water
mission
is
important
and
how
that
can
help
us
like
be
more
efficient
by
allowing
the
computers
to
take
care
of
many
of
our
issues.
We
also
showed
you,
you
know.
If
you
look
at
go
back
to
the
maturation
curve
that
we
showed
in
the
beginning.
B
We
also
showed
the
importance
of
having
data
science
come
to
bear
so
that
we
don't
have
to
manually,
set
various
static
thresholds
and
such
but
half
our
system,
learn
and
and
automatically
identify
alerts
and
our
plan.
So
hopefully
that
sort
of
gives
you
a
broad
sense
of
what
sort
of
we
talked
about
anything
in
the
in
the
beginning
of
the
talk
track
section
and
with
that
we
gonna
we
have
one
or
two
more
slides
right.
So
let's
go
back
to
index
equations.
Yes,.
B
So
there
really
is
not
much
in
here,
but
in
general
you
know
we
wanted
to
show
you
lots
of
pretty
boxes
at
the
end,
but
in
general
I
think
I
mentioned
I
know
much
of
these
qualities,
so
having
observability
is
not
easy,
especially
good.
Quality
of
the
visit
is
not
easy.
You
must
be
prepared
to
spend
time
and
people
and
money
or
whatever
the
case
may
be,
to
make
sure
that
you
have
a
good
system
in
place.
B
I
hope
we
were
able
to
make
the
case
that
it
is
there's
many
many
areas
the
way
it
can
help
you
it's
not
just
talking
or
taking
a
log,
and
we
we
need
on
a
UI
or
getting
a
metric
and
seeing
a
chart
for
it.
It's
far
more
than
that,
you
must
have
a
coherent
strategy
around
it
and
your
observability
system
must
have
certain
qualities
which
make
it
which
are
even
more
critical
in
cloud
native
environments,
especially
you
know,
with
native
things,
are
in
the
spinning
up
and
down
inelastic,
so
auto
discovery
is
important.
B
High
resolution
is
important
so
that
you're
able
to
find
problems
quickly.
Speed
of
data
processing
is
important
so
that
you
can,
when
a
problem
happens,
are
able
to
slice
and
dice
relatively
fast,
because
otherwise
that
might
also
slow
you
there.
So
you
down,
we
talked
about
data
science
and
analytics.
We
talked
about
dashboards,
precise,
alerting
is
key,
because
if
you
alert
on
too
many
low
level
things
you're
going
to
be
drowned
in
a
lot
of
noise
so
being
able
to
have
high
quality,
you
know,
SLI
driven
alerts
is
important.
B
We
talked
about
and
we
showed
how
end-to-end
visibility
is
important
and
the
last
thing
is,
you
know,
as
we
sort
of
move
to
more
and
more
cloud
native
and
DevOps.
What
we
find
is
you
know
infrastructure
and
deployments
driven
by
code
rather
than
by
humans,
so
your
observability
system
must
be
able
to
deal
with
that.
It
must
have
API
that
you
are
able
to
do
use
to
programmatically,
deploy
not
in
real
infrastructure,
but
also
be
associated
observability
and
monitoring.
A
Thank
you
very
much
folks.
There
was
absolutely
awesome
and
you
know
the
demos
were
really
demonstrating
what
I
wanted
to
say.
So.
Thank
you
very
much
for
that.
We
do
fortunately
have
some
time
for
questions
like
almost
ten
minutes
and
there
are
quite
a
bit.
So
what
start
was
the
simple
one?
Brian
hideous
here
asks:
will
this
work
in
a
Windows
environment.
B
So
that's
actually
interesting
because
our
environments
tend
to
be
quite
very
different
clouds
different
operating
systems.
So
the
typical
thing
tends
to
be
that
you
would
have
to
have
an
agent
that
is
working
on
a
Windows
environment
is
able
to
fetch
metrics
from
them.
For
example,
the
signal
effects
will
under
the
hood
the
windows
it
worked.
C
C
D
C
A
B
B
D
A
B
So
the
answer
is,
it
depends
so
for
the
most
part,
you
would
only
have
to
instrument
your
code
for
your
own
first
party
application.
So,
for
example,
if
you
use
a
you
know
a
host
agent
like
a
telegraph
or
collecti,
and
so
on
or
in
the
signification
which
is
based
on
collecting,
they
are
able
to
fetch
all
your
at
least
your
node
resource
metrics
infrastructure.
Metrics,
like
you,
know,
CPU
memory,
all
of
that
by
themselves.
B
All
of
these
have
very
good
support
for
third
party
OSS
services,
like
all
of
the
open
source
stuff
that
you
have
like.
You
know:
Kafka
elasticsearch,
my
sequel
and
whatnot,
so
they
are
able
to
monitor
them
as
black
boxes,
and
you
can
even
then
have
them
get
exported
to
commit
es
or
your
monitoring
system
that
signal
effects
to
where
they
won't
do.
That
is
your
own
code,
because
there
is
no
way
for
signal
effects
to
sort
of
unless
you're
running
at
an
ATM
agent.
B
In
the
tradition
ATM
agent,
which
is
sort
of
like
running
within
your
process,
you
wouldn't
be
able
to
get
the
metadata
or
the
matrix
from
within
your
application,
but
that
you
have
client
libraries
to
do
if
you
are
in
deployed
in
a
service
mesh
environment.
Maybe
you
won't
even
have
to
do
that
because
at
least
the
service
to
service
interaction
might
get
automatically
monitored
on
your
behalf.
So
in
the
short
answer
is
for
the
most
part
for
everything
other
than
your
own
code.
B
B
D
B
Ethics
does
run
on
a
tip,
so
we
are
actually
a
SAS
service,
so
we
have
effectively
abstracted
away
all
the
stuff
that
happens
after
the
the
metrics.
If
you're
you
know
systems
and
come
to
it,
so
you
know
we
think
of
us
as
a
Prometheus
plus
all
these
other
stuff,
that
we
talked
about
a
service.
So
we
handle
the
storage
for
you.
B
We
handle
the
scaling
for
you,
and
all
of
that
we
kind
of
the
API
is
alerting
all
of
that
for
you
and
then
the
main
thing
that
needs
to
happen
is
the
matrix
and
traces
etcetera
need
to
come
from
your
environment
to
us,
for
which
we
have.
You
know
smart
agents
and
client,
libraries
and
what-have-you
yep.
C
Running
there,
on
top
of
the
story,
special
or
finish
container
is
directly
you.
You
know
the
choice
of
either
using
an
application
agent
that
can
do
a
lot
of
automatic
instrumentation
for
you
or
to
manually
instruments
you're
using
a
lot
of
open
source.
Libraries
like
Jager
that
we
talked
about
that
is
an
open,
physically
tracer.
B
It
will
work
everywhere
so
in
some
sense
it's
a
matrix
and
this
video
tracing
platform,
and
it
does
not
have
an
opinion
on
where
those
metrics
and
the
suti
traces
are
coming
from
as
well
as
the
metadata
are
coming
from.
What
we
have
done
is
we
have
built
integrations
to
various
clouds
and
orchestrators,
because
the
ones
that
are
in
currency,
that
you
know
various
teams
and
team
organization
using,
but
theoretically,
you
could
have
sent
us.
You
know
whether
data
or
stock
prices,
and
we
would
monitor
than
justified.
B
A
Right
and
I
guess
last
question
that
we
have
time
to
take
a
life
will
be
that
you
mentioned
that
larger
deployments
and
organizations
may
may
move
to
commercial
solutions
for
sophisticated
do-it-yourself
systems
to
handle
their
monitoring
and
what
are
some
of
the
reasons
behind
it.
A
wire
tools
like
Prometheus,
Griffin
and
Zipkin
are
not
enough.
So.
C
B
What
is
fine,
it
starts
becoming
interesting,
is
at
an
organization
scales
or
as
your
infrastructure
scaled
up,
various
other.
What
you
might
think
of
as
peripheral
concern
start
becoming
more
and
more
important
like
so,
for
example,
you
may
want
to
be
in
multiple
different
clouds
right.
You
may
have
multiple
different
regions
that
you
are
deploying
so
then,
how
do
you
build
a
strategy?
B
Let's
say
using
primitives,
where
you
are
able
to
have
them
all
together
and
still
able
to
look
at
all
these
data
politically,
the
things
like
that
becomes
harder,
long
term
like
so,
for
example,
if
you're
doing
capacity
planning
and
such
having
this
long
term
trends.
Where
you
know
we
have
people
who
want
to
know
what
happened
in
your
back.
They
want
retention
as
far
back
as
a
year
and
so
on.
This
is
where,
because
of
the
primacy
of
storage
format,
it
starts
becoming
a
challenge.
B
Unless
you
have
many
many
many
instances
and
then
managing
them
becomes
a
problem.
Another
area
we
find
has
to
do
with
just
how
people
we
use
a
monitoring
system.
For
example,
if
you
have
50
teams-
and
one
of
them
makes
a
mistake
and
starts
sending
a
whole
lot
of
data,
then
we
bring
the
whole
infrastructure
down
or
a
big
part
of
it
down
for
everybody
else,
so
being
able
to
have
motors.
D
D
B
B
So
as
you
sort
of
think
about
companies
trading
up,
we
find
that
many
many
other
of
these
concerns
become
more
and
more
important
and
then
they
will
either
require,
let's
say
a
vendor
solution
or
will
require
a
pretty
sizeable
team
with
this
organization
to
build
these.
For
so
it
I
think
it's
easy
to
get
started,
but
then,
as
you
mature,
as
your
requirements
become
more
higher
level,
then
you
need
to
spend
some
serious
resources
or
money
on
it.
Yeah.
A
That
makes
sense,
I
think
it's
the
usual
build
versus
buy
dilemma
that
there
aren't
the
right
answer,
changes
on
the
scale.
Unfortunately,
that's
all
the
time
that
we
have.
There
are
more
questions
in
the
Q&A
tab,
so
I
hope
guys
you
will
be.
You
will
have
time
to
answer
them
in
writing
and
sand
together
with
the
follow-up
email,
and
thank
you.
Everybody
for
joining
us
today
is
the
webinar
recording
in
the
slides
and
the
answers
for
their
unanswered.
Questions
will
be
online
later
today,
and
we
are
looking
forward
to
seeing
you
at
the
future.