►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Understanding Service Mesh Metric Merging- Lawrence Gadban, Solo.io
Observability is a key tenet of any service mesh so each mesh implementation provides several features around observing the state of your mesh. One important aspect of observability is the exporting of metrics which enable operators or service owners to monitor both application and service mesh behavior. However, with service meshes that utilize a sidecar proxy, the exporting of metrics from both the application and the proxy itself is not as straightforward as it seems. In this session we will explore the challenge of consistently and transparently exporting metrics for all components comprising a service mesh workload. We will then dive into how Istio solves this problem with its “metrics merging” feature. This feature provides a simple way of exporting an aggregated set of metrics from all of the pieces in a sidecar-injected workload, including metrics from the Istio agent, sidecar Envoy proxy, and application workload itself. Attendees will leave with a clear picture of why metrics merging is important in general as well as how it works in the Istio service mesh.
A
Hello,
everybody
thank
you
for
attending
service,
meshcon,
north
america,
and
thank
you
for
attending
this
session,
where
we
will
be
talking
about
metrics
merging
in
service
mesh.
My
name
is
lawrence
godban
and
I'm
a
field
engineer
at
solo.io,
where
we
have
several
products
related
to
application,
networking
in
a
cloud-native
environment.
A
So,
let's
jump
into
it.
So
monitoring
and
metrics
in
general
are
very
important,
especially
when
you're
talking
about
distributed
systems,
because
you'll
have
applications
running
in
several
different
places,
spinning
up
and
down
et
cetera,
and
you
need
a
way
of
understanding
the
behavior
of
those
applications
and
those
deployments.
So
a
common
tool
to
do
this
is
prometheus.
You
know
prometheus
is
a
cncf
graduated
project.
A
It
defines
a
very
commonly
used
metrics
format
and
also
provides
a
way
of
scraping
those
metrics
and
giving
you
a
way
of
querying
that
data
so
that
you
can
do
things
like
monitor
and
understand
the
behavior
of
your
systems.
So,
looking
at
this
diagram
here,
we
have
a
pretty
you
know,
common
or
typical
deployment.
Where
you
have.
You
know
a
few
name
spaces
running
the
kubernetes
cluster.
You
have
different
workloads
running
in
those
name
spaces,
and
you
need
a
way
of
monitoring
them.
A
So
what
that
means
is
we'll
have
you
know
we
need
to
configure
prometheus
to
have
a
way
of
knowing
where
to
actually
go.
First
of
all,
how
to
find
these
endpoints
that
we
need
are
these,
these
workloads,
that
we
need
to
scrape
and
then
also
how
to
actually
scrape.
So,
how
do
I
find
the
metrics
endpoint
that
I
need
to
send
a
scrape
request
to
right?
A
So
that's
done
using
spray
scraped
configuration
and
service
discovery
and
prometheus,
so
straight
configs
will
tell
prometheus
how
to
actually
do
the
the
scraping
you
know,
gather
those
metrics
and
then
service
discovery
tells
prometheus
about
the
plate.
The
targets
that
it
does
need
to
scrape
so
the
way
that
a
very
common
way
to
do
this.
You
know
this
is
not
like
this
a
standard
but
very
close
to
it.
A
That
say
you
annotate
a
pod.
These
annotations
will
give
me
this
additional
information.
So,
looking
at
these
annotations
here
in
this
example,
we
have
a
path,
a
prometheus
path,
annotation
set
to
slash
metrics.
We
have
a
port
set
to
8080
and
scrape
to
true.
So
what
we're
doing
is
we're
telling
prometheus
that
hey?
A
I
want
you
to
scrape
this
pod,
the
the
port
that
you
need
to
talk
to
is
8080
and
the
path
that
you
can
send
the
request
to
in
order
to
get
those
metrics
is
slash,
metrics
right
and
so
then,
combined
with
a
given
pod
ip.
You
know
a
request.
A
script
request
might
look
something
like
this:
where
does
it
get
to
the
ip
and
the
correct
port
and
slash
metrics
right?
And
so
then
the
end
result
is
that
this
request
will
return
result
in
a
response
that
has
the
metrics
for
this
given
workflow.
A
So
when
we're
talking
about
a
service
mesh
just
quickly,
looking
at
the
architecture
right,
you
have
workloads,
and
traditionally
you
know
that
means
you
have
a
data
plane.
So
your
data
plane
are
the
proxies
that
are
usually
deployed
alongside
as
like
a
side
car
to
these
workloads,
and
you
have
a
control
plane
that
actually
configures
those
proxies
based
on
the
state
of
the
environment.
You
know
user,
config
or
so
on,
right
and
so
that,
that's
all
great,
you
know
that
makes
sense
from
a
service
perspective
from
a
monitoring
and
metrics
perspective.
A
What
that
means
is
that
now
prometheus
has
to
know
about
more
components,
to
scrape
right,
because
now
we
have
to
scrape
from
the
application,
as
well
as
the
proxies
and
the
control
plane,
control
plane's,
not
too
difficult.
Usually
it's
just
a
standard
deployment,
but
it
gets
a
little
tricky
when
you're
talking
about
having
a
two
components
in
a
single
pod,
for
example
the
the
proxy
and
the
workload.
A
So
looking
at
istio
in
particular,
it
gets
even
more
complex,
because
now
we
have
an
additional
component
called
the
agent
or
the
istio
agent.
So
this
this
component
does
you
know
several
auxiliary
tasks
related
to
the
workload,
but
it
it
is
a
component
and
you
do
need
to
scrape
its
metrics
usually
and
that
it
makes
it
challenging
right,
because
now
you
have
three
different
places
in
a
given
pod
that
you
need
to
scrape.
A
So
if
we
look
at
the
previous
example
we're
talking
about
the
annotations,
let's
see
what
that
would
look
like
in
istio.
So
you
know
our
pod
has
the
correct
annotations.
So
prometheus
is
going
to
try
to
scrape
our
application
metrics,
which
are
at
port
8080,
slash
metrics.
So
then
the
the
request
will
go
through
the
proxy,
like
in
a
you,
know,
typical
service
mesh
environment
and
get
routed
to
the
the
application
workload,
which
will
then
return
the
metrics
for
the
application
workload.
A
But
that
is
a
problem
right
because
then
we're
not
getting
the
metrics
for
the
proxy
and
we're
not
getting
metrics
for
the
agent
so
to
solve.
This
istio
has
a
concept
called
metrics
merging
which
provides
a
solution
to
this
by
exposing
a
single
endpoint,
and
so
remember
that
agent
is
a
standalone
component,
and
so
it
can
perform
logic.
A
It
can
perform
tasks,
and
in
this
case
the
agent
is
what's
responsible
for
exposing
an
endpoint
that
prometheus
can
scrape,
and
then
the
agent
you
know
when
it
handles
that
request
will
then
provide
a
merged
document
of
all
three
of
the
metrics
from
the
proxy,
the
application
workload
and
the
agent
itself.
So
in
order
to
accomplish
this,
let's
look
at
how
how
istio
solves
that.
So,
when
you
inject
a
workload
you're
in
istio,
typically,
that's
done
using
a
web
hook
a
mutating
web
hook,
and
so
in
this
example.
A
What
we'll
look
at
is
you
know,
we're
passing
in
a
pod
is
being
created
and
admitted
to
the
kubernetes
api.
So
then
it
will
go
to
them
through
the
web
hook
for
injection.
So
in
that
case,
what
we're
going
to
do
is
you
know
when
a
pod
has
the
prometheus
annotations,
those
annotations
will
be
changed
or
overwritten
to
have
the
correct
annotations
that
point
to
the
istio
agent's
endpoint.
A
So
then
what
it
looks
like
is:
okay,
we
have
a
pod
that
has
the
correct
annotations
to
point
to
the
agent's
endpoint
at
port,
15,
15020
and
the
path
is
slash,
that's
prometheus,
so
then
prometheus
will
generate
a
script
request
to
the
pod's
ip
to
the
the
correct
port
and
the
correct
path.
So
then,
the
request
will
make
it
to
the
agent's
end
point
then
it's
up
to
the
agent
to
since
we
stored
those
original
the
original
annotations.
A
It
knows
how
to
get
the
application
metrics
so
we'll
make
a
request
to
that
correct
location
and
we'll
get
back
some
application.
Metrics
then
we'll
also
scrape
the
metrics
from
the
proxy,
since
the
agent
is
also
responsible
for
launching
the
proxy.
It
knows
where
the
proxies
age,
the
proxy's
metrics
end
point
is
already
so
it
will
go
ahead
and
scrape
that
from
its
well-known
endpoint
and
then
it
will
merge
those
two
and
then
additionally
add
its
own
agent
metrics.
A
So
now
what
happens
is
we
have
a
full
document
that
contains
the
metrics
from
all
three
of
the
components
in
the
in
a
given
pod
and
now
that
can
be
returned
back
to
prometheus
and
we
have
a
single
endpoint
from
a
single
endpoint.
We
have
all
of
the
metrics.
We
need
for
a
given
pod,
so
that
works
great
in
most
cases,
it's
very
transparent
to
the
workload
as
long
as
you're
using
those
annotations
we'll
talk
about
that
here
in
a
second.
A
But
another
thing
to
consider
is
that,
since
the
agent
itself
is
exposing
this
endpoint,
it's
excluded
from
the
proxy,
and
that
means
that
you
don't
get
things
like
mutual
tls.
For
free,
so
that
in
most
cases
I
would
say
is
probably
not
a
problem,
but
if
you
do
have
super
strict
security
requirements
that
require
even
metrics
data
to
be
encrypted,
you
will
not
be
able
to
use
the
solution
so
that
you'll
have
to
use
an
alternative.
A
There
are
some
outlined
on
the
istio.io
website
that
talked
about
alternatives
for
handling
the
the
tls
portion
and,
like
I
just
mentioned
earlier
this
this
feature,
or
this
solution
requires
prometheus
annotations
to
work.
So,
if
you're,
using
a
tool
that
doesn't
respect
those
prometheus
annotations,
this
will
not
work
either.
A
very
common
example
of
that
is
prometheus
operator,
which
has
its
own
kind
of
mechanism,
kind
of
crd-based
mechanism
for
discovering,
which
crd
and
custom
resources
mechanism
that
that
you
know
tells
prometheus
where
to
go
to
scrape.
A
So
in
general,
though,
there
are
other
alternatives
to
solve
this,
so
you
could
configure
prometheus
directly
to
to
understand
where
the
data
planes
live
right,
so,
whether
that's
through
like
kind
of
convention
or
well-known
labeling
or
so
on.
It's
very
you
know
you
could
configure
prometheus
directly
to
to
understand
the
the
data
planes
and
you
don't
have
to
worry
about
kind
of
merging
them
together.
A
Another
interesting
thing
kuma,
another
cncf
project-
has
they
have
a
native
prometheus
service
discovery
plug-in,
which
means
that
you
know
it
tells
it
gives
prometheus
the
ability
to
to
really
understand
natively
what
kuma's
you
know.
Deployment
looks
like
so
you
have
data
planes
and
so
on.
The
proxies
prometheus
will
understand
that
it
does
require
a
change
to
prometheus
deployment,
which
kind
of
leads
into
the
next
more
sophisticated
approach,
which
would
be
a
very
dynamic
native
integration
that
would
use
a
service
to
configure
it.
A
You
know,
rather
than
having
to
modify
the
prometheus
deployment,
you
could
configure
it
through
a
service
and,
if
you're
familiar
with
envoy,
you
know
what
the
xds
apis
there
may
be
a
play
there
to
allow
the
control
plane
to
configure
prometheus
since
the
control
plane
knows
about
your
data
planes.
It
knows
about
your,
you
know
the
workloads
and
so
on.
That
means
the
control
plane.
Can
dynamically
configure
prometheus
with
a
native
integration
as
well,
but
yeah?
That's
pretty
much.
A
It
just
a
quick
plug
if
you
are
attending
in
person
or
virtually
come
check
us
out
at
the
solo.I
o
team.
We
have
an
in-person
booth
at
booth
s4
in
the
solutions
showcase.
We
also
have
a
virtual
booth.
Come
talk
to
us,
come
check
us
out,
and
you
can
also
enter
for
a
chance
to
win
some
airpods,
and
that
is
all
I
have.
Thank
you
very
much.