►
From YouTube: Webinar: How to Gain Insights from Istio by Leveraging Tools like Prometheus, Jaeger and Cortex
Description
Istio integrates with a rich set of tools that can add tracing, telemetry, logging and other functionalities to your microservices environment. This session will focus on how a myriad of tools, including several CNCF projects, work collectively to deliver the full functionality of Istio.
This webinar will cover:
- How Grafana and Prometheus make makes it easy for your entire team to understand what’s going on with your microservices.
- How Jaeger can be most effectively used for tracing to get to root cause analysis.
- How Cortex delivers long-term storage for Prometheus telemetry data, how horizontal scalability and cloud-native storage provides virtually infinite data retention, and how Cortex can solve multi-tenant Prometheus challenges.
A
Alright,
let's
go
ahead
and
kick
things
off
and
get
started.
Welcome
to
today,
CN
CF
webinar,
the
gain
NC
a--
insights
from
sto
by
leveraging
tools
like
Jaeger,
Prometheus
and
cortex
very
excited
to
host
this
today,
as
sto
is
in
service
matches,
are
very
much
the
rage
in
the
cloud
native
space.
At
the
moment
my
name
is
ariel
chatib,
I'm
a
founder
at
a
company
called
stack
point
cloud
and
a
CF
ambassador
I'd
like
to
thank
you
all
for
joining
us
and
extend
a
nice.
Warm
welcome
to
me.
Raj
board
are
from
Aspen
match.
A
Who
is
going
to
be
our
presenter
today?
We
appreciate
him
sharing
his
expertise
on
Aspen
mesh,
with
all
of
you
today,
a
little
bit
of
housecleaning
or
housekeeping.
If
you
will,
there
is
a
chat
window
at
the
bottom,
and
a
QA
function
go
ahead
and
use
those
as
we
go
through
the
talk
and
we'll
try
to
find
opportunities
to
interject.
With
your
questions,
this
session
is
also
being
recorded
and
we'll
send
out
the
link
later
on.
So
without
further
ado,
I'll
pass
it
over
to
me.
Raj
thanks.
B
Ariel
good
morning,
everyone
I
am
neeraj
and
the
platform
lead
at
Aspen
mesh.
Thank
you
all
for
joining
in
on
the
CNC
webinar
today,
I'm
going
to
talk
about
how
to
gain
insights
into
your
micro
services,
applications
using
sto
and
leveraging
various
tools
like
Jaeger,
Prometheus
and
cortex,
but
before
we
go
deeper
into
the
insights,
I
will
first
give
an
overview
of
what
the
service
mesh
is.
What
problems
service
solves?
B
What
are
the
various
architectural
choices
and
then
move
on
to
sto
architecture
and
then
to
the
insights
and
using
tools
to
unlock
the
full
functionality
of
sto.
So
with
that,
let's
get
started.
So
this
is
a
typical
microservices
installation
or
a
typical
microservices
lecture.
You
have
three
services
service,
a
B
and
C
service
is
talking
to
service,
be
service,
be
stopping
to
service,
C
and
then
service
C
is
talking
over
Internet
to
maybe
a
man
at
service.
All
these
services
or
micro
services
are
written
in
different
languages.
So
when
is
it
rent
Python?
B
The
second
is
written
about
GIS
and
heard
in
Java.
So
in
this
slide,
what
you
see
in
the
bottom
half
the
green
boxes.
They
represent
the
essential
business
logic
and
API
is
that
are
unique
to
each
micro
service,
so
the
microscope
is
developer.
Have
each
of
them
have
to
develop
this
for
providing
the
functionality
of
the
micro
services
the
boxes
in
the
top
in
black?
They
represent
somewhat
a
common
functionality,
which
is
kind
of
shared
between
each
micro
services
between
the
micro
services
and
they
are
related
to
communication
between
them.
B
So
they
involve
things
like
service
discovery,
load,
balancing
some
sort
of
tracing
telemetry
if
you
are
using
mutual
TLS
or
TLS
and
open
SSL
library.
So,
even
though
this
these
functionalities
for
communication
between
micro
services
are
shared,
they
are
currently
part
of
each
micro
service,
which
means
application.
Developer
is
responsible
for
that.
Secondly,
if
you
are
operating,
imagine
you
your
job
is
to
ensure
uniformity
between
these
micro
services.
So
you
can
query
traces
and
query
metrics
across
the
micro
services.
B
So
imagine
if
one
micro
service
is
a
different
format
of
request
count
than
the
other
or
if
you
want
to
maintain
a
uniform
security
posture,
ensuring
open
SSL
version
compatibility
across
micro
services,
again
a
big
operational
nightmare.
So
this
is
the
problem
with
service
mesh
solves
so
service
mesh
is
essentially
a
common
infrastructure
layer
which
manages
communication
between
micro
services.
B
So
with
a
micro
service,
the
operator
is
happy
or
it
makes
his
life
easier
as
they're
getting
a
lair
in
which
they
can
insert
uniformity
and
their
control
and
policies
at
the
same
time,
for
the
developer,
it's
easier
as
they
have
to
reason
about
just
the
code,
which
is
unique
to
the
micro
services.
They
don't
have
to
think
about
the
service
mesh
layer
or
they
don't
have
to
think
about
communication
between
micro
services,
so
typically
service
mesh
provides
these
advanced
communication
functionalities.
By
inserting
a
proxy.
B
This
proxy
is
going
to
intercept
traffic
going
in
and
out
of
your
applications
for
service
mesh
depending
upon
where
and
how
you
insert
your
proxy.
There
are
various
architectural
choices
and
trade-offs
that
come
with
them.
So
let's
look
at
some
of
the
architectural
choices
you
have
on
where
you
can
place
your
proxy.
B
So
first
up
is
what
we
call
upward
node
proxy
model.
This
was
pioneered
by
linker
D,
which
is
a
service
smash
from
buoyant.
So
in
this
model
there
is
a
proxy
inserted
on
the
node
on
which
your
applications
are
running.
There's
only
one
proxy,
but
node
the
proxy
intercepts
the
traffic
going
in
and
out
of
your
applications.
It
will
provide
all
the
advanced
communication
functionalities
like
load
balancing
circuit
braking
retry.
B
This
model
works
great
for
most
of
the
cases,
especially
if
you
have
some
efficient,
especially
you
will
have
some
efficient
resource
sharing
for
things
like
caching,
but
it
breaks
down
when
you
want
to
think
about
one
application
and
a
proxy
as
one
logical
entity
for
security
purposes.
In
this
model,
the
proxy
cannot
assume
the
identity
of
the
workload
and
for
security
purposes.
Imagine
if
one
application
in
your
node
is
compromised.
It
might
compromise
the
proxy
which
might
compromise
all
the
applications
running
in
your
log.
B
B
Sidecar
proxy
model,
so
in
this
model
the
idea
is,
the
proxy
is
inserted
as
close
to
the
applications
as
possible,
but
still
as
two
separate
containers.
So
what
that
means
in
kubernetes
lab
world
is
the
sidecar
proxy
is
inserted
as
a
container
in
the
same
pod
in
which
your
applications
are
running
as
they
are
in
the
same
pod.
B
They
shared
the
same
networking
namespace,
so
you
can
reason
about
the
application
container
in
the
sidecar
proxy
container
as
one
logical
entity
from
your
security
and
network
perspective,
the
proxy
can
assume
the
identity
of
your
workload,
the
proxy
and
the
application
container.
They
talk
over
loopback,
which
is
isolated
from
the
rest
of
the
world.
From
an
operational
point
of
view,
the
proxy
lives
and
dies
with
your
application
container.
So
there
is
some
simplicity
there.
B
This
is
the
architecture
which
Istria
chose,
and
this
is
the
architecture
diagram
of
Sto.
So
in
Sto
the
sidecar
is
on
void,
which
is
shown
here
in
pink
as
a
separate
container
in
the
pod
in
which
your
application
is
deployed
on
Y
is
a
hyper
performant
open
source
proxy,
initially
developed
at
left.
So
all
the
envoys
are
intercepting
traffic
from
your
application
container
and
they
are
communicating
with
the
AI
other
macro
services,
so
the
on
words
together
represent
what
we
called
the
HT
of
data
plane.
B
The
box
in
the
top
which
you
see
is
the
history
of
control.
Plane,
still
has
a
nice
separation
between
the
control
and
data
plane,
which
gives
you
benefits
of
independent
scaling
for
data
in
controlled
by
an
independent
upgrading
of
components
going
for
the
the
sto
control
plane
itself
is
distributed,
so
you
can
upgrade
and
scale
individual
components
independent
of
each
other.
So
let
me
go
over
what
the
control
plane
elements
are
of
the
components
our
industry
as
we'll
be
using
them
later,
as
we
go
through
the
functionalities
of
Sto.
So
first
step
is
sto.
B
Pilot
history
pilot
is
the
piece
which
looks
at
your
platform
config
and
translates
it
to
the
configuration
which
the
side,
cars
or
envoys
understand.
So
here,
using
kubernetes
pilot
will
look
at
your
services
on
you,
know,
endpoints.
It
will
translate
and
lower
the
config
down
to
envoy.
Similarly
pilot
will
consume
all
your
history
or
specific
configuration
and
lower
it
down
to
the
configuration
which
envoy
understands
operationally.
B
This
has
an
advantage
where
you
are
talking
and
configuring
history
o
in
your
platform
native
language
rather
than
you
have
dealing
with
the
low-level
configuration
of
envoy
second
controller
in
component
is
mixa.
Mixa
provides
three
functionalities
in
Sto.
First,
it
provides
policy
checks.
What
that
means
is
whenever
a
request
reaches
on
were
before
envoy
forwards
it
on
to
the
sidecar
container.
It
concerns
the
mixer
whether
the
request
should
be
allowed
or
if
the
request
is
allowed
via
policy.
Mixer.
B
This
is
very
similar
to
policy,
but
it
is
more
around
resource
enforcement,
like
global
rate,
limiting
for
requests
per
second
or
through
pups.
Third
mixer
is
a
telemetry
sync,
so
all
the
stats
which
are
currently
gathered
and
sent
from
on
voice,
they
are
reported
to
mixer
and
then
mixer
integrates
with
a
backhand
stab
system
for
storing
these
stats.
This
is
a
in
great
from
operational
perspective
as
as,
as
an
operator,
you
have
to
just
integrate
your
mixer
with
the
right
back-end
system
for
stats,
and
you
don't
have
to
worry
about
the
low-level
data.
B
Plane,
stats,
myth,
stats
format.
Moving
on
the
third
piece
is
sto
Citadel.
It
used
to
be
called
sto
earth
or
sto
CA
in
previous
incarnations.
This
piece
is
responsible
for
creating
TLS
certainties
when
you
have
mutual
TLS
enabled
for
data
path
or
control
path.
So
this
P
this
component
also
does
cert
rotational
from
an
operational
perspective.
It
is
great
because
this
is
the
only
piece
which
can
which
needs
to
integrate
with
your
authority.
If
you,
if
you
want
to
have
some
isolation
there,
the
fourth
piece
is
sidecar
injector.
B
This
is
a
very
specific
control
plane
element
only
applicable
in
kubernetes
world.
This
looks
at
the
new
pods
which
are
being
created
and
automatically
injects
on
voice
before
the
pods
startup.
This
works
by
registering
the
sidecar
injector
as
a
admission,
if
I
hook
and
add
that
enables
it
to
a
change
the
pod
configuration
before
it
starts.
This
is
again
very
beneficial
from
operations.
Point
of
view.
B
As
you
can
avoid
mistakes,
where
you
don't
have
sidecars
injected
in
various
spots
and
your
security
posture
is
now
we
can
so
this
is
a
quick
overview
of
sto
architecture,
so
moving
on
sto,
in
essence,
is
a
service
mesh
which
provides
a
bundling
of
this
advanced
capabilities
that
I
just
discussed,
which
includes
service
discovery,
load,
balancing
at
once,
traffic
management
policy
and
security
for
handling
communication
between
micro
services.
Today,
in
this
talk,
I
am
going
to
focus
more
on
two
of
these
capabilities
tracing
and
matrix.
B
Right
great,
so,
let's
move
on
to
tracing
so
tracing
has
often
been
described
as
one
of
the
most
needed
and
missed
functionality
in
the
micro
services,
world,
I.
Think
of
pressing
as
analogous
to
module
level
logging
in
the
monolith
world.
So
creasing
is
the
feature
which
gives
you
nice
service
dependency
analysis
for
the
different
micro
services.
B
Typically
tracing
works
by
adding
trace,
IDs
and
unique
span
and
parent
IDs.
These
IDs
are
propagated
between
each
micro
services
via
HTTP
headers.
So
let's
take
an
example
where
micro
service
a
when
receives
a
request.
It
talks
to
micro,
service,
B
and
then
B
talks
to
C,
so
microservice
a
will
create
a
route
trace,
ID
and
send
them
as
a
header
to
Metro
service
B
micro-service
B.
B
When
is
ready
to
send
that
request
over
to
C
will
transfer
these
headers
and
it
will
create
child
spans
from
those
routes
pans,
and
then
all
these
spans
are
reported
into
a
back-end
system
where
you
can
later
go
and
correlate
the
relationship
between
different
services.
This
is
great
for
analysis
like
I
was
saying,
and
it
kinds
of
it
kind
of
gives
you
a
distributed.
Logging
mechanism
into
the
micro
services
per.
B
So
with
Sto
we
often
say
you
get
tracing
out
of
the
box,
but
it's
not
well
explained
off
and
what
you
are
exactly
getting
in
Sto
as
the
side
cars
are
sitting
in
in
pods
next
to
your
application
containers
and
they
are
intercepting
all
the
traffic
going
in
and
out,
they
can
automatically
trace
all
network
requests
for
you.
What
that
means
is
when
our
request
reaches
the
side
car
and
if
the
side
car.
B
So
when
the
request,
which
is
the
side
car
and
if
the
request
does
not
have
any
tracing
headers,
for
example,
it
will
create
the
tracing
headers
before
it
sends
those
requests
to
the
application
container
and
then,
when
the
application
container
is
ready
to
make
an
outbound
requests
to
another
micro
service.
It
looks
at
the
headers
again
and
if
the
tracing
headers
are
present,
it
will
automatically
create
child
spans
for
those
places.
So,
as
you
can
imagine,
with
Sto,
you
don't
really
have
to
you.
Don't
really
have
to
have
your
applications.
B
A
lot
of
work
for
creating
of
these
fans
all
the
context
handling
for
creating
child
and
parent
in
spans
works
transparently.
Additionally,
the
spans
that
the
sidecars
create
they
are
asynchronously
reported
to
a
back-end
systems.
So
again
the
applications
don't
have
to
do
reporting
of
the
spans
themselves.
So,
let's
look
at
what
are
the
things
that
envoy
adds
in
these
spans,
so
typically
the
spans
have
start
and
end
times.
So
with
this,
you
can
understand
how
long
each
request
took
it
has
like
things
like
source
and
destination
service
names,
source
and
destination,
IP
z--.
B
It
has
a
host
of
HTTP
headers,
like
parts
URLs
user
agent
requests
and
response
sizes
error
response
codes.
So,
as
all
this
information
is
reported
back
into
a
back-end
system,
you
get
all
the
visibility
you
need
for
understanding
what's
happening
in
your
micro
services.
A
key
thing
to
understand
here
is,
as
sto
is
doing
this
or
specifically
the
side.
Cars
are
doing
this
from
a
network
point
of
view.
B
There
is
no
way
you
can
escape
this,
so
if,
for
example,
if
your
application
was
doing
tracing
itself
and
not
sto,
there
might
be
cases
in
which
the
developer
forgets
to
add
creasing
in
this
land,
you
will
always
get
tracing
requests
for
any
or
you
will
always
get
traces
and
spans
for
any
network
requests
which
happens.
One
thing
which
is
very
interesting
to
note
here
is.
B
Applications
in
this
land
or
in
sto
have
to
do
one
particular
thing
to
get
full
benefit
from
sto,
which
is
they
have
to
propagate
the
application
headers,
which
are
present
in
the
incoming
requests
and
transfer
them
to
the
outgoing
requests.
This
is
really
important
and
often
less
clear.
So
if
the
applications
don't
transfer
the
tracing
headers
from
incoming
to
outgoing
requests,
sidecars
won't
be
able
to
create
child
spans,
but
all
the
spans
which
will
be
created
will
be
routes
pants.
B
What
that
means
is
you
will
only
have
tracing
for
two
level
of
services
and
never
have
a
tracing
across
multi-level
services
like
10
or
20
services.
So
I'm
I've
put
two
links
here
to
the
blocks.
I
wrote
one
first,
one
is:
how
do
you
do
connect
header
propagation
in
your
applications,
if
you
are
contacting
AWS
services
and
if
you
are
using
them,
it
gets
really
interesting.
So
this
block
might
help
you
alleviate
some
of
the
pain
which
I
had
to
go
through.
Second,
is:
how
do
you
do?
B
Heading
are
tracing
correctly
with
G,
RPC
and
sto,
and
specifically
points
and
helps
you
how
to
do
header
propagation,
when
you're,
making
outbound
calls
from
GL
PC
to
non
gr,
PC
or
gr
PC
applications.
So
moving
on
just
like
any
open
source
project,
sto
integrates
with
various
backends
for
sending
and
querying
of
these
fans
correct
little.
It
supports
two
open
source
projects,
zip
Kaelin
Hager.
Often
it's
very
confusing
to
understand
and
evaluate
which
integration
works
well
or
which
back-end
system
is
better.
We
recommend
using
Hagar.
B
Hagar
is
a
distributed
tracing
system
for
collecting
and
querying
traces.
It
was
developed,
it's
an
open
source
project.
It
was
initially
developed
by
uber
out
of
the
Google
dapper
paper.
It
has
an
open
tracing
compatible
API.
So
for
most
of
the
sto
users,
you
don't
care
that
it
is
an
open
tracing
compatible
API,
but
it
is
really
useful
when
you
add
advanced
tracing
and
advanced
span
written
as
spans
in
your
application.
So
what
we
have
done
is
who
would
the
or
the
months
that
we
had
started
using
sto?
B
B
Tracing
from
sto
directly,
but
then
we
add
extra
spans,
like
user
information,
or
we
create
more
advanced
spans
like
around
our
logs.
How
long
the
logs
are
kept.
So
things
like
that,
when
you
add
it's
good
to
use
a
library
which
is
open,
dressin
compatible,
so
in
future
you
can
replace
it
with.
You
can
replace
your
back-end
without
changing
your
applications,
because
creasing
is
one
field
where
you
will
have
to
update
or
change
your
applications.
I
recommend
doing
it
the
right
way
and
use
some
open
tracing
compatible.
B
B
It
provides
support
for
storing
the
traces
in
multiple
backends
currently
provides
support
for
Cassandra
and
elasticsearch.
So
typically,
if
you
have
many
clusters
that
you
want
to
be
that
that
are
sending
traces,
you
can
have
Yaga
collector
and
Jakob
queries
which
are
single
tenant,
but
Cassandra
and
elasticsearch
both
support
multi-tenancy,
so
all
of
the
traces
can
go
to
the
same
back-end
system.
Lastik
search
is
a
very
interesting
choice
here,
especially
if
you
are
sending
your
logs
to
elasticsearch.
B
Fourth
thing,
which
is
interesting:
what
Yaeger
is
there,
is
work
going
on
in
the
open-source
community
for
adding
advanced
sampling.
This
means
you
won't
be
sending
all
the
spans
or
storing
them,
but
B
being
intelligent
about
what
samples
will
be
sent.
This
helps
reduce
the
storage
costs
and
speed
up
your
querying,
as
you
have
less
things
to
look
at
another
thing
interesting
about
Hagar
is
it
can
actually
accept
spans
and
Zipkin
format
so
currently
open
tracing
API
provides
sanitization
around
the
programming.
B
Api
is
which
people
should
use,
but
the
actual
headers
which
go
on
the
wire
are
not
standardized.
So
history
uses
Zipkin
format,
headers,
which
are
xb3
headers
Jager
collector.
Can
those
headers
and
store
them
so
with
Jager
you
actually
don't
lose
the
second
compatibility,
if
you
care
for
that,
and
with
is
tio.
Actually
you
will
have
to
configure
here
Jager
collector,
to
accept
zip
conformance
last
two
things.
It
is
a
stain
safe
project.
B
So,
let's
take
a
look
at
what
the
hey-ey
looks
like
the
top
part
of
the
slide
represents
a
represents
the
overview
of
what
you
get
when
you
do
any
queries
in
your
rig
or
UI,
so
the
x-axis
here
is
the
time
the
y-axis
represents
the
latency
of
chryses
and
the
size
of
the
circles.
You
present
how
many
spans
are
in
each
trace,
so
I
have
personally
used
this
mu
to
understand
distributions
of
mine
microservices.
When
this
you
can
kind
of
see,
this
is
a
bimodal
distribution.
B
B
I
have
also
used
this
to
understand
things
like
we
had
a
system
where
requests
were
all
starting
at
separate
times,
but
they
used
to
end
at
the
same
time
because
they
used
there
was
a
periodic
garbage
collection
or
some
periodic
trigger,
which
is
to
do
cleanups
so
very
useful
to
understand
a
global
perspective
of
your
micro
services.
The
second
half
of
this
slide
represents
the
detailed
view.
So
in
this
you
can
click
on
individual
traces
and
find
out
how
long
the
request
state
you
can
understand
your
service
dependency.
B
So
here
is
your
ingress
talks
to
product
page
product
page
talks
to
details.
You
can
zoom
in
on
each
of
those
pans
and
then
get
the
HTTP
attributes
and
the
other
span
attributes
are
talked
about
like
source
and
destination
services,
SCP
you
stdp
parts
again
great
for
visualizing
and
understanding
each
request.
We
have
found
some
interesting
cases
here
where
we
were
able
to
figure
out
that
we
had
loops
in
our
micro-services
by
looking
at
this
view.
So
this
is
what
I
wanted
to
say
about
creasing
next
I'll
be
moving
on
to
matrix
and
I.
A
A
B
Think
all
Zipkin
and
yoga
are,
as
you
said,
they're
solving
the
same
problem.
I
find
that
so
personally,
when
you
are
evaluating
zip
converse
as
if
you're
being
a
CNC
of
project
was
a
big
deal
for
us,
v1
sure
we're
set
keen
was
heading
I.
Think
I
am
personally
not
an
expert
on
Zipkin,
so
I,
don't
know
what
more
you
get
out
of
Zipkin
currently
or
last,
when
I
evaluated.
Actually
Giga
was
solving
most
of
the
problems
that
you
wanted
out
of
Sitka.
A
B
A
A
B
A
B
Neither
actually
so
Jaeger
has
bench
of
different
components,
so
he
got
has
a
Jaeger
collector
which
collects
prices,
then
a
Hager
query
for
wearing
creases
and
then
Cassandra
for
storage
of
those
places,
if
you
are
just
using
Jaeger
without
any
other
component.
So
though,
so
this
is
a
Hager
eight
agent
also
which
can
sit
on
every
node
as
a
daemon
set,
but
with
sto.
You
actually
don't
need
it,
because
all
the
traces
from
the
site
cast
can
go
directly
to
Jager
collector.
A
Okay,
let
me
ask
another
one:
do
you
have
some
best
advice
for
a
newbie
on
how
to
debug
when
they
have
a
problem
with
this
Co
ur
go
gateway,
virtual
service,
sidecar
injection
policy,
for
example?
Where
should
I
look
at
multiple
logs
disable,
some
part
of
a
component
exam?
Those
are
some
examples.
I
say.
B
For
always,
I
would
recommend
it's
normally
good
to
look
at
the
logs
and
see
if
you
are
getting
discovery
failures
for
when
you
are
configuring,
things
like
what
you'll
serve
a
virtual
services
or
gateways
I
would
recommend
always
use
sto
CTL
rather
than
cube
CTL,
because
sto
CTL
does
CRD
validation,
which
cube
CTL
does
not,
and
then
sometimes
if
the
CR
DS
are
accepted,
and
you
should
also
look
at
sto
pilot
logs,
which
might
give
you
good
view
into
a
fewer
route.
Route
rules
are
actually
applied.
B
Last
thing,
I
would
say,
is
Anwar
has
a
debug
endpoint
or
an
admin
at
endpoint,
so
for
every
side,
car
or
for
every
pot
in
which
you
have
the
side
car.
You
can
go
to
port
15,000
and
look
at
your
slash
clusters,
slash
routes
and
understand
if
the
routes
are
actually
taking
into
effect,
that
the
last
thing
is
the
key
thing
to
gain
insights
into
how
sto
works
and
also
debug
sto
itself.
A
B
A
B
B
A
B
A
B
Right
thanks
Ariel,
so,
let's
resume
so
next
thing.
I'll
talk
about
is
matrix
now,
all
right
so
matrix
and
time
series
data
is
a
great
way
of
knowing
what's
happening
in
your
application.
I
thing
that
hasn't
changed
from
monolith
to
micro-services.
The
problem
remains,
and
the
solutions
remain
it's
just.
We
have
some
optimized
x
rays,
databases
now
which
we
can
use.
It's
a
great
way
to
understand,
historically,
what's
happening
in
your
application
over
time
to
compare
when
your
applications
were
healthy
to
the
time.
B
Your
applications
are
broken,
it's
great
for
debugging
and
doing
fault
analysis,
and
of
course
you
can
do
some
monitoring
and
alerting
set
up
some
uptime,
not
only
just
for
your
individual
micro
services,
but
across
your
application,
which
is
made
up
of
many
micro
services
and
on
those
events,
you
can
do
things
like
scale
them
out.
Add
more
resources
in
the
list
grows
on
so
with
sto
again,
just
like
tracing
you
get
a
lot
of
matrix
and
a
lot
of
stats.
B
This
is
again
a
there's,
not
a
documentation
which
says
that
HT,
u
provides
lot
of
stats,
but
there's
not
clear
documentation
and
what
stats
are
provided
and
how
they
are
provided.
So
again,
let
me
break
this
down
for
you
in
this
T
of
because
it
is
sitting
in
your
network
path
and
in
the
data
path.
It
has
good
view
into
what's
happening
in
your
cluster
and,
what's
who
is
talking
to
what
in
your
micro
services
architecture,
so
it
can
create
a
lot
of
stats
automatically
for
you,
so
there
are
two
levels
or
there.
B
Two
types
of
stats
which
are
currently
exported
from
Sto
first
is
envoy.
Stats
envoy
has
a
unique
view
into
what's
happening
in
your
pod.
It
can
give
you
some
information
specific
to
the
pod,
which
no
button
no
other
entity
can.
So
there
are
lot
of
advanced
things
which,
on
work,
include
reports
like
request
pending
how
many
bytes
are
buffered
in
your
connection,
are
you
seeing
some
SSL
failures
again?
These
are
from
the
view
of
the
pod,
so
I
have
a
link
there
for
onward
stats
over
you
take
a
look.
It
might
help.
B
You
characterize
the
performance
of
a
pod
in
the
fair
using
Sto.
So
if
you
are
familiar
with
the
you
model
for
monitoring,
use
stands
for
utilization
saturation
and
errors.
These
stats
map
very
well
to
that
model.
So
you
can
see
you
can
have
alert
saying
if
you
have
too
many
requests
pending.
Probably
your
latencies
are
going
to
go
high,
so
I
have
we
have
used
them
personally
for
characterizing
our
applications,
and
sometimes
some
of
these
stats
are
very
difficult
to
obtain
from
applications
themselves
but
easier
to
do
it
from
a
network
point
of
view.
B
So
moving
on
the
second
level
of
stats,
which
are
reported,
our
mixer
level
starts
so
mixer
takes
a
higher
view
from
pods.
It
reports
starts
from
service
perspective,
so
mixer
will
tell
you
which
service
is
talking
to
what
and
what
are
the
latency
is.
Specifically,
you
can
draw
a
histogram
of
latencies
between
individual
micro-services.
You
can
easily
evaluate
P,
99
and
p95
latencies.
You
can
send
a
mixer
sense
and
reports
request
counts
between
micro
services
at
the
service
level.
B
It
also
sends
requests
by
HTTP
status
codes
paths,
so
you
can
filter
on
a
lot
of
things
if
so,
this
model,
or
these
type
of
record
metrics
I,
think
map
really
well
to
the
read
model
of
monitoring.
So
red
stands
for
rates,
error
and
durations
or
distributions,
and
those
are
supposed
to
be
from
a
service
point
of
view
to
see
if
your
up
times
are
being
met.
Third
thing,
which
is
important
with
this
to
you,
that
it's
very
easy
to
add,
custom
stats
and
these
customizations
are
currently
only
available
for
mixer
level
steps.
B
So
you
can
add
your
own
stats
that
you
want
to
be
that
you
want
to
monitor
and
collect
from
the
service
perspective,
and
this
is
a
configuration,
change
and
mixer
will
start
reporting
them
forth
just
like
tracing.
They
are
lot
of
options
for
sending
and
collecting
stats
and
their
various
integrations
with
lists
you,
the
stats,
D,
the
stag
driver,
which
was
mentioned,
which
is
an
attractive
solution
for
both
tracing
and
telemetry,
and
then
there
is
Prometheus
so
I
just
like
tracing
these
options
are
sometimes
confusing
and
it's
difficult
to
evaluate
the
benefits
across
them.
B
So
I
recommend
using
Prometheus
Prometheus
is
a
highly
optimized
time
series
database
for
collecting
stats
and
for
querying
them.
It
has
an
interesting
model
which
is
different
from
various
other
architectures,
so
it
has
a
pull
based
model.
What
this
means
is
the
Prometheus
server
pulls
or
scrapes
stats
from
applications
as
specific
endpoints
like
slash
metrics,
compared
to
some
of
the
other
models
like
stats
D,
which
is
the
push
model
where
you
have
agents
which
are
going
to
export
these
metrics
to
the
start's
D
back-end
server.
B
From
operational
point
of
view,
there
are
various
trade
offs,
but
a
pull
based
model
is
still
easier,
as
each
application
is
responsible
for
exposing
its
own
stats.
It
has
a
very
flexible
query:
API
Prometheus
API
is
more
or
less
becoming
the
standard
API
for
now
queering
stats
for
applying
functions.
It
has
a
large
repository
of
functions
and
transformations.
You
can
do
at
the
server
side
before
the
stats.
Are
you
put
it
back
to
you?
B
There
are
various
integrations
for
dashboards
for
creating
charts
and
visualizing
them,
so
I
would
actually
recommend
using
a
phoner,
which
is
another
open
source
tool
for
dashboarding
and
creating
charts.
So
normally
people
use
Prometheus
for
tell
acting
all
the
stats,
and
then
they
have
Gras
fauna,
which
integrates
very
easily
if
it's
Prometheus
and
use
Griffin
or
for
dashboarding,
charting
and
setting
alerts.
So
the
graph
on
the
right
is
a
refinement
chart
which
is
currently
available
from
the
sto
community.
This
gives
you
service
level
stats,
which
I
was
just
talking
about.
B
Mixer
exports,
another
great
benefit
of
Prometheus.
It
has
a
lot
of
exporters,
so
it
has
various
integrations
into
Kasana
databases
like
Cassandra
or
console
which
already
export
Prometheus
level
stats.
It
also
has
a
superb
aha
for
monitoring
kubernetes
itself.
There's
native
integration
through
Prometheus
starts.
So
take
a
look
at
those
links.
It
might
help.
You
see
if
you
have
some
other
systems
which
you
have
not
written,
that
the
stats
and
the
correct
format
of
those
stats
might
be
available
for
you
here.
These
integrations
last
again,
just
like
Hegel
it's
en
safe
project.
B
It
has
a
strong
open-source
community.
I.
Think
prompt
con
is
about
to
start
in
the
next
week
or
so.
There's
a
lot
of
structure
around
releases
and
maintenance
and
being
a
cien,
safe
project
helps
maintain
that
another
thing
which
I
want
to
mention
going
forward
here
or
about
prometheus
is
Prometheus
is
great.
If
you
want
to
monitor
one
or
two
or
three
clusters,
there,
few
restrictions
or
downsides
to
Prometheus
or
it's
an
architectural
choice
that
Prometheus
made
I
should
say
so
Prometheus
by
default
is
single
tenant.
B
Second
Prometheus,
the
open
source
Prometheus
by
default
will
only
support
on
disk
local
on
disk
storage
of
samples.
That
means,
if
you
have
three
clusters
and
three
Prometheus
instances,
you
will
have
to
manage
three
separate
stateful
systems,
three
separate
disks
and
again
handle
upgrades
and
whatnot
for
all
of
them
individually.
So
this
is
where
the
next
project
jumped
on
we'll
talk
about,
helps
which
is
cortex.
Cortex
is
a
open-source
project
which
provides
multi-tenancy
and
long-term
remote
storage
for
Prometheus.
This
was
built
at
V
works
and
is
currently
open
sourced.
B
So
the
first
benefit
you
get
from
cortex
is:
you
can
have
long-term
storage
to
manage
services
instead
of
on
disk,
so
with
cortex
you
can
use
AWS
DynamoDB,
you
can
use
BigTable,
which
is
a
Google
managed
highly
optimized
database
or
you
can
have
Cassandra
if
you
manage
it
to
yourself
for
some
reason.
Second
thing
which
is
great
about
the
cortex
is
it
provides
a
scalable
Prometheus
query
API,
another
problem
which
is
often
seen
with
Prometheus
as
your
as
the
number
of
samples
grow
or
beyond
a
certain
size.
B
It's
pretty
scalable
up
to
a
point,
but
after
that
the
queries
start
to
slow
down
in
the
kotas
cortex
architecture.
The
storage
and
query
paths
are
separate
and
the
database
is
also
is
handled
separately.
So
you
can
scale
the
query
path
of
cortex
independently,
based
on
your
needs.
It
provides
multi-tenancy,
so
you
can
have
all
the
various
clusters
that
can
report
stats
into
the
same
cortex
multi-tenant
system.
So,
typically,
what
you
do
in
this
case
is
you
run
Prometheus
any
all
of
your
clusters,
but
the
Prometheus
is
are
configured
as
stateless
services.
B
Prometheus
supports
remote
storage
API,
so
you
can
figure
Prometheus
to
talk
all
the
Prometheus
is
to
talk
to
one
cortex
and
then
that
cortex
handles
the
right,
banking
and
storage
and
multi-tenancy.
It
stores
them
into
the
databases
I
just
mentioned.
Similarly,
although
on
the
query
side,
it
provides
the
right
Prometheus
API
is
of
Prometheus
compatible
API
is
to
take
the
samples
out
from
for
the
right
tenant,
as
you
can
imagine,
it's
much
easier
to
manage
because
of
the
architecture
I
just
mentioned
compared
to
managing
20
stateful
services
with
disks.
B
Lastly,
is
work
going
on
I
think
they
are
proposing
very
soon
that
permit
that
codex
should
be
added
as
a
CN
CF
project.
All
the
benefits
that
Jaeger
and
Prometheus
and
kubernetes
got
from
being
a
CN
safe
project.
Codex
can
also
receive
with
that.
The
last
thing
I
want
to
say
is:
if
you
ever
get
more
from
Sto.
These
are
the
following
recommendations.
First,
he
was
0-8
was
released
like
I.
B
Think
two,
two
and
a
half
weeks
ago
there
was
lot
of
work
done
in
the
open
source
community
for
stability
of
the
core
and
four
control
plan
scalability,
which
had
been
an
issue
for
the
last
few
releases
on
yv2
api's
are
now
used
on
were
V
Pugh
API
is
helped
a
lot
for
scalability
because
they
use
the
new
G
RPC
brisk
subscription
mechanism
for
receiving
route
rules
and
updates,
rather
than
the
query
mechanism
they
had,
and
it
I
think
they
have
migrated
to
the
new
eight
years,
which
is
the
aggregated
query.
Services.
B
Third
thing:
the
new
stov
one
alpha
three
networking
api's
are
now
available
in
zero.
Eight
from
the
configuration
and
manageability
perspective,
it's
much
easier
to
understand
and
reason
about
Sto,
which
will
service
in
the
various
route
routes
you
have
compared
to
the
policy
is
available
in
compared
to
compared
to
the
route
rules,
configuration
system
and
in
p1
alpha
three
last
we
are
active
users
of
sto
and
raspberry
mesh.
We
will
be
more
than
happy
to
talk
in
greater
depth.
B
Please
feel
free
to
reach
out
to
me
at
needed
at
a
spur,
measured,
IO
or
take
a
look
at
our
blogs.
It
might
help
you
say
if
some
clicks
and
save
some
time,
if
you
are
experiencing
the
same
thing
that
we
experienced
or
you
are,
you
have
similar
problems
so
with
that
I
think
I'm
done
for
the
slides
and
we
can
move
back
to
Q&A.
A
Are
you
so
much
Nina
Raj?
So
a
couple
questions
we
have
from
the
group
really
high
level.
Somebody
asked
about
probably
a
question
better
suited
to
Matt
Klein
from
lift
but
somebody's
curious
about
the
usage
of
C++
to
write
envoy
as
opposed
to
Golan
just
a
conduit,
and
if
you
have
any
insights
as
to
why
that
decision
was
made
so.
B
Secondly,
it
has
to
be
lighter
weight,
so
I
think
with
modern
C++.
You
can
achieve
high
performance
easily,
which
might
not
be
available
to
you
with
go,
especially
with
the
dynamic
memory,
allocation,
forego,
I.
Think
and
second
might
be,
the
comfort
level
they
had
with
the
languages
which,
again,
like
you
said,
is
a
more
appropriate
question
for
my
time.
A
B
Yes,
so
what
I
recommend
actually-
and
this
is
useful-
is
because
the
sidecar
will
automatically
insert
tracing
headers
for
you
so
use
the
open
tracing
API
is
to
dig
the
span
context
out,
add
all
the
spans
you
want
or
create
child
spans
from
it
before
you
make
further
downstream
or
upstream
requests.
So
this
is
very
helpful
and
very
useful.
Like
our
case,
we
were
adding
and
eggs
for
user
information
like
user
IDs,
which
sidecar
does
not
have.
A
B
Great
question
I
think
it
still
is
built
around
that
model
that
you
can
slowly
opt
into
the
service
mesh
without
all
your
services
being
in
the
mesh.
So
currently,
if
you
use
sidecar
injector,
you
can
enable
namespaces
on
which
the
services
will
be
or
in
which
the
site
cuts
will
be
injected,
even
if
you
have
enabled
them
in
your
namespace
for
specific
deployments,
you
can
annotate
the
pods
and
given
the
injection
of
sidecars.
B
Secondly,
if
you
are
actually
using
the
manual
process
for
injecting
sidecars,
you
can
always
opt
in,
and
it's
perfectly
fine
to
have
services
which
are
in
the
mesh
and
services
not
in
the
mesh
and
they
can
happily
coexist.
You
just
have
to
realize
that
the
benefits
that
you
get
from
mesh
you
won't
get
from
the
services
a
lot
l.
A
A
B
No,
that's
the
thing,
so
you
have
to
have
your
own
tracing
back-end
system,
which
you
want
to
use
so
envoys
in
sight.
If
you
have
the
right
configuration,
they
will
start
reporting
the
traces
you
have
to
have
the
service
like
the
Jaeger
service
or
Zipkin
service,
which
will
be
the
collection
point
for
your
traces.
And
then
you
will
have
to
have
a
back-end
database
for
storage
of
your
date
of
your
traces
like
Cassandra,
elasticsearch
or
stack
driver.
B
Great
question
I
mean:
definitely
it's
not
free.
You
are
adding
proxies
in
both
ingress
and
egress
paths.
In
your
communication,
I
think.
Currently
the
open
source
community
was
saying
there
is
a
30
to
40
percent
overhead
when
you
are
running
sto
or
onwards.
There's
a
lot
of
work
going
on
to
reduce
that
overhead,
but
I
think
in
reality
you
will
always
have
to
have
this
trade-off.
In
your
mind,
you
are
getting
a
lot
of
benefits
and
you
have
to
sacrifice
at
least
a
little
bit
of
the
performance.
B
Those
benefits,
in
my
mind,
outweigh
the
performance
penalties.
Now
the
performance
cannot
be
outrageously
bad,
but
the
benefits
are
so
much
and
as
a
developer
myself,
it
reduces
my
load
by
a
lot
as
the
end
for
an
operator,
it's
so
much
beneficial
that
you
might
be
able
to
live
with
some
of
the
penalties.
A
So
if
there
are
no
more
questions,
we're
gonna
go
ahead
and
wrap
this
up.
I
want
to.
Thank
me
Rajesh
for
sharing
all
this
great
information
with
us
today
and,
as
I
mentioned
at
the
beginning,
look
for
a
an
email
with
a
link
to
this
recording
and
we
hope
to
see
you
again
next
week
on
Tuesday
when
we
handle
the
next
CN
CF
webinar.
Thank
you
so
much
for
attending
and
until
next
week.