►
Description
SoloCon 2022:
Overcoming Observability Woes
Speakers:
Jahvon Dockery
Software Engineer, Solo.io
Shane O'Donnell
Team Lead - Gloo Mesh, Solo.io
Session Abstract:
The complexities of the Observability landscape is ever-growing and it becomes even more complicated as you scale your architecture. However, it continues to be an essential aspect of operating any environment reliably. In this session, we will discuss the unique characteristics of observability for various types of architecture (from monolithic to micro-services to service meshes) and how the Gloo Mesh UI provides a “single pane of glass” that can supercharge your abilities for even the most complex service mesh architectures.
Track:
Service Mesh and Application Networking
A
Hi
everyone,
thanks
for
coming
to
our
talk
on
overcoming
observability
woes
as
you
scale
your
architecture
in
this
talk,
we're
going
to
talk
about.
Why
scaling
your
service,
mesh
observability
is
so
hard
and
we're
going
to
talk
about
some
of
the
things
you
can
do
to
make
it
a
little
bit
easier,
we'll
get
started
by
introducing
ourselves.
So
my
name
is
shane
o'donnell.
I'm.
Currently
the
team
lead
on
the
blue
mesh
team
here
at
solo.io.
A
Finally,
we're
going
to
talk
about
how
important
it
is
to
have
a
single
pane
of
glass,
regardless
of
whether
you
have
a
single
cluster
or
a
hundred
clusters.
You
always
want
one
place
to
come
back
to
to
view
the
health
and
metrics
of
your
service
mesh.
So
I'm
going
to
pass
it
over
to
javon
to
talk
about
service,
mesh,
observability
benefits
and
challenges.
B
Thank
you
shane,
as
you
all
probably
know,
observability
as
a
whole
may
include
things
like
logging,
metrics
tracing
and
even
more
and
oftentimes.
Observability
goes
hand
in
hand
with
monitoring
for
this
specific
presentation,
we'll
be
focusing
more
on
the
monitoring
and
metrics
side
of
observability
and
first
I
want
to
introduce
you
all
to
a
method
of
monitoring
called
the
red
method.
Monitoring
with
this
method
of
monitoring.
It
defines
three
key
metrics
rate,
error
and
duration.
B
B
It
really
reduces
the
amount
of
server
service,
specific
knowledge
that
the
team
needs,
because
it
creates
a
common
language
for
the
monitoring
of
your
applications
that
you
can
use
throughout
your
organization
and
we'll
be
talking
and
focusing
more
on
this
specific
type
of
monitoring.
It
will
be
mentioned
throughout
this
presentation
for
this
presentation,
we're
also
assuming
that
you
made
the
jump
to
microservices
already
and
that
you
have
already
adopted
a
service
master.
You
are
in
the
process
of
adopting
service
meshes
with
service
mesh.
You
are
unlocked.
B
Another
good,
really
another
really
good
benefit
here-
is
that
there's
a
single
point
of
configuration
for
observability
when
using
the
surface
mesh
you'll
still
have
to
use
third-party
libraries
most
likely
like
prometheus
or
grafana,
to
get
the
full
observing
benefits.
But
the
nice
thing
about
using
the
service
message.
You
can
define
one
configuration
for
all
of
your
services
and
then
that
is
already
propagated
out
to
all
of
your
services
so
that
you
can
have
like
standardized
monitoring
metrics
in
place.
B
So
in
this
diagram
you
can
see
on
the
top
we
have
seod
for
and
that's
the
control
plane
for
the
seo
service
mesh.
Essentially,
what
you
will
do
is
you
apply
telemetry
configuration
within
your
cluster
and
then
seod
will
take
that
configuration
and
use
that
when
it's
injecting
your
sidecars
and
you
apply
that
to
all
of
your
services-
and
there
are
three
different
types
of
metrics
that
will
be
collected
essentially,
so
the
first
type
is
a
control
plane
metric.
So
those
metrics
are
things
that
all
deal
with.
B
How
is
cod
is
performing
so
that
can
include
the
number
of
known
services
and
the
number
of
failed
injections
there's
also
proxy
level
metrics
and
those
metrics
are
metrics
about
the
inbound
and
outbound
traffic
passing
through
the
proxies
and
then
the
an
example
of
that
would
be
the
number
of
requests
with
503
response
codes,
for
instance,
and
then
the
service
level
metrics
are
specifically
metrics
about
the
service
communication.
So
with
those
metrics,
you
can
see
the
number
of
requests
in
from
app
a
to
app
b.
B
B
So
you,
in
order
to
resolve
that
you
split
up
teams
into
different
clusters.
Good
example
here
is:
if
you're
at
a
health
care
organization,
may
want
to
limit
access
to
patient
data,
so
you
completely
put
that
in
one
cluster
and
then
give
other
teams
access
to
different
clusters
where
you
have
other
services
running.
B
B
You
also
will
probably
could
run
into
a
case
where
you
want
your
application
infrastructure
to
be
closer
to
your
users,
so
you'll
want
to
replicate
your
infrastructure
in
different
geographical
areas
so
that
your
users
have
better
performance
on
their
end,
then.
The
last
example
that
we
have
is
your
jurisdictional
compliance
and
regulations
may
require
specific
isolation
of
the
data
and
your
services,
so
those
data
and
services
may
have
to
reside
in
specific
place
and
may
have
to
be
separate
from
all
of
your
other
services.
B
In
terms
of
observability
and
the
challenges
that
multi-cluster
service
mesh
have
with
observability,
there
are
three
key
points
that
I
want
to
touch
on
here.
So
one
point
is
that
you'll
have
to
deploy
your
observability
systems
into
each
of
your
clusters.
So
that's
your
prometheus,
your
grafana.
Those
will
have
to
live
in
each
cluster
separately,
so
the
metrics
collection
and
correlation
is
scoped
to
that
single
cluster,
resulting
in
multiple
data
stores
for
that
data,
you're
also
losing
context
for
cost
cluster
traffic.
B
When
you
have
multiple
clusters,
so
your
data
needs
to
be
stitched
together
for
top
level
analysis
and
it's
much
harder
to
visualize
the
data,
especially
data
regarding
how
different
services
are
communicating
with
each
other
across
clusters,
and
this
cross-cluster
communication
also
just
creates
some
more
operational
overhead.
Providing
data.
Visualizations
is
very
difficult
at
the
scale
of
multi-clusters,
especially
when
you
have
hundreds
of
clusters.
B
A
A
A
Another
really
strong
point
about
this
glue.
Mesh
2.0
architecture
is
that
we
built
it
from
the
ground
up
to
support
strong
multi-tenancy
guarantees,
and
this
uses
a
concept
called
workspaces
which
we're
going
to
get
into
in
a
minute.
But
it
really
allows
you
to
scale
beyond
not
only
just
the
scale
in
terms
of
requests
per
second,
but
also
in
terms
of
how
many
developers
can
realistically
be
working
within
your
single
service
mesh
in
one
organization.
A
We
also
want
to
make
sure
that,
even
as
you
scale
across
multiple
clusters,
that
developers
are
able
to
have
a
single
point
of
configuration-
and
this
kind
of
comes
back
to
that
point
of
the
the
amount
of
clusters
that
your
application
is
running
on
should
be
at
least
somewhat
abstracted
away
from
you.
You
should
be
able
to
configure
where
everything
goes
in
a
single
place
and
then
just
have
your
your
platform
extracted
away
and
push
out
of
the
config
exactly
where
it
needs
to
go.
A
And
finally,
you
want
to
make
sure
you've
got
that
single
pane
of
glass,
so
you've
got
one
place
to
view
all
of
your
observability
metrics,
regardless
of
which
clusters
it's
coming
from
or
how
many
clusters
you
even
have
in
the
first
place,
so
to
look
at
how
we
accomplish
some
of
this,
I'm
going
to
take
a
step
back
and
look
at
the
glue
mesh
architecture
and
kind
of
how
some
of
this
data
flows
now.
I
know,
there's
a
lot
going
on
in
this
diagram,
so
I'm
going
to
break
it
down
here,
you
can
see.
A
We've
got
five
clusters
with
a
cross
section
of
our
main
two
cluster
types,
and
the
core
principle
here
is
that
we
install
an
blue
mesh
agent
on
each
of
what
we
call
the
workload
clusters,
which
are
clusters
that
are
serving
user
request
data,
and
then
we
also
designate
one
cluster
to
be
what
we
call
the
management
cluster.
The
management
cluster
runs
what
we
kind
of
consider
a
control
plane
of
control
planes.
A
So
you
can
imagine
istio
is
a
control
plane
for
envoy,
but
glue
mesh
is
a
control
plane
for
istio
and
what
we
call
this
control
plane
of
control
planes
is
the
management
plane
in
terms
of
how
did
all
this
configuration
works
together
and
how
that
flows?
So
at
the
start
of
our
flow,
you
can
see
the
agent
here
in
the
bottom
right
is
going
to
discover
all
of
the
things
that
are
in
its
local
clusters,
so
all
of
the
kubernetes
services,
all
of
the
existing
istio
resources.
A
Then
the
essentially
the
management
cluster
is
going
to
take
all
of
this
data
that
it's
receiving
from
all
of
the
clusters
in
your
global
service
mesh
and
it's
going
to
run
a
translation
loop
to
figure
out
based
on
all
of
this
data
and
based
on
all
of
the
inputs
that
the
users
have
have
created
using
glue
mesh
api
which
outputs
we
need
to
generate
for
every
single
output
workload.
Cluster
and
the
nice
thing
about.
This
is
there's
a
lot
of
nuance
depending
on
where
things
live
in
specific
clusters.
A
So,
if
you
can
imagine
considering
configuring
something
like
a
isteal
ingress
gateway,
if
that's
got
some
routes
to
go
to
the
local
cluster,
the
cluster
that
the
gateway's
living
in
and
some
routes
to
go
to
another
cluster
in
your
global
service
mesh,
the
the
way
that
routing
works
is
going
to
be
different
from
an
istio
configuration
perspective.
Some
of
them
will
be
able
to
just
use
that
local
cube,
dns
or
and
it'll
be
able
to
write
straight
to
the
the
local
services.
A
But
if
it's
going
across
cluster,
you
might
need
to
set
up
things
like
destination
rules
and
service
entries
in
istio,
and
this
is
one
of
those
things
as
a
developer.
You
don't
have
to
worry
about
that
right.
It's
an
implementation,
an
implementation
detail
of
istio,
so
we've
obstructed
that
away.
So,
regardless
of,
if
you're
in
a
single
cluster
or
a
hundred
clusters,
the
configuration
should
be
the
same
in
terms
of
blue
mesh
resources
and
we'll
figure
out
all
the
nitty
gritty
details
in
terms
of
what
istio
needs
to
do.
A
So
I
mentioned
a
second
ago
workspaces
and
that's
something
that
helps
us
scale
quite
a
bit.
So
I
just
wanted
to
talk
about
workspaces
a
little
bit.
So
what
is
a
workspace?
I
just
wanted
to
kind
of
define
this
term,
because
it's
a
little
vague
so
at
its
core,
a
workspace
is
pretty
much
how
we
obtain
multi-tenancy
in
an
extremely
large
service
mesh.
A
It's
really
a
logical
partition
that
helps
you
kind
of
config
configure
different
configuration
boundaries
and
you
can
use
it
to
divide
responsibilities
and
it's
really
going
to
depend
on
how
your
org
and
how
your
engineering
teams
are
set
up.
But
you
can
divide
boundaries
in
multiple
different
ways.
For
example,
it
could
be
a
specific
team.
Has
ownership
over
a
specific
workspace
or
it
could
be.
A
specific
application
is
siloed
to
a
specific
workspace
or
maybe,
depending
on
your
organization.
A
All
the
workspaces
from
a
developer
perspective
is
a
list
of
clusters
and
a
list
of
namespaces
within
those
clusters,
and
we
kind
of
want
this
workspace
paradigm
to
replace
as
a
developer,
instead
of
you
having
to
think
about
which
clusters,
which
name
spaces
am
I
running
in
you
should
just
think
about
this-
is
my
workspace
and
everything's
in
there
and
then
all
of
the
rest
of
it
kind
of
fades
away.
So
the
same
way
that,
like
kubernetes,
makes
containers
be
cattle
instead
of
pets.
A
So
why
would
you
use
these
these
workspaces-
and
I
touched
on
a
few
of
these
points
just
a
second
ago,
but
essentially
just
to
give
some
examples,
workspace
isolation,
so
you
can
imagine.
Maybe
a
billing
team
is
working
in
a
cluster
that
might
also
have
a
storefront
app
and
you
don't
want
those
two
teams
to
be
worried
about
stepping
on
each
other
or
if
someone
applies
some
kind
of
configuration-
and
I
feel
like
this
is
a
scenario
that
everyone's
run
into
once
you've
reached
scale
with
service
mesh.
A
You
don't
want
one
team
to
be
able
to
apply
some
kind
of
accidentally,
overly
broad
configuration
that
shoots
another
team
in
the
foot.
So
really
what
you
can
do
is
you
can
have
admins
set
some
smart
defaults
here
and
because,
ultimately,
if
you
think
about
it,
your
application
teams
probably
don't
care
about
some
of
this
kind
of
dev-opsy
stuff.
They
just
want
their
applications
to
work.
So
you
can
have
your
admin
team
set.
A
A
Having
said
that,
you
know
the
reason
you're
going
with
service
mesh
is
because
you've
got
a
lot
of
services
that
are
running
inside
a
single
organization
and
a
lot
of
the
time.
These
services
are
probably
going
to
be
useful
in
more
than
one
context,
so
we
really
want
to
promote
sharing
services
across
the
org.
A
So,
even
though
everything
is
by
default
isolated
to
its
own
workspace,
we
allow
users
to
explicitly
pick
individual
resources
and
mark
them
as
exported,
which
makes
them
publicly
available
within
the
service
mesh
to
any
other
workspace,
or
you
can
specify
a
subset
of
workspaces.
So,
for
example,
you
might
have
a
notifications
team,
that's
working
on
a
service
for
push
notifications
and
a
service
for
email
notifications
and
under
the
hood
that
might
be
represented
by
20
different
microservices
running
in
a
local
cluster.
A
So
some
of
these
shared
services,
like
notifications
or
billing,
or
checkout
or
user
profiles,
are
available
across
teams
and
that
way
teams
can
just
expose
the
public
interface
while
keeping
the
implementation
details,
all
private,
so
kind
of
looking
at
what
that
might
look
like
as
a
concrete
example,
I
know
there's
a
lot
going
on
in
this
diagram
as
well,
but
we
can
kind
of
take
it
slowly.
So
here
we've
got
three
clusters
essentially,
and
you
can
see
on
the
top
left.
We've
got
cluster
one
in
the
top
right.
A
We've
got
cluster
two
and
then
on
the
bottom.
We've
got
a
very
large
cluster
for
cluster
three
within
these
clusters,
you
can
see
these
kind
of
teal
colored
boxes,
which
represent
workspaces,
so
you
can
see
in
the
top
left.
We've
got
our
api
gateway
workspace
and
that's
just
confined
to
cluster
one
and
the
interesting
thing
about
this
is
like
we
mentioned
earlier,
because
this
is
kind
of
a
very
critical
piece
of
your
infrastructure.
A
We
can
cordon
off
your
api
gateway
into
a
specific
cluster,
and
maybe
you
have
much
more
stringent
security
architecture
or
security
posture
for
this
cluster
everything's,
going
to
be
a
lot,
a
lot
more
locked
down
and
specifically
who's
able
to
configure
things
and
tweak
things
on
this
cluster
can
be
a
lot
more
strict
than
your
other
kind
of
more
internal
facing
clusters.
Cluster
two
is
is
an
interesting
one,
because
you
can
see
here.
We've
got
our
book
info
application
workspace.
A
But
if
you
look
at
this
blue
line,
you'll
see
it
actually
extends
beyond
cluster.
One
and
go
our
excuse
me
beyond
cluster
two
and
goes
into
cluster
three,
so
this
workspace
actually
crosses
cluster
two
and
cluster.
Three-
and
importantly,
like
I
mentioned
before,
to
the
developers
working
in
this
book
info
application
workspace.
A
The
cluster
boundaries
should
be
invisible.
They
shouldn't
care
which
of
these
two
clusters
their
application's
running
on
it
all
should
just
feel
like
a
single
cohesive
unit
to
them,
but
you
can
also
see
in
cluster
3,
in
addition
to
half
of
the
book
info
application
workspace.
It's
also
housing
the
notifications
workspace,
so
that
could
be
a
service
that
is
exposes
some
kind
of
an
api
to
send
push
notifications
to
our
users.
A
So
how
all
of
these
things
kind
of
interact
with
each
other
there's
some
kind
of
just
additional
arrows
drawn
here,
so
we
can
kind
of
like
walk
through
it,
but
essentially
our
bottom
left
workspace,
which
is
our
notifications.
Workspace
is
being
exported
and
you
can
see
in
our
book
info
application,
we're
importing
the
notification
service.
So
it's
using
that
service
from
a
different
workspace
and
then
it's
actually
exporting
to
the
api
gateway
workspace,
a
route.
So
ultimately,
this
application.
A
Now
these
are
some
of
the
api
glue,
mesh
resources
that
are
used
for
this,
but
we
don't
want
to
get
too
much
into
detail
on
this.
It's
a
little
bit
too
low
level,
but
if
you
are
interested
in
kind
of
knowing
exactly
how
this
works
under
the
hood,
I'd
strongly
recommend
you
check
out
lynn
and
scott's
talk
on
the
glue
mesh
v2
api.
They
go
into
a
lot
more
detail
on
how
each
of
these
individual
crds
works
and
how
everything
plays
together.
B
Thanks
so
here
you
can
see
a
very
similar
graph
to
what
we
were
looking
at
earlier.
So
in
the
bottom,
we
have
our
mini
workflow
clusters
and
in
terms
of
observability
within
each
of
our
workflow
clusters.
We
already
mentioned
that
we
have
an
agent
deployed
part
of
that
agent's
responsibilities
is
to
serve
as
a
metric
server,
so
the
side
cards
for
all
the
services
within
your
cluster.
B
B
We
have
that's,
also
served
the
purpose
that
it's
also
serving.
Is
that
there's
a
metric
sync
for
prometheus
and
we
also
have
a
separate
dashboard
deployment
which
I'll
talk
about
in
a
bit,
but
essentially
the
way
that
we
glue
mesh
really
makes
observability
super
powerful
for
multi-cluster
service
mesh
is
that
we
have
our
agents
in
all
of
our
workload
clusters,
forwarding
metrics
and
access
logs
to
the
management
cluster.
B
The
dashboard
specifically
queries
prometheus
and
the
kube
api
provide
a
single
place
for
viewing
all
the
service
metadata,
but
you
can
also,
as
I
mentioned,
hook
up
grafana
for
custom-made
dashboards
in
the
management
cluster,
and
it
will
show
you
all
the
data
for
all
of
your
workload
clusters
and
sheen
will
actually
talk
a
little
bit
more
about
the
glue
mesh
dashboard.
Now.
A
Absolutely
thanks
javon.
So
when
we
talk
about
glue,
mash
dashboard,
we
keep
kind
of
coming
back
to
this
point
of
single
pane
of
view
glass
and
now
we're
going
to
go
through
exactly
what
that
means.
And
you
know
our
dashboard
actually
is
going
to
support
multiple
different
user
roles
depending
on
what's
important
and
how
much
access
you
have
within
this
large
global
service
mesh.
A
Second,
a
lot
of
bandwidth,
a
lot
of
resources
which
teams
are
maybe
have
an
increased
error
rate
or
high
latency,
and
it
kind
of
comes
back
to
some
of
those
red
statistics
that
javon
mentioned
earlier.
You
want
to
be
able
to
see
all
of
this
at
a
glance,
so
it's
kind
of
like
your
check
engine
light.
Is
everything
doing
okay
at
the
service
mesh
now
as
an
admin
you're
able
to
drill
down,
and
you
get
a
similar
view
to
what
you
might
get
as
a
workspace
admin?
A
So
a
workspace
admin
is
going
to
see
something
closer
to
this
and
a
workspace
admin
is
someone
who's
in
charge
of
running
an
individual,
specific
workspace?
So
perhaps
like
our
notifications
team,
as
we
were
talking
about
in
the
earlier
example,
and
really
what
kind
of
things
you're
looking
for
here
is
which
other
workspaces
is
your
workspace
interacting
with
so
which
upstream
dependencies
do
you
have?
A
In
addition
to
this,
you
cannot
get
access
to
that
service,
catalog
that
I
mentioned
earlier
so
services
that
maybe
you're
not
yet
using.
But
you
get
a
list
of
here's
some
things
that
other
folks
within
your
service
mesh
are
exporting
that's
available
for
other
teams
to
use,
and
maybe
you
can
use
some
of
those
to
not
reinvent
the
wheel
and
you
know,
get
better
code,
reuse
and
make
sure
you're
not
rebuilding
the
same
services
over
and
over
again
and
focusing
on
what
really
matters.
A
So
next
up,
you
know
we
want
to
talk
about
this
graph
view,
so
our
graph
view
is
essentially
a
view
of
all
of
the
mesh
service
relationships
all
of
the
services
within
your
mesh,
regardless
of
which
workspace
they're
running
on,
regardless
of
which
cluster
they're
running
on
and
regardless
of
which
name
space
they're
running
on
you
want
to
see
an
overview
of
all
of
these.
Now,
of
course,
this
view
is
going
to
be
tapered
down
to
whatever
logged
in
user
has
access
to.
A
So,
if
you're
a
super
admin,
you
might
see
a
graph
of
just
all
of
your
workspace
level,
things
that
you
can
drill
into.
If
you're
a
workspace
admin,
you
can
see
a
graph
of
everything
that
you
have
access
to
and
all
of
the
exported
resources
that
you're
using
and
if
you're,
just
a
workspace
developer.
Maybe
you
can
only
see
some
of
the
services
that
you're
personally
allowed
to
administer,
but
really
what
we're
looking
at
here
and
some
of
the
really
strong
value
add
over
something
like
a
vanilla,
istio
deployment
is,
you
can
see
here?
A
What's
working,
what's
not
working
and
of
course
all
of
these
communications
is
all
are
also
encrypted
using
mtls.
So
you
can
see
in
this
diagram.
You
see
these
little
lock
icons,
and
so
you
can
guarantee
that
these
are
all
being
encrypted
end
to
end.
You
could
also
highlight,
if
anything
was
not
encrypted.
We
could
you'd
see
like
a
red
line
here
and
an
open,
lock
icon.
A
A
What's
the
response
time
and
how
much
of
an
error
rate
if
any
of
these
numbers
start
creeping
up
a
little
bit
instead
of
these
nice,
pretty
blue
circles,
you're
going
to
start
getting
yellow
for
warning
or
red
for
errors,
and
it
should
be
again
pretty
much
immediately
obvious
to
someone
glancing
at
this.
Is
there
anything
wrong?
A
Is
there
anything
we
need
to
investigate
and
you
should
be
able
to
drill
down
even
further
into
any
individual
service
to
see
where
it's
deployed
which
clusters
it's
running,
on
which
workspaces
it's
accessible
from
and
which
are
all
of
the
policies
that
you've
applied
to
it?
But
this
is
just
kind
of
like
your
jumping
in
point
to
get
that
that
high
level
view
and
again
we're
kind
of
like
focusing
on
the
fact
that
this
is.
A
So
in
summary,
you
know:
we've
talked
about
how
gmesh
helps
you
enable
large
scale.
Multi-Tenant
service
mesh
we've
talked
about
how
it
lets
you
scale
from
a
single
cluster
service
mesh
to
100
cluster
service
mesh
and
beyond
that,
without
changing
the
configuration
that
your
own
engineers
and
developers
need
to
worry
about,
we've
talked
about
how
glue
mesh
can
help.
You
monitor
the
service
mesh
health
across
all
of
your
clusters.
We've
talked
about
how
it
aggregates
it
all
into
a
single
pane
of
glass.