►
Description
Istio supported metrics, tracing, and logs since the beginning when we launched Istio, without application developers needing to do anything other than injecting the sidecars and propagating the B3 trace headers. While telemetry works great for single cluster, how does it work with massive volumes of telemetry for large multi-cluster Istio service mesh deployments?
In this Livestream, we will review some of the scalability challenges we have for collecting telemetries for Goo mesh, why we decided to leverage Open Telemetry to tackle our scalability issues, and how it worked with live demos!
--
Join Solo on Slack: https://slack.solo.io
Follow us on Twitter: https://twitter.com/soloio_inc
Follow us on LinkedIn: https://www.linkedin.com/company/solo.io/
Past episodes: https://github.com/solo-io/hoot
A
A
B
C
B
A
A
Yeah,
we're
very
excited
to
have
you
too
Josh.
Thank
you
so
much
and
let's
say
hi
to
our
audience:
Peter
thanks
so
much
for
joining
and
bonjour,
and
thank
you
so
much
for
joining
we're
excited
you
guys
are
here
so
feel
free
to.
Let
us
know
if
you
have
any
questions
with
that.
Oh
by
the
way,
I
have
one
big
news.
I'm
I
would
like
to
announce
so
Peter
actually
wrote
A
Blog
about
this.
Let
me
send
in
the
chat
so
Friday.
A
It's
your
ambient
service
match
is
merged
to
Upstream
istio.
So
what
does
this
mean?
This
means
it's
your
1.18.
It's
going
to
have
it's
still
ambient.
Whenever
we
release
it's
your
1.18,
so
you
can
already
grab
one
of
the
istio's
develop
fields
and
try
ambient
yourself.
So
you
no
longer
need
to
build
ambient
to
follow
up
the
latest
development
and
in
fact,
Peter
actually
wrote
A
Blog
about
this
just
over
the
weekend,
which
I
will
put
in
the
chat
for
everybody
to.
C
Yeah
so
as
we're
gonna
as
we're
gonna
show
in
some
of
the
content,
we
have
a
lot
of
the
scalability
challenges
we
had
with.
Our
existing
system
had
to
do
came
more
with
restrictions
in
flexibility,
so
the
open
Telemetry
allowed
us
to
create
a
pipeline
and
Implement
a
pipeline
that
ended
up
because
it
was
separate
because
it
was
more
flexible.
It
allowed
us
to
to
just
build
it
in
a
way
that,
where
I
had
a
better
scaling
factor
and
we'll
demonstrate
that
as
we
go
through,
it.
C
Yeah
so
I
mean
I
I
want
to
go
through
like
a
little
bit
of
our
old
Pipeline
and
I.
Think
that
will
help
highlight
the
challenges
that
open
Telemetry
ended
up,
solving,
we'll
go
through
how
we
applied
it
to
our
product
and
then
even
on
top
of
that,
like
the
as
far
as
the
scalability
aspect,
Renoir
has
a
great
demo
to
show
us
exactly
what
we
can
do
and
how
it's
actually
applied
to
a
running
system.
C
S
all
right
so
I
want
to
give
a
little
background
before
we
jump
In,
Too
Deep.
So
one
way
that
envoy
can
export
metrics
and
propagate
metrics
is
by
using
something
called
the
envoy
metric
service.
So
this
is
really
just
an
interface
that
can
be
implemented
through
their
public
library
in
any
and
register
with
any
grp
server.
A
C
It's
it's
outside
of
it,
so
it
can
be
anywhere
and
I'll.
Show.
I
have
a
small
example
here
of
how
it's
how
it's
configured.
So
when
you're
configuring
your
mesh,
you
can
add
a
singular
like
Envoy
metric
service,
so
for
a
given
mesh,
you
can
export
that
data
to
that
like
ongoing
metric
service
and
in
our
example
in
our
infrastructure,
that
is
our
glute
mesh
agent.
C
However,
with
that
flexibility,
it
also
doesn't
come
with
a
lot
of
like
baked
in
things
that
otel
does
that
we
want
to
make
use
of
so
we'll
kind
of
show
the
pros
and
cons
of
it
being
very
just
open-ended,
like
this,
going
to
our
infrastructure
to
really
Zoom
back
at
a
high
level.
This
is
that
connection
that
we
were
just
showing
between
a
workload,
an
Envoy
sidecar.
C
This
is
this
is
great
in
some
ways,
because
it's
so
simple
and
you
can
Implement
anything
on
top
of
it.
It's
really
easy
to
install
the
traffic
is
all
secure
via
grpc.
So
there's
a
lot
of
there's
a
lot
going
for
this
one
big
issue
with
it.
Is
that,
like
a
big
con
of
it
for
us,
is
that
it's
coupled
to
our
product
and
we
don't
want
the
glue
agent
to
be
coupled
with
the
metrics
Pipeline
and
same
with
the
glue
management
server?
C
A
So
so,
basically,
you
are
saying
because
of
the
glue
agent,
if
there's
a
need
of
volumes
of
metrics
increase
to
a
bigger
degree,
then
you
would
have
to
kind
of
scale
up
glue
agent
and
also
the
management
server
to
accommodate
the
increased
volume
of
metrics,
which
you
don't
necessarily
want
to
do
that
right,
because
that
components,
the
agent
and
management
server
handles
many
other
things.
In
addition
to
the
metrics,
is
that.
C
Exactly
the
we
didn't,
we
wanted
to
separation
of
concerns
there,
and
if
that
was
the
only
problem
with
this
setup,
we
would
have
maybe
split
that
off
to
its
own
service
and
still
done
it
the
same
way
but
there's
a
lot
of
other
benefits
to
like
hotel
we'll
go
into,
but
that
is
a
big
driver
of
a
big
driver
and
a
big
factor
when
talking
about
the
scalability
of
it
as
well.
Okay,.
A
C
Yes,
so
we
we
evaluated
a
couple
things,
but
like
we,
we
definitely
landed
on
hotel,
pretty
quick,
mainly
because
we
were
getting
a
lot
of
feedback
from
our
field.
Engineers
they
love
using
otel,
there's,
actually
some
examples
of
it
being
used
manually
to
implement
like
our
metrics
pipeline
out
in
the
field.
C
So
it
was
right
at
the
top
of
our
list
to
evaluate
because
of
the
requests
and
the
pushback
from
the
field,
and
it
ended
up
just
being
great
advice,
because
you
know
once
we
once
we
really
dug
into
it
and
tried
to
try
to
make
sure
it
was
solving
every
one
of
our
problems
tried
to
break
it
tried
to
push
it
to
its
limit.
It
really
just
like
exceeded
our
expectations
and
did
everything
we
need
so
that
we
they
made
it
easy
for
us
with
the
right
advice.
There.
C
So
we
there's
no
simple
approach
for
forwarding
data
to
customer
back
end,
so
we
could
forward
to
our
own
management
server,
but
we
couldn't
like
forward
to
other
existing
customer
back-ends
like
different
cloud
provider
back-ends
or
like
Splinter
datadog
right.
Then
we
talk
about.
We
talked
about
the
scaling
being
coupled
together.
C
Another
big
one
related
to
forwarding
that
data
is
the
Transformations
were
directly
in
our
application
code,
because
we
were
just
reading
from
that
stream
directly
and
it's
not
easily
reproducible
or
exposed
to
the
user
for
them
to
even
know.
What's
going
on
there
and
then
Advanced
resiliency
features
that
old
approach.
Definitely
he
definitely
puts
it
the
burden
on
the
developer
to
build
in
any
kind
of
resilience
we
did,
but
when
you're
doing
that,
it
also
feels
a
little
like
you're
Reinventing.
The
wheel.
C
Hotel
has
a
lot
of
support
here
so,
and
it
just
comes
baked
in
automatically.
So
we
wanted
to
leverage
that
all
right
so
before
again
before
we
get
into
the
new
architecture,
I
wanna
Prime,
with
just
a
little
bit
of
knowledge
of
Hotel,
because
we're
going
to
be
talking
about
some
of
these
Concepts
a
lot
and
open
Telemetry
hotel
for
short.
C
So
before
we
get
on
to
the
right,
the
left
here
is
an
is
an
example
of
what
a
configuration
file
would
look
like
for
configuring,
the
hotel,
collector
and
the
reason
I'm
showing
this
is
because
it
really
highlights
well
the
four
main
Concepts,
and
these
might
sound
a
little
abstract
at
first,
but
there's
four
things:
the
receivers,
processors,
exporters
and
pipelines,
and
so
you
can
configure
receivers,
processors
and
exporters
to
do
the
different
job
of
like
receivers
are
either
are
bringing
in
the
data
from
different
types
of
sources.
They
can
be
push
or
pull.
C
Processors
are
transforming
the
data
and
exporters
are
writing
it
out
to
whatever
destination
you
want.
These
are
reusable.
So
when
you
configure
one,
it's
not
actually
doing
anything
until
you
put
it
in
a
pipeline.
So
down
here,
I
have
a
theoretical
pipeline
where
I've
combined
this
receiver
processor
exporter.
This
concept
is
very
important,
though,
because
it
we
have
like
baked
in
when,
during
our
installation
receivers
processes,
exporters.
C
The
exporter
goes
to
our
back
end
right,
but
it's
really
important,
because
users
could
easily
just
write
their
own
little
pipeline
to
reuse,
Our,
receiver,
processor
and
just
add
their
own
exporter.
So
it
becomes
really
simple
for
them
to
like
get
that
data
exporting
to
their
own
Telemetry
backend.
So
I
want
to
highlight
this
because
that's
how
that's
how
hotels
solving
a
couple
of
those
problems
we
highlighted
in
the
last
slide,
yeah.
A
C
So
the
receiver
is
a
receiver
is
very
generic,
so
you
can
have
a
bunch
of
different
receivers.
You
can
have
receivers
that
are
pulling
information
from
different
sources.
You
can
have
receivers
like
the
one
we
use
Prometheus,
that
is
that
is
scraping,
and
you
can
have
receivers
that
that
actually
are
just
listening
for
information
to
get
pushed
to
them.
C
So
the
hotel
framework
is
extremely
generic
and
these
receivers
can
really
be
pretty
much
anything.
A
developer
wants
to
make
them
to
be
right.
So
there's
a
lot
of
variety
and
I
I.
Think
renor
is
going
to
show
us
like
if
we
have
time
he's
going
to
show
us
a
few
more
examples
of
this
too.
So
we
can
get
more
ideas
of
what
that
can
do,
and
even
our
application
gives
a
few
of
these
examples
just
to
on
this
side.
C
All
I
really
want
to
show
out
of
this
is
some
of
the
really
important
processors
which
are
batch,
attributes
and
filters
so
batch
or
batching
up
the
messages.
So
it's
more
efficient
to
send
over
the
wire
attributes,
drop
I
mean
I'm.
Sorry
attributes
relabel
metrics,
so
just
like
Prometheus
would
like
you
can
relabel
your
metrics
and
filters
can
drop
metrics
that
you
don't
want
to
use,
or
you
don't
want
to
forward
so
continuing
on.
C
Why
open,
Telemetry
we've
answered
a
lot
of
these
questions
already
I
just
want
to
highlight
a
few.
Are
that
it's?
It
is
easier
to
collect
from
other
components
because
of
a
variety
of
receivers.
We
don't
just
want
to
collect
for
misdial
workloads.
We
don't
just
want
to
collect
from
this
DOD.
We
also
wanna
be
able
to
collect
from
optional
components
that
can
get
installed
by
our
products.
C
So
this
allows
us
to
easily
like
add
those
different
types
of
collection
in
we
touched
on
pretty
much
everything
else
here,
except
the
hotel,
collector,
Builder
and
renor
is
going
to
go
over
the
scalability,
but
I
next
thing
I
do
want
to
show,
is
Hotel
collector,
Builder
and
I'll
be
very
quick
on
this
one,
because
it's
not
terribly
important,
but
it
is.
It
is
a
Big
Boon
for
maintaining
the
hotel
collector.
If
someone
decides
to
go
down
this
path
so.
A
C
Yeah,
so
for
the
open,
Telemetry
Builder.
This
is
I
want
to
highlight
this
because
it
was
really
cool
for
us.
It
allowed
us
to
manage
much
much
less
code
when
implementing
this
pipeline,
so
the
open,
Telemetry
Builder
is
used
to
in
for
the
open
source
to
build
its
images
and
binaries
that
it
ships
through
the
public
chart
right.
C
C
So
yeah
to
get
to
the
architecture
at
a
high
level
rather
than
data
flowing
between
the
agent
and
the
glue
server.
Now
for
metrics,
we
have
it
going
between
the
otel
collector
and
the
metrics
Gateway
One
thing
I
want
to
highlight
and
emphasize
is
that
just
like
we
were
talking
about
with
that
hotel,
collector,
Builder
and
last
one
we're
building
our
own
image.
C
So
the
hotel
collector
on
this
side
we've
configured
it
to
run
as
a
Daemon
set.
So
we
get
that
automatic
scaling
Factor
per
node,
so
it
only
collects
metrics
from
the
node
that
it's
looking
for
and
then
we've
also
it's
collecting
from
istio
workloads,
istio-d,
custom
application
metrics
from
our
agent
and
then
other
static
services
that
we
may
optionally
install.
C
This,
then
just
for
allows
the
metrics
to
be
scraped
directly
to
this
Prometheus.
You
can
see
here
that
the
minimum
metrics
set
is
emphasized
here,
because
we're
filtering
down
to
the
only
the
metrics
we
need
in
our
UI.
That
way
we
don't
overburden
Prometheus
and
we
allow
it
to
scale
better
this.
In
our
experience
in
testing
this
metrics
Gateway
and
hotel
collector
can
far
exceed
anything
Prometheus
can
handle.
So
it's
important
to
like
only
send
the
metrics.
We
need
on
really
really
big
setups.
C
So
then
also
I
just
want
to
highlight.
We
mentioned
this
before,
but
like
the
match
or
Matrix
maximum
metric
set,
it's
very
easy
for,
like
customers
to
add
a
new
Pipeline
and
ship.
Actually,
all
the
metrics
if
they
want
to
their
own
like
Splunk
or
datadoc,
and
this
actually
was
added
into
our
installation-
to
add
this
extra-
add
the
extra
processors
to
not
for
the
maximum
metric
set.
So
even
though
we
don't
use
it
in
our
pipeline,
it
is
in
the
configuration.
C
C
C
So
there's,
if
you
want
to
take
like
existing
pre-meath,
these,
like
scrape
config
and
just
drop
it
into
the
hotel,
collector
config,
you
can
do
that
and
it
just
works
and
it
works
exactly
the
same.
But
it
just
allows
you
to
like
propagate
up
and
centralize
them
all.
So
you
can
write
it
at
one
place,
but
I
find
that
very
cool.
Then
we
have
our
processors,
which
are
a
custom
one
that
is
doing
some
in
like
custom
Transformations
for
application,
attributes,
which
is
relabeling
filter,
which
is
dropping.
C
The
last
thing
I
want
to
touch
on
is
the
open,
Telemetry
protocol,
which,
if
you
look
at
if
you're
looking
in
the
documentation
is
referred
to
as
otlp
very
commonly
and
O2
otlp
is
important
important,
because
it's
the
language
that
the
whole
collector
talks
in
so
once
it's
received
and
data
is
like
brought
into
the
collector.
It's
Autumn.
It's
immediately
converted
to
this
format.
So
if
you're
actually
like
building
your
own
processor
receiver,
exporter
you're
going
to
be
working
with
this.
C
This,
like
Proto
API
to
begin
with,
but
it's
all
it's
all
public.
If
you
end
up
like
wanting
to
make
your
own
own
listener
for
otlp,
you
even
could
so,
but
it's
just
an
important
thing
to
discuss,
because
this
is
a
very,
very
core
exporter
receiver
for
open,
Telemetry
and
you'll,
see
on
the
Gateway
side,
it's
otlp
receiver,
and
so,
if
you're
talking
between
collectors
ever
this
is
pretty
much
the
de
facto
way
to
do.
It.
A
C
It's
it
has
it
like.
Resiliency
features
are
very
well
supported,
it's
secure
because
it's
grpc,
and
so
this
is,
and
it's
the
least
work,
because
it's
like
not
transforming
the
data
back
to
anything
again,
and
so
it's
it's
definitely
the
de
facto
way
to
like
to
move
information
from
one
collector
to
the
other,
which
is
exactly
what
we're
doing
with
this
line
like
right
here.
A
Yeah,
thank
you
so
much
for
explaining
this,
because
I've
been
got
hung
up
on
the
previous
slides,
where
you
had
the
premises
as
a
receiver.
I
was
like
that's
auto,
run
its
own
premises
right.
Can
you
kind
of
explain
it's
pulling
some
of
the
code
from
the
premises
project,
so
it
can
pull
data
and
script
the
endpoint,
so
that
was
very
cool.
C
Read
it
you're
like
oh,
this
is
the
scraper
like
it
runs
it
itself,
but
it
is
a
little
confusing.
We
first
look
at
it.
The
exporter
is
actually
just
hosting
the
metrics
and
making
them
available
to
scrape
from
an
actual
Prometheus
server
or
another
hotel,
collector
I
guess
if
you
want
to,
but
so
yeah
from
from
here.
I
think
I'll
hand
it
over
to
renor
for
his
demo.
A
B
B
Now,
it's
just
one
small
component,
so
in
a
lot
of
ways,
authorital
collector
would
win,
especially
considering
flexibility,
supporting
multiple
backends,
and
so
on
is
like
how
much
we
can
scale
and
how
much
traffic
or
metrics
and
as
well
the
traffic
that
generates
that
metrics.
We
can
support
for
our
customers.
A
Yeah
all
right,
we
actually
have
a
question
in
the
chat,
let's
say
hi
to
Moreno
thanks
so
much
for
dropping
bye
and
we
have
a
question
from
Geo.
This
looks
like
a
really
cool
setup.
Thank
you.
We
are
exporting
metrics
and
logs
to
our
external
service.
Would
it
be
possible
to
config
exporters
for
the
management
cluster
as
well.
C
Okay,
yes
yep,
and
that
is
a
less,
not
an
architecture
we
talked
about
here,
but
also
you
can
install
the
collector
in
the
form
of
the
demon
set
as
well.
So
if
you
want
to
use
that,
because
demon
sets
do
scale
a
little
better
if
you're
running
like
a
big
cluster,
if
you
want
to
use
that
you
can
install
it
like
that
and
use
that
to
export
the
data,
but
the
Gateway
could
also
be
used.
If
you
want
to
reuse
that
component.
B
A
B
So
here's
lens
and
it's
a
nice
IDE
for
like
showing
anything
in
kubernetes,
and
it
makes
a
little
easier
for
the
demos
Prometheus
is
in
the
management
plane.
So
we
will
switch
to
the
management
cluster.
We
deployed
these
in
the
glue
mesh
namespace.
So
let's
go
to
Google
mesh,
and
here
we
have
Prometheus
and
this
vertical
Matrix.
Let's
start
with
Prometheus
I
will
port
forward
this,
and
then
we
can
see
in
the
UI,
the
workloads
and
the
traffic
and
so
on.
B
So
we
left
this
running
for
24
hours
or
even
longer,
but
we
will
look
only
at
the
24
hours
to
make
sure
that
this
is
actually
pulling
from
The
Matrix
Gateway.
This
is
like
always
with
Prometheus.
You
can
go
to
targets
and
you
can
make
sure
that
it
is
pulling
from
your
job.
In
our
case,
this
job
is
pulling
from
the
metrics
Gateway
in
the
port
1991..
So
if
it's
doing
that,
it
means
that
the
metrics
Gateway
is
exposing
those
in
1991
and
let's
remember,
to
check
this
and
we
get
there.
B
The
next
thing
that
we
can
do
is
we
can
just
make
sure,
like
a
simple
hello
world
test,
like
all
the
metrics
really
coming
here.
We
see
that
the
metrics
are
already
here,
so
we
verified
this
step.
Now,
let's
go
lower
to
the
metrics
Gateway.
The
first
thing
that
we
want
to
check
is
that
the
metrics
are
being
exposed
in
1991,
and
then
we
will
check
how
these
are
received
from
the
auto
collectors
from
the
data
planes.
B
Switching
over
to
here,
let's
go
to
workloads.
We
said
this
is
a
deployment
it's
configured
with
a
config
map.
It's
on
the
config
map.
We
find
that
Google
metric
scale,
the
conflict,
and
these
are
like
what
Josh
explained
to
us.
Here's
the
receiver
here
is
the
processor,
and
here
is
the
exporter,
and
as
we
see,
we
are
exporting
everything
import,
1991
and
the
default
platform.
Metrics
is
forward,
slash
metrics,
so
we
don't
have
to
write
that.
But
if
you
want
to
change
it,
you
can
configure
that
as
well.
B
B
It
has
a
maximum
message,
size
of
12
megabytes
and
then
that
certificates
to
authenticate
itself
to
the
clients.
Additionally,
it
has
as
well
like
Prometheus
receiver,
which
means
that
it's
another
way
to
get
metrics
into
the
auto
collector.
What
this
is
doing.
It's
just
escaping
its
own
Telemetry
in
the
port,
8080
8080..
B
So
quite
a
simple
receiver
and
as
you
see,
this
is
like
the
configuration
simple
configuration
that
you
can
drag
and
drop
from
what
you
had
in
promoters
and
use
it
here
with
a
small
difference
like
improvis,
you
can
use
just
a
dollar
sign,
but
when
you
are
drag
and
dropping
and
using
like
configuration
from
Prometheus,
you
would
have
to
escape
this
dollar
sign
with
double
dollar
sign.
This
is
an
environment
variable,
that's
why
it's!
B
After
that
we
have
two
processors
patching
those
metrics
that
are
pulled
together
and
then
as
well.
The
memory
limiter,
if
it
ever,
is
almost
getting
to
the
point
of
exceeding
memory
so
that
we
start
to
drop
metrics
and
not
get
the
Pod
killed,
or
all
of
these
wouldn't
really
have
much
or
bring
any
value.
If
we
don't
combine
these
in
a
pipeline
and
here's
the
pipeline
here
we
have
the
Matrix
pipeline,
which
is
receiving
through
otlp
and
Prometheus.
B
It's
processing
false,
received
metrics
with
these
two
processors
and
then
it's
exporting
those
in
Prometheus
format
in
the
port
1991,
which
is
then
collected
by
Prometheus,
as
we
showed
here
good.
So
now,
if
we
are
receiving
metrics
in
dashboard,
it
means
that
we
need
a
public
IP
address
so
that
we
can
so
that
all
the
hotel
collectors
can
push
the
metrics
over
here
so
to
make
sure
that
we
have
the
public
IP
address.
We
just
have
a
service
here
it
is,
and
this
is
the
public
IP
address.
B
Now,
let's
switch
over
to
one
of
the
client
clusters.
The
data
plane
clusters
here,
if
we
go
to
demon,
sets
and
then
in
the
glue,
Mash
namespace
you'll,
see
that
we
have
the
Glu
Matrix
collector
agent,
which
has
four
our
replicas.
So
one
per
node
and
we
have
three
clusters
each
for
replica
four
nodes,
meaning
that
in
total
there
are
12
nodes,
and
this
is
as
well
configured
simply
with
a
config
map
and
it's
the
same
image.
It's
just
that
the
configuration
changes.
Here's
the
receiver,
it's
using
promo
sales
for
scraping.
B
B
A
B
Exactly
so,
and
then
here
is
the
double
Escape
like
when
we
create
those
reg
x's,
and
we
take
one
group
and
we
want
to
use
it
usually
in
promoters,
you
would
just
use
dollar
sign
one
when
you
paste
it
here.
You
just
have
to
keep
in
mind
to
add
another
dollar
sign
so
that
it
escapes
it
and
it
doesn't
think
that
it's
an
environment
variable
or
something
similar.
B
B
So
now,
if
you
scroll
down,
we
have
other
static
configuration
for
scraping,
like
the
estudi
metrics.
Let's
skip
over
this
and
get
to
the
processors.
Here
is
our
own
custom
processor,
and
then
we
have.
We
are
filtering
to
the
minimum
of
the
metrics
so
that
we
don't
like
overload
promotheos
and
it
dubs
metrics.
That
would
be
important
just
because
it's
full,
so
we
are
limiting
it
to
only
what
we
need
to
show
all
these
service
to
service
traffic
in
our
glumash
dashboard.
B
B
A
B
B
B
So
this
is
how
this
comes
together.
So
there
was
the
question
by
Joe
like
what,
if
we
want
to
push
to
another
backhand
or
what,
if
we
want
to
push
other
metrics
like,
maybe
you
don't
want
to
drop
and
reduce
it
to
a
minimum,
and
you
want
to
push
the
full
data
set,
so
actually
what
it
would
mean
and
I
will
just
copy
this
open.
A
new
window
means
that
you
would
have
to
define
a
new,
let's
say,
you're
happy
with
the
metrics
that
you
receive.
You
just
want
to
export
those
to.
B
Let's
say
to
data
doc:
that's
a
nice
example!
So
let's
go
over
to
the
exporters,
here's
the
exporter,
then
we
would
have
to
define
a
new
exporter
and
as
soon
as
we
want
to
do
that,
the
first
step
is
really
to
just
go
and
open
the
hotel
con
trip
repository
where
everybody
contributes
their
own
components,
which
could
be
exporters
and
some
inner
level
which
can
be
exporters,
processors
and
receivers.
So
in
this
case
we
want
to
export
to
data
doc.
So,
let's
open
this,
do
we
have
a
data,
doc
exporter?
B
B
Data
doc
apparently
has
as
well
the
official
documentation
page
where
you
can
read,
but
what
I
quite
often
do
as
well
is
I
just
check
the
test
data,
where
they
specify
different
configs
on
how
you
can
like
configure
your
thesis
specific
component
like
in
this
case.
They
are
specifying
this
one.
So,
let's
copy
it,
let's
copy
the
name
as
well
get
over
here
paste
it,
and
that's
done
this
correctly.
This
API
is
not
mandatory,
so
this
is
optional.
B
C
B
In
this
case,
we
called
it
API
we
can
drop
this.
We
can
keep
this,
let's
keep
it.
We
will
change
this
to
the
hostname
and
then
your
API
key
the
site
and
so
on.
So
after
you
create
this
exporter,
you
would
come
here
make
this
pipeline.
The
metrics
pipeline
duplicate
this
here
and
here
as
well.
You
have
to
rename
it
like
export
to
data
doc.
B
He
said
that
we
are
happy
with
the
receiver.
Let's
say
we
want
the
full
data
set.
We
drop
this.
Let's
not
drop
the
labels
as
well.
Let's
say
in
this
case
you
don't
need
our
custom
preset
Services,
which
is
specifically
for
our
UI,
and
you
want
to
export
to
apps
brow
paste.
So
you
want
to
export
to
data
dog
and
that's
it
so
now,
when
you
start
your
metrics
will
be
pushed
to
otlp
to
our
metrics
Gateway
and,
in
addition,
tools
would
go
to
data
dog
as
you
configure
the
exporter
above
here.
B
So
in
case,
if
you're
a
groomers
user,
we
made
sure
to
expose
all
of
these
options
of
creating
additional
pipelines
and
additional
receivers.
So
you
would
prepare
all
of
those
within
a
Helm
values
file
and
then
you
could
like
specify
and
edit
another
exporter
which
would
be
almost
like
copy
pasting
this
into
the
location
in
that
proper
flag.
So
that's
how
we
would
use
it
in
Google
mesh.
A
B
A
B
So
right
now
we
don't
recommend
people
to
do
those
changes
within
the
config
map
like
we
don't
want
people
to
do
this
change
and
go
apply
it
because
in
the
next
upgrade
of
glumash
they
would
revert
these
changes.
They
would
be
overrated.
So
what
we
recommend
is
to
integrate
this
into
the
gumash
installation.
A
A
B
Yeah
so
now
we
solve
this
process
like
this
was
the
first
part
like
the
the
walkthrough
of
how
the
metrics
go
from
the
workload.
They
are
screened
either
from
a
tail
scraper.
They
are
pushed
here
and
then
they
are
scraped
by
Prometheus
and
those
are
kept
between
Prometheus
and
visualized
in
our
dashboard.
So
the
next
thing
that
we
can
check
and
that
we
want
to
check
is
like
we
as
well
solve
the
flexibility,
which
was
quite
nice
with
a
a
use
case.
What,
if
you
want
to
add
more
exporters?
That's
how
you
do
it.
B
B
So
let's
go
back
to
Prometheus
and
let's
see
like
how
much
we
were
able
to
handle.
Oh
I
think
it's
here
so
and
I
see.
I
will
try
to
be
as
fast
as
I
can.
So
if
it's
sum
of
the
history
requests
metrics
like
a
total
of
capacity
card,
we
all
got
quite
a
huge
number,
which
doesn't
mean
much
because
this
could
have
been
running
for
years.
So,
let's
just
find
the
rate
per
second.
So
to
do
so,
let's
get
the
rate
and
get
the
service
man
it's
so
this
is
like
335.
B
B
B
Which
are
even
more
intensive
because
you
have
more
buckets
and
then
the
data
points
fall
into
more
metrics
and
as
you
see
this
would
like
fail.
It
would
take
more
time
to
Resource.
So,
let's
switch
over
to
the
table
format
that
we
get
just
one
value
about
the
number
of
metrics
that
we
are
handling
every
second.
B
So
this
is
like.
Fourth
well,
let's,
let's
check
this
step
by
step
like
it's
four
million
four
comma,
seven
million
of
metrics
data
points
of
metrics
through
these
through
the
buckets,
so
that's
the
amount
that
it
can
handle,
and
these
are
not
the
only
metrics
we
saw
or
way
more
metrics
than
our
metrics
for
auto
Matrix
for
the
istio
control,
plane
and
so
on.
But
these
are
the
key
you
wants.
B
So
what
we
saw
is,
let's
open
a
calculator.
We
had
335
000
requests
per
second,
we
multiply
this
for
60
60
seconds
and
then
we
60
minutes
and
then
24
hours,
and
this
is
the
the
number
which
is
quite
difficult
to
read-
that
we
handled
for
a
day
like
4
million
and
then
eight
billion
28
billion
wow
requests
that
you
would
get.
B
Another
very
important
thing
to
check
is:
let's
keep
this
in
mind
is
how
many
workloads
we
can
support
like
for
that.
We
can
count
all
the
different
destinations
count
count
so
list
your
request,
total
by
destination
workload
ID,
and
we
see
that
we
are
currently
pulling
the
Matrix
for
3758
workloads.
Let's
delete
this
divide
this
by
12,
because
we
have
three
clusters
each
with
four
nodes.
So
it's
about
310
workloads
per
node,
which
is
more
than
Google
cloud
and
Azure
support
in
kubernetes,
like
you
have
a
limit
of
around
250
Bots
per
node.
B
This
way
we
ensure
that
we
are
supporting
any
number
of
workloads
within
a
node,
a
neuron
that
demon
set
so
forth
for
the
auto
collector,
it's
very
easy
to
support
all
the
workloads
that
you
need
within
a
cluster.
In
our
case,
we
were
deploying
less
workloads,
but
we
were
mocking
those
and
generating
more
traffic
for
those
workloads
which
were
mimicking
real
rest,
your
workloads,
and
with
that
we
can
go
back
to
the
presentation
like
let's
fill
quickly,
some
data
ends,
so
we
with
one
hotel
replica.
B
We
were
handling
310
workloads
and
then,
and
even
more
than
that,
and
then
for
requests
visuals
across
all
clusters,
not
just
for
one
replica.
You
are
able
to
handle
167
000
requests
per
second,
considering
you
have
a
two
hops
when
a
request
comes
into
your
cluster
so
and
what's
important,
and
what
puts
these
into
perspective
is
how
much
CPU
and
memory
do.
Does
one
single
Auto
replica
use
like
if
you
if
it
uses
three
CPUs,
that's
already
too
much,
because
this
is
replicating
in
every
node?
And
that's
that's
a
lot.
B
Now,
under
close,
so
here
we
go
to
pods,
switch
to
the
glue
mesh
namespace
and
then
let's
select
one
of
the
replicas
in
this
cluster.
So
it's
a
little
over
300
Milli
cores
per
second.
This
is
the
CPU
usage.
Now,
if
you
scroll
down,
you
will
see
that
in
the
memory
side,
we
are
using
approximately
one
gigabyte
of
memory
which
considered
the
amount
of
load
we
are
applying
for
every
replica.
B
That's
quite
a
lot,
so
it's
like
300
anti-millicores
and
it's
one
gigabyte
of
memory
and
as
you
see,
there
is
a
lot
of
room
to
scale
up
and
you
can
even
configure
higher
amounts,
but
this
ensures
us
that
it
will
be
able
to
handle
the
traffic
for
any
of
our
customers,
and
this
already
so
currently
we
just
released
it
and
it's
in
in
beta,
but
we
are
already
deploying
it.
The
field
Engineers
are
deploying
it
for
some
customers
and
they
are
happy.
A
B
Good
and
that
was
running
for
24
hours
as
well
like
it,
they
didn't,
have
any
drop,
so
it's
perfect
so
and
lastly
like-
and
this
is
probably
the
place
that
we
had
most
of
the
conversation
with
Josh
like
we
wanted
to
expose
this
in
a
way
that
is
as
simple
as
possible
from
from
glue
mesh
2.2.
You
can
very
just
specify
this
flag,
Matrix
gateway.enabled
and
set
it
to
two,
and
if
you
want
to
support
Legacy
metrics
pipeline,
you
can
either
keep
it
enabled.
If
you
want
to.
B
If
you
have
multiple
clusters-
and
you
want
to
gradual
rollout,
you
can
have
both
of
them
running
at
the
same
time
or
if
you
are
just
making
a
new
installation
or
you
don't
worry
about
some
metrics
being
dropped,
you
can
disable
the
Legacy
metrics
pipeline
in
the
agent
side.
So
in
the
data
plane
clusters
as
well,
you
would
just
specify
the
IP
of
the
metrics
Gateway
and
you
would
enable
The
Matrix
collector
and
all
of
it
would
work
out
of
the
box.
B
You
wouldn't
have
to
to
worry
much
about
it,
and
this
is
as
well
thanks
to
the
charts
that
are
provided
by
author,
the
auto
Community,
it's
all
quite
good,
and
you
can
very
easily
extend
those
to
meet
your
needs
and
with
that
these
are
the
resources
you
can
just
browse.
All
of
these
most
likely
we
will
add
those
in
the
show
notes.
B
A
That's
awesome
so
folks,
if
you
have
any
questions,
please
type
in,
if
not
work
on
your
wrap
up
the
session,
because
it's
been
a
really
good
session
for
me
personally
to
learn
open
Geometry
collector
how
to
solve
our
scalability
challenges,
and
the
demo
was
awesome
to
go
through
like
the
configurations
of
receiver,
how
to
config
the
exporter,
how
to
set
up
the
pipeline.
So
that's
really
really
cool
and
also
see
the
scalability
number
all
right,
I,
guess,
I,
don't
see
any
questions
coming,
let's
go
ahead
and
wrap
up.
A
I
would
really
love
to
thank
you
both
of
you,
renoa
and
Josh,
and
particularly
Reno
I
know
it's
the
evening
your
time.
So
thank
you
so
much
for
doing
this
live
stream
to
both
of
you.
Folks,
if
you
guys
find
this
live
stream,
interesting,
feel
free
to
reach
out
to
us
in
the
solo
community
or
go
to
our
GitHub
for
the
food
and
open
up
a
open
up
a
ticket
for
any
future
suggestion
you
may
have
and
also
give
us
a
thumbs
up.