►
From YouTube: Observability at Scale: Running OpenTelemetry Across an Enterprise - Jonah Back & Kranti Vikram
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Observability at Scale: Running OpenTelemetry Across an Enterprise - Jonah Back & Kranti Vikram, Intuit
Observability has been a huge topic of interest in the software industry over the last few years. One of the major components in observability is distributed tracing. Tools like Jaeger, Zipkin, OpenCensus, and OpenTelemetry have made it really easy to get started. Less easy, however, is getting your tracing infrastructure to a place where it can be fully leveraged across hundreds, if not thousands, of services. This talk will cover Intuit's experience deploying tracing infrastructure using Kubernetes, Jaeger, and OpenTelemetry. It will cover a few key areas in Intuit's journey to running a highly available, multi-region tracing solution. 1) Scaling ElasticSearch to support 500M+ traces per day. 2) Secure, automated on-boarding of OpenTelemetry agents to central collectors 3) Leveraging open-source libraries to provide high quality trace data, enhanced with domain-specific attributes
https://sched.co/Zeqy
A
A
The
problems
that
our
engineers
and
our
on-call
engineers
were
facing,
the
solutions
that
they
were
using
and
really
how
adopting
tools
like
open,
telemetry,
jaeger
and
other
vendored
tools
has
really
enabled
intuit
to
solve
some
of
those
problems
from
there
we'll
go
into
collection
and
instrumentation
of
trace
data.
So
we'll
talk
about
how
we've
managed
to
collect
all
of
the
data
from
the
services
running
it
into
it
and
how
we
instrumented
those
services
in
a
pretty
easy
way.
A
So
that
they
could
collect
trace
data,
then
we'll
talk
about
some
of
the
processors
we've
built
on
top
of
open
telemetry,
specifically
how
we
are
able
to
extract
metrics
from
our
trace
data
and,
finally,
we'll
talk
about
tracing's
role
in
reducing
mttr
and
mgtd.
That's
mean
time
to
repair
and
mean
time
to
detect
for
our
services.
A
Okay,
so
first,
let's
look
at
kind
of
the
state
into
it
within
18
to
24
months
ago,
we
were
in
a
state
18
to
24
months
ago,
where
not
too
far
from
where
we
are
today,
but
we
had
services
running
in
a
huge
polygon
environment.
You
know
we
had
teams
running
services
in
kubernetes.
In
aws
we
had
teams
running
just
on
bare
metal,
ec2
instances.
A
We
had
teams
running
serverless
functions,
we
had
teams
running
in
an
on-prem
data
center
and
we
had
a
variety
of
tools
that
these
services
and
teams
were
interacting
with
you
know
we
had
an
istio
service
mesh.
We
had
dynamodb
resources,
rds
resources,
really
a
really
truly
polyglot
stack
and
our
engineers.
Well,
while
this
gave
them
a
lot
of
flexibility,
we
started
to
notice
some
pretty
common
problems
across
teams.
A
One
of
the
biggest
ones
being
that
when
teams
were
talking
and
communicating
with
other
services,
troubleshooting
both
latency
issues
and
air
issues
became
really
difficult.
You
know
we
had
several
incidents
where
teams
would
jump
on
a
call,
and
it
was
basically
like
domino's
trying
to
page
the
right
on-call
team
team.
One
would
notice
latency,
they
would
get.
You
know
on
onto
some
kind
of
virtual
call
and
then
they
would
realize.
Oh
no,
it's
team
two
and
then
we
would
page
team
two
and
then
pinch
would
be
like.
A
Oh
no,
it's
hit
team
three
and
we
just
keep
paging
teams
down
the
stack
until
we
realized
where
the
root
cause
was
because
teams
didn't
have
visibility,
past
their
own
services
and
the
reason
they
didn't
have.
This
visibility
is
because
we
really
didn't
have
the
tool
chain
to
actually
identify
what
these
dependencies
were
and
how
far
they
went.
We
had
a
very
basic
tracing
mechanism.
We
had
a
a
header
called
intuit
underscore
to
yeti,
which
was
intuitive,
which
services
generally
propagated,
not
100
coverage.
A
But
you
know
they
were
pretty
good
about
propagating,
but
this
header
really
didn't
tell
us
anything
about
the
relationships
between
services
other
than
that
they
both
had
a
common
api
call.
You
know
the
only
way
you
could
use
this
header
was
searching
in
our
log
storage
and
even
then
we
had
you
had
to
search
across
three
or
four
different
log
indexes
to
actually
be
able
to
use
that
data
and
that
data
still
wasn't.
A
You
know
it
didn't
solve
for
problems
like
asynchronous
communication
over
a
kafka
broker
or
I
didn't
solve
for
clock
ski
between
services
and
it
didn't
have
information
about
like
parent-child
relationships.
So
it
was
really
really
hard
to
use
that
header
to
actually
understand
what
dependencies
between
services
were.
A
You
know
this
diagram
really
just
shows
kind
of
that
that
state
where
we
have
all
of
these
different
application
instances.
You
know
maybe
central
services
or
services
specific
to
like
our
quickbooks
platform,
all
pushing
data
to
different
log,
indexer
and
search
tools.
You
know
so
the
shared
services
would
have
their
own
instance
of
a
log
indexer.
The
quickbooks
services
would
have
their
own
instances
of
a
log
indexer
and
these
instances
weren't
talking
to
each
other
and
it's
the
same
story
for
like
apm
tools.
A
So
you
know
intuit
leverages
a
tool
called
appdynamics
and
it
was
sort
of
the
same
story
where
these
some
of
our
services
were
pushed
to.
One
instance
on
the
app
dynamics,
task
platform
and
other
services
would
push
through
a
second
instance,
and
these
instances
couldn't
talk
to
each
other.
So
none
of
this
correlation
data
between
services
was
actually
visible
in
a
core
like
single
place,
developers
had
to
go
through
multiple
tools.
You
know
multiple
log
instances,
multiple
apm
tools
really
to
get
a
sense
of
the
the
system.
Why
health?
A
They
couldn't
just
go
into
one
place
and
say
hey,
so
this
request
this.
These
are
all
the
services
involved.
These
are
the
latency
times
that
each
of
them
spent,
and
it
was
a
real
problem
right
engineers
couldn't
troubleshoot
issues
in
just
a
minute
or
two.
It
often
would
take
20
30
minutes
just
to
understand
how
one
certain
api
or
request
was
working
across
the
system
and
some
other
problems.
A
That
kind
of
you
know
in
the
same
area
were
that
we
didn't
have
a
standard
trace
log
format
that
intuit
tid
was
really
the
only
thing
that
was
powering
it
and
nobody
interrupted
it.
You
know
app
dynamics,
didn't
understand
it
new,
relic,
wouldn't
understand.
None
of
these
solutions
would
go
out
there
and
understand
what
that
header
meant.
A
We
didn't
have
a
single
tracing
solution
across
into
it.
Some
teams
had
looked
at
different
standards.
You
know
within
small
groups,
teams
might
be
using
something
like
a
zipkin
to
actually
trace
the
requests
across
their
three
or
four
services,
but
across
the
company
there
is
no
single
place
that
you
could
go
and
actually
visualize
a
trace
across
all
intuit
services
involved
in
the
request,
which
was
a
huge
problem,
and
so
you
know
with
these
problems,
we
had
a.
We
had
a
team
get
together
and
really
think
about
like
what
is
the
right
solution.
A
We
we
looked
at
the
industry
where
they
were
at.
There
were
a
couple
standards
at
the
time
that
were
competing.
There
was
open
tracing.
There
was
open
census,
both
in
the
distributed
tracing
space,
both
had
their
own
standards
in
terms
of
how
to
propagate
data,
and
we
we
looked
at
first
attempting
to
build
our
own
tracing
library.
That
kind
of
leverage
some
of
those
standards
to
propagate
information.
A
Now,
before
we
get
into
that,
though,
I
want
to
talk
about
the
role
of
observability
in
reducing
those
outages
that
we
were
talking
about
or
those
incidents
that
we
were
talking
about.
Intuit,
you
know,
as
a
sas
provider,
has
a
pretty
high
standard
for
how
much
time
that
we
want
to
spend
during
an
incident
down,
obviously
as
as
little
as
possible,
but
in
this
case
we
have
a
budget
max
across
all
services
of
around
22
minutes
per
month
of
downtime.
A
At
most-
and
you
know,
one
of
the
things
to
highlight
here
is
that
that
budget
isn't
just
for
time
to
repair
things,
but
it
also
includes
time
waiting
for,
like
time
to
detect
as
well
waiting
for
alerts
to
go
off
waiting
for
the
on-call
to
join
an
incident
call
and
page
other
people.
So
when
we
think
about
it,
at
22
minutes
is
really
almost
no
time
at
all,
and
so,
if
we
can
bring
that
any
tooling
that
we
can
get
that
can
help
us
improve
the
time
to
both
detect
and
time
to
recover.
On.
A
Okay,
quick
highlight
of
some
of
the
terminology,
I'm
mentioning
here
in
terms
of
how
distributed
tracing
systems
generally
work.
There's
two
key
concepts
to
understand
here:
there's
the
concept
of
a
trace,
so
the
trace
in
this
diagram
is
every
basically
everything
under
this
a
block
and
including
this
a
block
and
all
of
those
individual
objects
are
called
spans
and
those
fans
are
usually
just
single
operations
in
that
are
part
of
a
trace.
A
All
of
those
would
be
individual
spans
that
ladder
up
to
serving
that
api
call,
and
so
the
way
that
these
tools
are
actually
and
these
models
are
actually
used
is
that
you
know
you
have
a
request
come
into
a
microservice.
That
request
is
going
to
have
some
kind
of
trace
data
on
it.
You
know
maybe
a
trace
header,
or
maybe
nothing
at
all,
that
microservice
when
it
calls
other
services
or
does
other
operations
is
going
to
create.
A
Those
new
span
objects
that
we
talked
about
and
those
span
objects
are
going
to
interact
with
some
kind
of
local
tracing
api.
So
in
our
case
it's
an
open
tracing
api
in
the
jvm.
Typically,
that
tracing
api
is
going
to
be
configured
to
talk
to
some
tracing
library
or
tracing
collector,
and
then
that
tracing
collector
is
going
to
send
data
to
a
trace
storage
layer,
and
this
is
generally
how
you're
going
to
see
most
tracing
architectures
work
at
a
high
level.
Is
that
they're
going
to
have
some
kind
of
instrumentation
on
your
app
servers?
A
A
We
have
services
that
run
on
kubernetes,
so
they,
you
know,
they'll
have
multiple
pods
running
at
kubernetes
cluster
or
we
have
services
running
on
just
pure
ec2
on
aws,
so
they'll,
you
know,
they'll
have
a
ami
that
they
use,
they
will
start
their
server
and
they
might
have
one
or
several
daemons
running
on
that,
and
so
for
both
of
those
cases.
Today
we
run
an
open
telemetry
agent,
which
is
just
you
know,
basically,
a
configured
container
using
the
open
source,
open
telemetry
agent
container
that
has
configuration
to
send
to
a
aws
private
link
endpoint.
A
Endpoint
that's
automatically
configured
in
that
account
to
talk
to
a
central,
collector
tier,
and
it's
the
same
thing
on
our
ec2
instances.
Today,
when
you
provision
a
new
ec2
instance
using
the
intuit
base,
ami
there's
a
daemon
already
pre-configured,
you
just
have
to
run
like
service.start
on
that
daemon
and
that
daemon
is
basically
going
to
auto.
Discover
that
aws
privately
gun
point
and
again
it's
going
to
use
grpc
to
communicate
with
from
the
agent
to
the
open,
telemetry
collector
node.
A
Now
those
collector
nodes
are
typically
running
in
the
same
aws
region.
So
if
you're
running
in
the
us
west
two
region,
there
will
be
a
collector
tier
in
us
west
too,
to
receive
this.
That
trace
data,
and
the
same
goes
for
any
other
region
that
into
it
operates
in
so
any
major
aws
region.
We
have
a
collector
tier
set
up
to
kind
of
collect
all
of
that
data
and
from
there
that
open
telemetry,
collector
tier
uses,
grpc
and
envoy
to
basically
push
that
data
to
a
jager
collector
tier.
A
Initially
it
was
a
separate
tier,
but
the
jaeger
team
has
actually
been
working
to
consolidate
a
lot
of
that
into
the
open
telemetry.
You
know
community,
as
well
in
terms
of
extra
receivers,
extra
processors,
extra
collector
extra
exporters,
and
so
that's
actually
really
becoming
a
consolidated
tier
today
and
then
from
there.
B
Is
the
dependent
service
throwing
an
error,
or
is
it
the
high
latency
of
the
dependent
service?
That's
causing
this
repercussion?
Typically
in
such
cases,
what
do
we
do?
We
take
the
time
window
and
dig
through
the
logs
to
find
something
useful.
We
also
pull
in
teams
from
different
services
to
troubleshoot
this
issue.
This
process
is
very
time
consuming
involves
many
people,
and
often
we
don't
find
the
root
cause
of
the
issue.
So
whom
do
we
blame
if
you
don't
find
the
root.
B
B
B
A
trace
context
gets
created
and
also
generates
a
server
span
which
would
have
trace
id
to
identify
a
distributed
transaction.
A
span
id
to
identify
the
transaction
flow
within
the
order
service.
A
parent
span
id
to
identify
the
span
that
initiated
this
transaction
since
order
service
is
a
service
or
the
first
service
receiving
the
request.
B
B
B
Let's
look
at
metrics,
there
are
a
lot
of
different
types
of
metrics
that
you
can
capture
for
your
system,
starting
with
system
resource
metrics,
cpu
memory
for
your
node.
There
are
a
bunch
of
aws
resource,
metrics
cloud
watch,
metrics
coming
from
elasticsearch,
dynamodb,
rds,
serverless,
so
on
and
so
forth,
and
we
also
have
transaction
metrics.
These
are
the
metrics
that
give
you
insights
on
the
api's
request
rate,
error
rate
duration
or
at
the
service
level
as
well.
So
these
are
the
transaction
metrics.