►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenTelemetry Collector Deployment Patterns - Juraci Paixão Kröhling, Red Hat
The OpenTelemetry Collector is a versatile piece of software and can be deployed in numerous ways. In this talk, we’ll explore some of these patterns, starting with the classic “agent and collector” and landing at “scalable tail-based sampling,” talking about “multi-tenancy” along the way. We’ll see how they look in theory and practice with live demos on a real Kubernetes cluster. You’ll leave this session with knowledge about some of the essential processors for the OpenTelemetry Collector, how to employ them individually, and as part of a vaster deployment topology.
A
Hello
and
welcome
to
open
telemetry
collector
deployment
patterns.
My
name
is
judas
paschenkrolling,
I'm
a
software
engineer
at
red
hat
and
I
work
on
the
distributed
tracing
team.
I
am
a
maintainer
on
the
eager
project
and
a
collaborator
on
open
telemetry
now
for
this
conversation
here
today,
we're
talking
about
patterns
before
we
go
into
that
we're
going
to
invest
a
couple
of
minutes
talking
about
open
telemetry
and
the
open
telemetry
collector.
A
Now
those
are
the
patterns
that
we're
going
to
cover
here
today.
The
first
one
is
the
very
basic
pattern,
and
if
you
followed
a
quick
start
of
open
temperature
collector,
you
know
them
already.
The
second
pattern
is
the
normalizer
pattern
and
followed
by
a
couple
of
variants
of
a
pattern
for
deploying
open,
telemetry
collector
on
kubernetes.
A
We
talked
about
load,
balancing,
multi-cluster
and
multi-tenant
scenarios
as
well
all
right,
so
all
of
the
the
patterns
that
we
talked
about
here
today.
They
are
available
in
this
repository
here
it.
This
repository
contains
images
and
configuration
file,
examples
and
a
deeper
explanation
on
those
patterns.
A
Now
you
can
download
this
slide
deck
here,
either
from
the
conferences
website
or
from
this
box.
From
here,
and
I'm
also
sharing,
this
live
deck
right
now
on
twitter,
all
right!
So,
let's
get
started.
The
open
telemetry
is
a
project
that
was
created
with
a
fusion
of
open
tracing
and
open
sensors
it.
It
is
actually
composed
of
two
big
parts.
A
A
A
It
kind
of
hints
at
the
internal
architecture
of
the
collector
and
the
collector
is
composed
of
the
following
components:
right,
it
has
receivers,
it
has
processors,
it
has
exporters,
and
it
has
extensions
we're
not
talking
about
extensions
that
much
here
right.
So
there
are
a
special
type
of
component
that
is
not
part
of
the
pipeline
and
a
collector
is
view
as
the
pipeline
for
the
telemetry
data.
A
Now
processors
they
might
be
doing
sampling,
they
might
be
changing
attributes
or
adding
removing
updating
some
attributes
from
the
data
points
themselves.
They
might
be
doing
some
matching.
They
might
be
doing
some
routing
and
so
on
and
so
forth,
and
when
it
comes
to
exporters,
we
have
a
bunch
of
them
for
pretty
much
all
the
relevant
systems
and
vendors
out
there
all
right.
So
we
have
exporters
for
eager
zipkin
and
ultlp.
A
We
have
exporters
that
and
that
emulate
or
that
expose
a
prometheus
compatible
endpoint
so
that
other
prometheus
compatible
systems
could
I
can
scrap
data
out
of
it
out
of
the
process,
and
we
have
exporters
for
pretty
much
all
of
the
commercial
commercial
vendors
out
there
all
right
and
besides
the
collector
proper.
Besides
the
open
transfer
collector
itself,
we
have
a
a
couple
of
repositories
or
a
couple
of
projects
that
are
part
of
the
same
ecosystem
and
the
first
one
and
the
biggest
one
is
the
contrib.
A
The
contrib
is
where
all
the
non-core
components
live,
including
the
vendor-specific
ones
right.
So
if
you
want
to
use
a
very
specific
component
for
open
telemetry,
collector
you'd,
probably
download
the
contrib
distribution
first
now
you
might
notice
that
the
contrib
distribution,
the
contrib
binary,
is
actually
quite
big.
So
you
might
want
to
consider
using
the
open
transfer
collector
builder
to
pick
and
choose
which
components
you
want
as
part
of
your
own
distribution
of
the
collector
right
and
then
for
that
binary
you
have
only
the
components
that
you
need.
So
it's
very
slim.
A
It
does
it's
very.
It's
very
lightweight
all
right.
This
is
how
a
configuration
file
looks
like
for
the
collector,
so
we
have
sections
for
for
extensions
for
the
receivers,
processors,
exporters
and
we
tied
them
all
together
under
the
service
node
right.
So
we
specified
extensions
that
we
want
there
for
this
process
and
we
specified
the
pipelines.
Now.
The
pipelines
can
be
traces,
metrics
and
logs
pipelines,
and
we
might
even
have
multiple
pipelines
for
the
same
data
type.
A
So
we
can
have
multiple
trace
pipelines
here,
for
instance,
in
our
example,
we
have
only
one
receiver,
one
processor
and
one
exporter
all
right.
So
let's
get
started
with
the
patterns,
then
the
very
first
one
is
a
basic
pattern
and
in
this
case
we
have
our
application
instrumented
using
the
open,
telemetry
sdk
exporting
data
with
otlp
to
an
open
parametric
collector
located
somewhere.
A
Now
that
open
temperature
collector
then
exports
data
to
a
final
destination,
in
this
case
here
jaeger
note
that,
throughout
this
presentation,
I'm
using
here
as
an
example
here
of
the
final
destination,
but
you
can
replace
that
with
pretty
much
everything
that
you
want
right.
So
it
can
be
a
different
tracing
solution
or
it
might
be
a
specific
vendor,
but
or
it
might
be,
even
something
not
tracing
specific,
so
it
can
be
in
most
of
the
cases.
It
can
also
be
like
prometheus
in
here,
and
so
this
is
our
first
pattern.
A
I
hope
it's.
It
was
enough.
You
know
to
warm
up
and
a
second
pattern
that
we
have
is-
or
it
is
a
variant
of
the
first
pattern,
and
it
is
a
fan
out
pattern
going
back
to
our
original
image.
We
have
the
same
application
instruments
with
open
term
3
sdk,
exporting
data
with
otlp
to
open,
transfer,
collector
and
open
terms
with
collector
exports,
data
to
jaeger
and
additionally,
to
a
an
external
vendor.
A
A
Now.
The
second
pattern
that
we
have
today
is
the
normalizer
pattern,
and
in
this
case
here
we
might
probably
have
a
promises,
client
instrumenting,
our
application
for
for
metrics,
and
we
might
have
our
application
instrumental
using
open
tracing
for
the
traces
with
the
jager
client
as
the
actual
tracer
now
data
is
then
either
made
available
to
prometheus
or
sent
to
jager,
and
in
this
pattern
here
we
are
using
the
collector
as
a
drop
in
replacement
for
eager
and
prometheus.
A
A
A
It
has
the
ability
of
looking
into
all
the
data
points
that
are
flowing
through
this
through
this
pipeline
here
or
through
this
communication
channel
and
ensuring
that
they
all
have
the
same
set
of
basic
labels
right.
So
let's
say
that
we
want
the
all
data
points
to
contain
the
cluster
name.
They
they
originated
at.
So
we
can
ensure
that
the
collector
has
a
couple
of
processors,
adding
a
collector
label
to
metrics
and
two
all
right
now.
Our
third
pattern
here
is
a
couple
of
patterns
actually
on
to
deploy
on
kubernetes.
A
The
first
one
is
using
a
sidecar,
so
a
sidecar
on
kubernetes
is
basically
a
a
second
container
as
part
of
your
pod.
So
you
have
your
application
pod
with
one
container
in
it,
and
you
add
a
second
container
with
the
open
telemetry
collector
now
that
open
telemetry
collector
acts
as
an
agent,
and
it
sends
data
then
to
an
external
collector,
possibly
deployed
on
on
another
namespace
observability
here
in
this
case,
and
from
that
collector,
we
export
data
together.
A
Now
there
are
a
few
advantages
to
this
kind
of
approach
here.
The
first
one
is:
if
we
decide
to
change
the
way
that
we
send
data
to
eager
or
if
we
change
our
yield
location
or
if
we
decide
to
not
use
jaeger
anymore
or
if
we
decide
to
use
a
another
exporter
in
addition
to
jager,
then
we
can
just
change
this
one
deployment
here
in
our
observability
namespace.
We
don't
have
to
change
any
of
the
sidecars
now.
A
So
what
we
the
advantage
here,
is
we
get
better
client-side
cloud
balancing
when
we
have
multiple
instances
of
the
clients
making
a
connection
to
one
server
right.
So
we,
when
we
scale
up
the
number
of
instances
or
when
we
scale
up
our
open
function,
collector
on
the
observability
namespace,
then
all
the
new
collectors
or
all
the
new
instances
of
the
collectors
on
the
workload
namespaces.
A
They
would
then
be
using
those
new
instances.
So
it's
very
likely
that
the
load
balancing
would
work
better
when
you
have
more
clients
than
fewer
clients.
Now
a
another
advantage
of
having
side
cars
on
a
per
application
basis.
Only
per
pod
basis-
or
you
know,
per
deployment-
is
that
we
can.
We
can
fine
tune
the
configuration
for
that
collector
to
the
necessities
of
that
application
of
that
deployment
right.
A
So
if
we
have
a
critical
application
here,
if
we
have
a
critical
deployment,
we
can
have
a
very
specific
configuration
for
the
collector,
perhaps
with
a
more
you
know,
more
resilient,
more
retry
mechanisms
and
and
perhaps
even
more
memory,
more
cpu
allocation
for
that
specific
process.
Whereas
for
lower
criticality
services,
we
can
have
a
lightweighter
configuration
for
for
that
agent
for
that
sidecar.
A
A
The
the
second
variant
of
this
pattern
is
not
using
a
side
car
but
using
a
daemon
set,
so
the
advantages
are
very
similar,
so
we
have
a
a
collector
that
is
very
close
to
our
application
to
our
workload,
making
it
easy
for
the
application
to
offload
data
very
quickly
to
the
sidecar
or
sorry
to
the
agent
in
this
case
here
as
in
the
demo
side.
But
we
don't
have
many
of
the
advantages
of
the
sidecars.
For
instance,
we
don't.
A
It
is
harder
to
do
much
tendency
in
this
kind
of
scenario
right,
because
we
have
multiple
namespaces
running
on
the
same
node
and
we
have
only
one
collector
running
on
that
node.
It
means
that
that
collector
is
going
to
see
data
from
all
the
tenants
in
there.
So
it's
harder
for
us
to
manage
a
tendency
at
that
level,
not
impossible,
of
course,
but
harder.
A
A
Now
the
biggest
advantage
of
demon
sets
over
side.
Cars
is,
as
you
can
imagine,
the
the
overhead
right
so
open
times
require
itself
is
not
big
or
it
doesn't
have
to
be
that
big
in
here.
So
as
a
side
car,
I
would
say
that
the
memory
consumption
of
the
collector
itself
should
be
around
five
to
ten
megabytes,
but
it
does
add
up
when
you
have
hundreds
of
of
those
instances
right
when,
when
we
have
as
a
demo
set,
we
only
have
one
collector
per
node,
so
we
hit
our
overhead
years
is
slower.
A
Now
the
idea
is
that
each
one
of
those
collectors
within
collect
data
from
the
applications
running
on
those
nodes,
so
it's
local
to
an
application
and
resiliently
securely
and
safely
send
data
to
a
central
collector
all
right
now.
Our
next
pattern
is
about
load
balancing
and
to
explain
a
little
bit
about
why
you
need
load
balancing
at
the
collector
level
and
instead
of
you
know,
just
a
regular,
jrpc
or
http
load
balancer.
We
have
to
go
back
a
little
bit
and
understand
how
tracing
actually
works
right.
A
The
way
that
tracing
works
is
not
that
we're
gonna
wait
for
all
the
services
to
complete
and
then
service
a
sends
data
to
a
tracing
backend.
It's
not
like
that
right.
The
way
that
it
works
is
service,
a
is
responsible
for
collecting
and
sending
data
for
related
to
its
own
operations.
All
right,
so
only
the
expands
belonging
to
service
a
are
going
to
be
sent
from
the
service
a
to
a
collector
somewhere.
A
A
The
idea
that
we
have
here
is
that
each
collector
should
have
a
complete
view
of
the
trace
so
that
it
can
make
a
decision
based
on
the
base
on
that
on
that
trace.
So,
for
instance,
if
we're
doing
tail-based
sampling,
we
want
to
take
a
look
at
the
whole
trace
and
make
a
decision
whether
we
want
to
sample
or
not,
or
perhaps
we
are
doing
some
some
some
analytics
on
the
tracing
data.
A
So
we
want
to
take
a
look
at
the
whole
trace,
perhaps
compress
it
in
some
way,
perhaps
just
extract
some
metrics
and
discard
the
trace
itself
and
anything
things
like
that.
Now
to
do
that,
we
need
to
ensure
that
all
the
traces,
all
the
spans
for
the
same
trace
are
at
the
same
collector.
What
we
do
not
want
to
have
is
spent
for
the
same
trace
at
different
collectors.
A
Now
we
ensure
that
by
having
two
layers
of
open-source
collectors,
so
the
first
layer
is
a
is
a
load
balancer
layer
with
it
is
a
basic
open,
telemetry
collector
with
the
load
balancing
exporter.
Now
this
load
balancing
exporter
will
split
the
batches
that
it
receives
from
the
clients.
It
will
look
at
each
single
span
in
it.
It
will
take
the
extract,
the
trace,
id
hash
it
and
determine
which
collector
should
be
receiving
this
data
point,
and
it
sends
data
to
that
data.
A
So
you
can
have
a
ha-like
deployment
of
the
load
balancer
with
the
three
replicas,
for
instance,
where
and
at
the
same
time
having
hundreds
of
instances
of
the
collector
backing
that
load
balancer
right
so
as
a
final
destination
or
an
intermediary
destination
for
this
data.
Those
collectors
might
then
be
doing
they'll
be
simply
and
sending
data
to
the
final
destination
like
eager
or
your
or
your
other
tracy
system,
all
right.
A
So
the
next
partner
that
we
have
is
a
multi-cluster
pattern
and
in
in
this
case
here
it
is
very
similar
to
one
that
we've
seen
before
for
the
kubernetes,
either
aside
car
or
demo
site.
So
the
idea
is
that
we
have
our
application
very
close
to
our
application.
We
have
a
sidecar
or
an
agent
that
receives
this
information
from
that
application
and
sends
information
to
a
central,
collector
local
to
the
cluster.
A
Now
that
collector
then
makes
decisions
about
data
from
the
cluster
itself,
so
perhaps
it
is
adding
cluster
specific
information
to
the
data
points.
Perhaps
it
is
doing
database
sampling
at
the
cluster
level,
but
the
point
here
is
that
we
centralize
all
the
data
to
that
collector
and
that
collector
then
makes
a
very
secure
connection
to
a
collector
on
a
cluster
on
a
control
plane
cluster.
A
Now
that
communication
might
have
different
resiliency
different
reliability
requirements,
they
may
have
different
security
requirements,
so
it
makes
sense
to
have
to
to
to
extract
that
knowledge
or
that
that
logic
into
one
collector
that
that
works
at
the
boundary
and
then
on
the
other
side
on
the
control
plane
cluster.
We
don't
have
this
collector,
that
is
receiving
data
from
other
workloads.
So,
of
course
we're
not
talking
about
one
cluster.
Only
we're
talking
about
multiple
clusters,
sending
data
to
a
central
control,
plane
cluster.
A
A
A
A
We
should
have
one
central
location
for
data
to
arrive
at
so
our
open
function
collector.
That
is
then
taking
care
of
some
of
some
logic.
That
applies
to
all
the
tenants,
for
instance,
in
in
doing
security
and
doing
data
cleanup.
Perhaps
we
are
removing
some
personally
identifiable
information
from
these
pens.
Perhaps
we
are
adding
some
other
information
to
those
pens
and
sending
data
to
the
final
destination.
A
So
in
this
case
here
the
two
different
eagers
one
for
each
tenant
and
then
we
can
charge
each
one
of
those
tenants
based
on
their
usage
and
the
open
challenge
collector
central
here.
It
could
then
be
owned
by
ita
itself
all
right.
We
have
a
bonus
pattern
here
and
that
is
a
per
signal
deployment
of
open,
telemetry
collector.
A
In
this
case
here
we
we
have
one
collector
taking
care
of
the
metrics
part
and
one
collector
here
taking
care
of
the
tracing
parts.
Now
there
are
some
reasons
why
we
would
want
to
do
that
and
the
first
one
is.
It
is
the
way
that
we
scale
an
open
country
collector
doing
scraping
is
different
than
the
way
that
we
scale
as
in
a
in
a
push
based
model
right.
So
when
we
have
a
push
based
model,
we
can
just
scale
the
number
we
scale
up.
A
The
number
of
replicas
that
we
have
and
the
new
clients
would
then
just
find
new
new
backends
and
send
data
directly
to
them.
Now,
when
we
are
talking
about
pulling
data
out-
or
you
know
in
a
typical
scraping
mechanism
from
prometheus,
we
cannot
just
increase
the
number
of
replicas
for
the
for
the
collector,
because
you
know
the
more
replicas
that
we
have
the
more
scrapers
we
have
and
they're
all
going
to
the
same
targets.
A
So
we
might
want
to
have
a
different,
a
different
instance
of
the
collector
taking
care
of
metrics
than
the
one
taking
care
of
the
of
the
tracy
data
points.
Another
reason
to
have
it
displayed
per
signal
is
that
right
now,
the
the
maturity
of
each
one
of
those
signals
are
not
the
same
right.
So
now,
tracing
components
are
very
mature,
whereas
the
matrix
components
are
not
that
mature
right,
so
we
might
want
to
to
mitigate
a
risk
we
might
want
to
get.
You
know
all
the
traces
on
a
highly
available
scenario.
A
On
a
you
know:
production
quality
deployment
for
the
tracing
pipeline
and
for
the
metrics
pipeline.
We
have
something
that
you
know
it
is
more
fragile,
fragile
than
the
tracy
one,
all
right.
So
that's
wha.
Those
are
the
patterns
that
I
had
for
today
and
I
think
there
are
key
takeaways
here
right.
The
first
one
is
the
hope
and
telemetry
collector
is
very
versatile
right.
So
it's
it's.
It's
incredible.
A
How
many
things
we
can
accomplish
with
the
collector-
and
I
think
the
key
here
is
to
understand
and
getting
to
know
the
existing
components.
So
once
you
know
which
components
you
have
you
can
start
planning,
you
can
start
drawing
and
architecting
how
you're
going
to
deploy
a
open,
timesheet
collector.
A
A
So
we
have
side
cars
talking
to
other
collectors
as
local,
in
local
genetic
cluster,
talking
to
collectors
on
a
control,
plane,
cluster
and
so
on,
but
they're
all
the
same
binaries
they're
just
configured
differently
and
finally
mix
and
match
components
and
collector
instances
right
so
plan
your
deployment
and
and
mix
and
match.
And
if
you
you
don't
actually
have
to
use
the
same
binary,
you
can
build
your
own
binaries,
depending
on
the
needs
that
you
have
at
each
of
those
levels,
all
right.
A
So
those
are
some
of
the
resources
that
you
can
use
to
continue
from
this
conversation
here.
So
some
information
about
open
country,
the
collector,
the
location
here
for
the
contrib
repository
computer
and
the
patterns
for
from
this
presentation
here,
if
you've
used
the
open
transfer
collector
before
I'm
quite
sure
that
you
have
your
own
patterns.
So
here's
my
actual
item
to
you
go
to
this
repository
fork
it
right
now
and
include
your
own
patterns.