►
From YouTube: Lightning Talk: OTel Me How to Build a Data Pipeline for Observability - Daniel Kim & Reese Lee
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: OTel Me How to Build a Data Pipeline for Observability - Daniel Kim & Reese Lee, New Relic
Observability boils down to a data problem. To gain observability, we need to collect signals from many different data sources, frameworks and programming languages. It is important to turn heterogeneous data across so many different signals into actionable insights through a framework like OpenTelemetry, an open source standard enabling us to collect, process, and export telemetry data. In this session, we will discuss the complexities that come with handling telemetry data from distributed systems as they scale. Next, we will show you how to build an observability data pipeline using the OpenTelemetry collector and open source plugins that can perform a wide variety of data processing, such as injecting infrastructure metadata into traces and metrics, implementing tail-based sampling, and exporting data to any backend via OTLP. Finally, we will share some of the challenges with running the collector, so that you can take them into account as you build out your own observability pipeline with OpenTelemetry.
A
Okay,
hello,
everyone,
my
name
is
Daniel
and
I'm
with
my
friend,
Reese
and
I'm
excited
to
bring
this
talk.
Oh
tell
me
how
to
build
a
data
pipeline
for
observability
before
we
get
started,
I'm
going
to
tell
you
a
little
bit
about
me.
I
made
developer
relations
engineer
at
New,
Relic
I'm,
really
passionate
about
getting
developers,
better
visibility
into
their
systems
and
yeah.
Who
are
you
Rhys?
My.
B
A
So,
let's
just
scooter
right
into
our
presentation.
So
when
you're
starting
to
develop
your
applications,
you're,
probably
not
worried
about
the
amount
of
observability
data
that
you're
generating
right.
You
start
with
a
couple
thousand
metrics
a
couple
hundred
traces,
a
couple
thousand
logs:
it's
not
a
huge
deal
but
Azure
application
scales.
A
It
becomes
to
become
a
large
problem
and
that's
because
doing
anything
with
data
costs
a
lot
of
money,
whether
you're
processing
it
storing
it
moving
it,
and
that
could
go
to
a
particular
vendor
to
do
that,
for
you
or
you
could
be
paying
a
team
of
Engineers
to
maintain
your
Prometheus
grafana
stack
whatever
ever
you
decide
it's
going
to
cost
you
money.
So
how
do
we
make
sure
that
you
get
the
most
bang
out
for
your
buck?
A
So
here's
a
couple
things
that
we
at
New
Relic
think
that
you
can
do
to
make
sure
that
you
can
increase
the
ROI
of
your
observability
setup.
The
first
thing
you
should
do
is
have
a
centralized
data
pipeline
for
all
of
your
observability
data,
and
this
is
because,
as
you
have
more
and
more
complex
Cloud
environments,
you
might
have
multiple
sources
of
observability
data.
A
The
second
thing
you
do
is
increase
the
usefulness
of
the
data.
How
can
we
make
sure
that
the
data
that
we
collect
has
all
the
context
as
well
as
the
information
we
need
to
make
sure
it's
as
useful
as
possible?
The
third
thing
you
can
do
is
control
the
amount
of
data
that
you're
exporting
to
your
back
ends,
because
a
lot
of
the
data
that
is
generated
by
our
systems,
whether
it's
through
logs
or
metrics,
are
kind
of
never
used.
A
So
how
do
we
make
sure
that
we're
really
cognizant
and
mindful
of
what
we
export
to
our
backends
so
open
Telemetry
has
an
awesome
solution
for
this
and
I
think
our
previous
slightly
talked
a
little
bit
about
the
open
Telemetry
collector.
So
the
Open
Country
collector
is
an
open
source
data
pipeline
for
your
observability
data.
So
it's
made
out
of
three
different
parts.
A
The
first
part
is
the
receiver,
so
the
this
is
this
part
of
the
system
kind
of
collects
all
of
your
data,
whether
it's
through
open,
Telemetry,
sdks
or
Frameworks,
or
it's
through,
like
other
observability
sources,
whether
it's
like
Prometheus
or
Loom,
bit
and
receivers
kind
of
connect.
All
of
that
data
into
the
pipeline,
then,
with
the
processor
you're
able
to
take
all
that
data
that
you
collect
and
decorate
it
you're
able
to
batch
it
you're
able
to
do
a
bunch
of
things
to
make
it
more
valuable.
A
As
well
as
to
reduce
the
overall
amount
of
data
that
you're
trying
to
export
and
finally
with
the
exporter,
you're
able
to
take
all
the
data
that
you
collect
and
process
and
send
it
to
whatever
backend
you
want,
whether
it's
a
managed
backend
or
a
open
source
backend.
So
with
the
open
Telemetry
collector,
how
can
we
have
a
centralized
data
pipeline?
So
let's
jump
right
into
it
so
because
open
Telemetry
is
a
Community
Driven
project.
A
There
are
so
many
open
source
receivers
that
you
can
leverage
out
of
the
box,
and
so,
if
you're,
already
running
Prometheus
or
fluent
bit
in
your
kubernetes
environment
or
your
Cloud
native
environment,
you
don't
have
to
just
kind
of
turn
those
down
and
then
leverage
all
open
Telemetry.
You
can
still
connect
those
to
your
open,
Telemetry,
collector
data
pipeline.
A
So
an
example
of
doing
this
is
collecting
infrastructure
data.
So
if
you
already
are
running
Prometheus
and
in
your
kubernetes
environment,
you
can
directly
connect
Prometheus
using
the
Prometheus
receiver
to
the
open
Summit
collector.
So
you
can
continue
using
Prometheus
data
that
your
kubernetes
clusters
are
generating
in
their
endpoints,
but
just
make
sure
that
you
stream
the
data
through
an
open
time
to
collector,
and
if
you
leverage
platforms
like
AWS
or
Azure,
they
have
also
written
receivers
specifically
for
their
various
services
that
connect
directly
to
the
open
television
collector.
A
So
recently,
I
actually
had
to
instrument
Kafka
environments
for
a
particular
customer
to
get
metrics,
and
there
were
three
different
ways
that
we
could
pull
data
specifically
metrics
from
Kafka
clusters
and
here's
a
couple
of
ways
that
you
can
do
it
using
the
open
energy
collector.
So
the
first
way
is
through
the
Kafka
metrics
receiver.
So
the
Kafka
Matrix
receiver
leverages
the
admin
API
to
pull
data
directly
from
your
Kafka
server,
to
get
very
specific,
like
partition,
topic,
specific
metrics
for
your
Kafka
clusters.
A
That's
like
one
way
to
do
it,
but
what,
if
you
don't
want
to
give
your
open
Sumter
collector
access
to
your
admin
apis,
which
is
very
understandable?
Then
what
you
can
do
is
use
the
jmx
receiver.
So
the
jmx
receiver
basically
uses
targeted,
basically
Target
specific
nbin
servers
to
directly
pull
metrics
from
Kafka
jvm
Cassandra,
whatever
you
want
and
we
were
able
to
collect
health
and
performance
metrics
from
our
Kafka
servers
using
the
jmx
receiver
and
putting
that
all
through
the
open2mg
collector.
A
A
The
way
you
want
you
can
write
your
own,
so
that's
the
real
power
of
open
Telemetry,
it's
very
flexible
and
there's
so
many
ways
to
do
one
thing
so
yeah
once
we've
gotten
all
of
that
data,
whether
it's
from
Kafka
or
other
Technologies
into
the
pipeline.
How
do
we
increase
the
usefulness
of
the
data?
A
So
a
really
cool
processor
that
I
use
in
at
New
Relic
is
the
kubernetes
attributes
processor
and
it's
really
cool,
because
it
really
dramatically
increases
the
value
of
the
logs
traces
and
metrics
being
generated
from
my
application.
And
that's
because
the
open,
Target
collector
with
this
attributes
processor
injects
metadata
about
my
pod
about
my
deployment.
A
B
B
The
first
one
is
the
filter
processor,
which
can
be
used
to
you
can
configure
it
to
include
or
exclude
specific
metrics
logs
and
traces.
For
example,
you
can
use
it
to
filter
logs
based
on
severity
level
or
even
the
text
itself.
The
second
one
is
going
to
be
the
tail
assembling
processor,
which
is
specifically
for
Trace
data.
B
Let's
say
these
are
the
choices
that
your
system
is
producing.
That
sounds
so
loud
if
you're
using
head-based
sampling,
you
are
either
one
going
to
get
a
lot
of
noise
because
you
are
keeping
all
the
chases
that
your
system
is
producing
or
two
you're
getting
a
randomized
sampling
based
on
the
probabilistic
rate
that
you've
configured
as
it
can
help.
So,
for
simplicity's
sake,
let's
say:
you've
configured
it
to
be
ten
percent,
as
you
can
imagine,
that
means
you're
getting
one
out
of
roughly
every
10
traces.
B
By
setting
up
the
tail
someone
processor,
you
can
Implement
one
or
more
policies
to
keep
traces
that
are
important
for
your
system,
so
you
can
keep
chases
with
errors,
specific
span,
attributes
or
that
have
crossed
a
certain
latency
threshold.
This
helps
to
ensure
that
you're
exporting
only
the
data
that
you've
determined
is
important
for
understanding
your
systems.
B
Finally,
as
Daniel
mentioned
you're
going
to
want
to
export
all
this
data
somewhere
to
either
a
managed
or
open
source
backend,
so
you
can
analyze
it.
The
component
that
you're
going
to
use
to
do
this
is
called
the
exporter.
You
can
use
the
default
otlp
exporter
since
most
back-ends
now
support
otlp
data
natively,
and
you
can
also
use
different
exporters
to
send
multiple
data
formats,
as
well
as
to
multiple
backends.
At
the
same
time,
you
could
even
export
your
traces
of
metrics
to
one
backend
and
your
logs
to
another.
B
Once
you've
got
your
receivers,
processors
and
exporters
set
up.
You
now
have
the
ability
to
build
your
data
pipelines
using
the
open
television
collector.
All
the
configuration
is
handled
within
a
yaml
file,
so
updating
is
fairly
trivial,
depending
on,
of
course,
the
complexity
of
what
you
have,
and
you
now
also
have
a
centralized
location
for
your
observability
configuration.
B
These
are
just
some
challenges
that
we
want
to
leave
you
with
before
we
go
to
take
into
consideration
with
using
the
collector
the
first
one
is,
it
can
be
complex,
a
scale
you
will
have
to
consider
deployment
patterns
as
well
as
little
balancing
the
more
data
your
system
is
producing.
Monitoring.
The
Collector
is
only
available
with
the
collector
at
this
time,
which
is
not
an
ideal
situation,
and
the
current
stability
status
is
mixed,
since
the
core
components
are
also
at
varying
levels
of
stability
and
that's
it.