►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Observability Pipeline with Fluent-Bit - Chao Xu, LinkedIn
This talk will discuss LinkedIn's efforts and experiences to build up the observability pipelines with the help of open source standards and tools, specifically OTEL and fluent-bit. Starting with tracing, we adopted the industry standard OpenTelemetry (OTEL), including both tracing schema and SDKs. This gave us a chance to leverage the already existing and supported tools and components. We also decided to introduce the transport agent to hide all the details of accessing the data backend from the application, therefore get the much needed multi-language support. (Not) surprisingly, Fluent-bit was chosen as our transportation agent, as opposed to the OTEL agent already provided in the OTEL SDK. This decision was made mainly based on the following considerations: - Kafka is our preferred and chosen centralized data collector - Log streaming is the required functionality. - Resource efficiency and performance. We also enhanced fluent-bit in the following ways: - OTEL transport agent with protobuf support - Metrics emission for Monitoring - Tag management and enhancement We'd be happy to contribute these enhancements back to the open source community in the future.
A
A
So,
first,
a
quick
agenda,
we'll
start
with
the
observability
data
pipeline
existence,
data
status
and
then
what
the
issue
associated
with
this
data
Pipeline
and
then
we'll
introduce
the
real
animation
work.
We've
been
doing
for
the
open,
using
with
the
open
standard
and
open,
open,
Source
tools
and,
finally,
the
status
and
the
summary.
A
A
So
you
may
or
may
not
notice,
that
in
the
past
few
years
we
are
entering
an
error
of
observability,
so
for
any
company
with
Distributing
system,
the
observability
becomes
the
critical
support
to
provide
the
functionality
for
diagnosing
debugging,
tracking
and
detecting
bottlenecks.
So
what
is
this
observability?
There's
many
definitions.
A
One
of
them
is
that
it
provides
the
insight
and
of
the
service
runtime
state
or
property
in
the
Distributing
system.
So
what
does
that
real
mean?
It
really
means
that
in
the
high
obsibility
requirement,
make
it
very
necessary
to
provide
sufficient
context
or
insight
for
information
about
the
system
by
combination
of
a
few
different
and
key
types
of
information
together,
so
that
you
can
get
the
information
of
the
system
you
can
have
the
inside
of
the
system.
A
There's
no
exception.
Linkedin.
For
years
we
build
up
observability
data
pipeline
to
collect
different
insights
and
informations
into
the
Distributing
system.
We
have
logs.
We
have
graphs,
we
call
in
graphs.
We
have
our
Trace
information,
we
we
can
leverage
and
look
through
the
some
application
called
core
tree.
We
have
a
lot
of
other
tools
built
around
them.
A
However,
since
this
this
system
was
designed
and
evolved
and
maintained
by
different
teams
many
years
ago,
most
of
the
tools
built
around
this.
This
data
is,
is
like
isolated
and
they
can.
They
cannot
like
interoper,
operate
between
different
database
to
get
more
insight
out
of
the
data
to
meet
the
ever
increasing
requirement
of
the
data
traffic.
We
LinkedIn
has
started
several
projects
in
the
past
few
years
trying
to
improve
this
situation.
A
A
A
A
A
The
the
that
project
was
based
on
the
some
open
external
tools
which
end
up
a
failure
or
like
a
very
unsuccessful
of
this
project,
because
the
the
scale
that
tool
support
cannot
meetings
requirement.
We
have
much
more
large,
larger
requirement
of
the
data
pipeline.
So
for
years,
our
engineer
or
like
SRE
has
to
log
into
the
different
box,
or
virtual
or
virtual
machine
just
trying
to
to
to
grab
or
to
to
to
get
the
that
did
the
log
message.
A
So
this
this
situation
was
not
solved
until
recent
year
when
we
started
inlock
data
pipeline
project.
Basically,
we
built
our
own
data
pipeline
trying
to
ingest
the
message.
The
in-log
message:
in-log
event
message
was
emitted
from
application
and
sent
to
Kafka,
ultimately
to
cool
store,
which
also
known
as
Azure
data
Explorer.
So
custom
provides
much
better
ingestion
and
query
capability
and
make
make
it
very
like
convenient
for
the
for
the
engineering
to
get
the
log
messages,
which
is
it's
a
very
a
great
Improvement.
A
A
This
somehow
limited
the
capability
of
the
the
whole
data
pipeline.
Another
limitation
is
that
only
Java
application
has
the
capability
to
do
the
data
emissions.
So
that's
another
thing
we
like
to
improve
Matrix
data
pipeline
is
not
the
focus
for
my
today's
work,
so
I'm
just
quickly
touch
upon
the
of
it
and
detailed,
a
very
brief
story
of
it.
So
we
have
similar
data
pipeline
that
a
service
service
metrics
event
is
emitted
from
application
that
can
go
direct
to
Kafka
or
go
through
a
service
called
AMF
to
Capcom.
A
This
data
was
then
directly
emitted
to
our
tsds,
which
is
time
series
data
storage.
We
have
a
series
of
tools
built
around
this
and
to
visualize
to
generate
alerting,
generate
a
lot
of
different
data
or
an
ns2,
so
I'd
like
to
first
to
to
to
summarize
that
what
is
the
issue
for
our
existing
data
Pipeline
and
how
we
want,
then
how
we
want
to
improve
this.
A
A
A
To
address
all
these
issues,
LinkedIn
decided
to
renovate
the
data
Pipeline
with
the
Consolidated
data
to
consolidate
some
of
the
components.
So
here's
the
the
proposed
solution
we
started
with
generally.
The
new
data
pipeline
that
can
fit
both
tracing
and
logging
cool,
slow
again
is
is
being
used
as
the
data
storage,
which
can
provide,
as
I
mentioned
earlier,
the
the
efficiency
of
of
query,
as
well
as
like
ingestion.
A
A
A
Starting
from
open,
Telemetry,
I'm
sure
everyone
here
here
is
only
already
familiar
with
what
often
terminatory
standard
is
so
we
leverage
open
Telemetry
in
both
the
the
tracing
standards
tracing
schema,
as
well
as
the
sum
of
the
sdks
with
open
Telemetry.
We
can
easily
instrument
multiple
language
applications.
A
A
A
A
A
This
decision
was
was
made
like
for
a
lot
of
people's
surprise,
because
Hotel
SDK
only
provides
several
implementation
of
The
Collector.
We
didn't
choose
that
we
we
made
the
decision
even
before
we
became
aware
of
that
Flume
bit
will
support
open
or
open
telemetry,
so
why
we
made
this
decision.
There
are
several
reasons,
the
first
reason
as
I
mentioned.
A
If
you
look
at
the
past
few
slides,
we
have
Kafka
sitting
as
our
chosen
main
data
backbone,
so
any
output
won't
go
through
Kafka
and
provide
a
lot
of
different
plugins
to
support
different
customization,
including
Kafka.
This
gives
us
a
lot
of
advantage
using
using
fluency
to
directly
get
this
capability.
A
A
A
A
The
first
Improvement
we
added
was
like
the
open,
Telemetry
protocol
support.
That's
the
the
the
basically
the
the
open
Hotel
exporter
need
to
talk
to
the
agent
that
can
understand
its
language
and
open
Telemetry
protocol
was
the
standard
and
we
add
the
open
television
protocol
over
HTTP
support
with
protobuf
encoding
in
it.
So.
A
So
that's
that's
the
the
first
enhancement
we
added
next.
We
also
enhanced
the
The
Matrix
emission
mechanism
inside
Flume,
bit
from
bead
by
itself
is
already
have
the
the
the
building
monitoring
mechanisms
that
already
support
Source,
supported
by
externally
through
external
tools,
like
promise
use
I
think
mentioned
earlier,
and-
and
this
is
like
the
more
like-
a
pulling
a
method
in
the
used
by
Prometheus,
but
in
LinkedIn.
A
We
are
going
to
deploy
this
in
a
lot
of
different
nodes
and
pooling
methods
does
not
work
well
for
our
use
case.
So
we
want
to
adapt
this
Matrix
monitoring
mechanism
into
our
linkedin's
own
Matrix
collection
of
flow.
That's
where
we
add
the
adapter
adapter
trying
to
convert
some
of
the
The
Matrix
information
into
linkedin's
own
formats
and
emitted
it
to
a
central
place
so
that
we
can
have.
A
So
another
thing
we
we
enhanced
for
LinkedIn
is
the
the
tag
management
for
most
people
probably
already
know
that
tag
is
used
by
Flume
bit
for
the
streaming
management,
because
we
have
now
more
complicated
data
pipeline.
We
need
a
more
stronger
functionality
in
terms
of
the
tags
so
that
we
can
pass
some
of
the
information
directly
from
the
from
the
the
client
that
the
exporter
to
to
the
to
flown
bit
without
decoding
the
data.
A
There
are
two
choices
like
we've
been
thinking.
One
is
like
the
per
box
still
deploying
that
is
we
for
each
box
or
like
the
kubernetes
nodes.
We
have
one
copy
of
float,
running
there
and
collect
information.
Another
is
like
per
instance.
For
example,
like
you
have
each
application
instance,
you
have
one
copy
of
fluent
build
a
side
with
it
and
there's
a
pros
and
cons
of
each
approach.
That's
why
this
this
caused
a
lot
of
discussion
so
for
for
per
instance,
it
has
the
advantage
of
like
the.
A
It
has
the
advantage
of
like
the
better
like
resource
and
error
isolation.
It
can
avoid
the
the
noise
in
the
neighbor
neighboring
problem
that
we've
been
seeing
in
our
like
the
on-prem
on
attended,
Data
Center
and
it
make
the
configuration
and
configuration
and
the
monitoring
easier.
This
is.
This
is
obviously
like
if
you
have
multiple
instances
of
running
on
the
same
host
and
you
have
the
port
management
issue
that,
if,
if
you
have
you,
you
have
the
the
per
box
deployment
model
used.
A
A
Floombate
agent
is
now
already
rolled
out
in
a
lot
of
boxing
in
the
company.
We
are
in
the
process
of
the
raw
output,
the
enhanced
floam
bridge
to
all
the
bugs
in
in
the
company
and
the
currently
it's
not
fully
enabling
in
terms
of
configuration,
but
it's
sitting
there
waiting
for
the
traffic
from
the
different
applications
distribution
project.
We
are
have
some
pilot
application
chosen.
That
is
already
start
to
image.
They
are
tracing
information
and
to
collect
the
info,
collect
the
data
and
for
logging.
A
A
Okay,
a
quick
summary
of
of
the
of
the
talk
I
had
today,
so
I
introduced
the
the
Consolidated
data
pipeline
in
LinkedIn
to
support
both
logging
and
tracing.
There
are
other
works
that
has
by
my
team,
has
been
doing
as
well,
so
I
haven't
got
a
chance
to
discuss
them.
They
are
equally
important
and
hopefully
in
the
future.
I
can
introduce
more
of
them
this.
A
This
includes
this
effort
include,
like
the
data
partitioning
for
better
correlation
and
inquiries
and
a
visualization
we're
going
to
introduce
a
new
visualization
tool
to
handle
the
distribution
data
and
also
we
are
also
in
the
design
phase
of
data
sampling
and
acquiring
our
data
sampling
as
well.
This
is
mostly
for
query
efficiency,
as
well
as
like
the
lower
the
the
cost
and
and
and
we've
been
always
trying
to
optimize
different
components,
to
making
them
more
efficient
and
and
run
it
more
smoothly.
C
C
A
Will
actually
so
there's
a
couple
of
different
different
enhancements?
Some
of
the
enhancement
we
are
going
to
contribute
back
to
the
to
the
open
Community
like
to
that
is
used
for
everyone,
like
some
of
the
average
enhancement,
the
bug
fix
and
some
of
the
the
attack
tag
management.
But
some
other
parties
like
linking
only
like
the
specific
we're
doing
the
conversion
internally,
that
nobody
outward.
We
needed
that
we're
not
going
to
or
to
send
them
back
to
open
source.