►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Logging: Fluentd & Fluent Bit - Eduardo Silva & Masahiro Nakagawa, Treasure Data
In this presentation, we will cover the basics, internals and best practices of Logging applied to distributed systems, we will do this through the CNCF projects Fluentd and Fluent Bit. We will introduce the pipeline of data collection, filtering, buffering and routing to external services for data centralization. In addition we will cover best practices for Kubernetes for different types of deployments such as DaemonSets and Sidecars. If you are interested in observability, this session is a must!
https://sched.co/ekHb
A
The
front
d
is
written
in
louis
with
c
extension
and
friendly
unit
regimes
for
the
plug-in
distribution.
So
we
can
use
whole
ruby
ecosystem
for
brawling
development,
the
whole
setup.
We
provide
several
approaches
so,
for
example,
the
friendly
community
provides
always
packages
like
the
rpm
and
msi
packages,
so
you
can
run
friendly
on
linux
and
windows.
A
A
A
A
A
Again,
so
friendly
consists
of
the
core
part
and
the
plugins
parts.
This
page
shows
the
details
of
the
front.
Zipper
design
friendly
core
provides
several
important
features
for
data
collection.
For
example,
progress
don't
want
to
implement
the
complicated
buffering
or
literary
mechanism
in
each
of
each
implementation
if
out
of
the
programming
hits
the
temporal
failure.
A
A
A
A
A
A
A
A
Buffer
programming
is
used
inside
the
output
plugins,
so
not
standalone.
Incoming
events
are
stored
into
buffer
before
right
to
destinations.
Buffer
plugin
is
the
core
of
robust
buffering
and
the
literal
mechanism
for
the
output
plugin.
So
if
you
use
file
buffer
data
stored
on
the
path
center
disk,
this
allows
the
data
loss
even
frame
friendly
crashes.
So
we
recommend
to
use
the
file
buffer
on
production.
A
A
A
This
is
a
confirmation
example
for
this
us
cases,
so
the
confirmation
syntax
is
apache
performance,
a
source
directive
for
the
input
programming
and
the
multi-direction
of
the
programming.
This
example
doesn't
have
a
filter
planning
example.
Sorry,
we
mentioned
so
much
directive
as
a
target
button
for
the
event
routine.
So
in
this
case,
some
events
from
deliberate
programming
has
the
autographic
style.
So
these
events
are
sold
into
the
mongodb.
A
A
A
First,
I
chose
the
several
resources
for
the
container
logging,
so
for
docker
we
provide
the
rp
and
debian
images,
so
we're
required
to
use
the
debian
image
so
production,
because
the
devia
images
works
with
the
system
and
j
mark.
We
use
j
mark
to
optimize
the
memory
usage
for
kubernetes.
We
provide
the
demo
set
settings
and
images
for
the
popular
destinations
or
classic
search
or
kafka
and
other
destinations
supported
by
the
original
repository.
A
Okay,
next
is
we
shall
we
talk
about
how
to
collect
log
streams
containers,
and
this
is
a
docker
use
cases.
The
car
has
loading
driver
for
sending
logs
to
the
external
systems
and
the
docker
supports
when
drilling
driver
by
default,
so
you
can
send
application
loads
to
the
friendly
using
login
driver.
A
A
A
A
On
quantities
we
use
demo
set
for
collecting
data
from
containers,
kubernetes
stores
container
looks
into
the
bulk
containers
directory
and
friendly
eat.
These
looks
using
in-depth,
plug-in
observantly,
demonstrate
images,
use
this
approach
for
correction
and
demonstrate
images
include
the
metadata
metadata
filter
by
default.
B
Thank
you,
masa,
hello,
everyone.
My
name
is
eduardo
silva
and
I'm
one
of
the
maintainers
of
the
fluenbeep
project.
Fluenbit,
is
part
of
the
fluidity
ecosystem
and
it's
a
project
that
started
around
2015
initially
for
ember
linux,
but
quickly
evolved
to
solve
cloud
native
problems
and
it's
under
the
apache
line.
Sense.
Of
course,
one
of
the
advantage
of
fluid
in
general
is
that
it's
written
in
c
language,
and
this
is
very
optimized
for
a
low
cpu
and
low
memory
footprint.
B
Actually,
it
already
has
a
plugable
architecture
where
you
can
have
more
than
60
plugins
available
built
in
natively
in
c.
So
what
this
is
one
of
the
advantages
that
when
you
deploy
the
agent,
you
don't
need
to
insult
a
dependencies
and
from
a
data
workflow
perspective,
it's
very
similar
to
fluency.
We
have
an
input
section
where
we
collect
data
from
we
have
the
parsers
to
parse
the
data
and
convert
from
instructor
format
to
structure
format.
We
have
the
filtering
phase
for
data
enrichment.
B
If
you
think
about
uber
needs,
you
need
to
enrich
your
data
with
kubernetes
a
metadata
like
labels
annotations.
We
have
all
these
buffering
section
and
rolling
mechanism
to
send
your
data
from
one
place
or
many
others,
these
kind
of
destinations
or
the
input
source
of
data
all
of
them
are
configured
by
plugins
and
fluent
bit
in
general.
It
is
made
for
a
high
performance
at
low
cost
solution.
B
So
if
you
have,
for
example,
you
are
working
now
in
a
common
cloud
environment
and
you
have
a
very
intense
application
that
generates
a
lot
of
logs.
Actually,
what
you
want
is
to
have
that
your
agent
consume
as
less
view
as
possible,
but
be
able
to
process
your
data
and
ship
your
data
and
that
it's
it's
fluent
in
kubernetes.
B
You
can
have
many
nodes
right.
A
cluster
has
a
master
api.
You
have
notes
and
you
have
your
applications
inside
codes,
so
this
is
kind
of
a
scenario.
It's
a
it's!
Not
that
easy
to
to
process
the
locks
and
ship
all
the
laws.
At
two
months,
as
master
said
in
the
previous
present
in
the
previous
section,
he
said
that
yeah
we
have
all
this.
It
can
be
deployed
as
a
demon
set
and
can
solve
the
problem.
So
we
follow
the
same
pattern.
B
We
are
able
to
be
deployed
as
a
demon
set
and
solve
all
this
problem.
Imagine
that
if
you
have
an
application
that
is
generating,
for
example,
an
apache
web
server
is
generating
the
log
messages
right.
All
of
them
doesn't
have
a
structure
and
goes
to
the
file
system
or
goes
to
journal
d
when
these
messages
are
generated.
For
example,
if
you
consider
that
they're
using
the
json,
the
json
format
for
the
file
system,
all
of
them
will
have
a
every
container
will
have
its
own
specific
log
file.
B
So
the
agent
needs
to
be
able
to
correlate
all
this
information
together,
and
that
message
is
not
just
a
message.
It
also
has
a
strain
information
and
the
time
that,
when
this
message
was
created
inside
the
kubernetes
node,
but
also
it's
really
important
to
understand
and
get
the
context
of
the
information.
B
So
a
simple
message
that
is
start
in
kubernetes
or
as
a
in
a
pod
like
this
becomes
something
with
more
content
like
the
processed
data
right,
the
kubernetes
metadata
labels,
annotations
the
port
name
and
so
on.
So
with
this,
we
correlate
all
the
information
from
the
cluster
and
every
record
has
all
the
information
together.
So
when
you
process
this
is
a
more
straightforward
process
and
how
this
works
internally,
fluent
data
is
deployed
as
a
demo
set,
but
also
can
be
deployed
as
a
cycad
cycler
container.
B
It
reads
the
logs
from
the
file
system,
then
it
goes
to
the
api
server,
as
I
said,
to
retrieve
the
metadata
and
then
to
be
able
to
finally
push
the
old
data
that
is
correlated
to
your
final
destination.
Final
destination
can
be
any
type
of
service
that
supporter
or
any
kind
of
custom.
Http
endpoint
you
can
use
your
own
elastic
search,
kafka,
aws,
possibly
sql
will
have
many
connectors
for
your
different
and
favorite
back-ends.
A
B
In
2015,
we
are
close
to
have
a
to
turn
five
years
next
year
and
what
is
really
important
here
to
understand
what
are
the
new
things
that
are
coming
up
in
the
project
so
a
few
weeks
ago
we
released
fluid
bit
1.6
and
one
of
the
major
a
feature
set
that
we
ship
are
the
new
enterprise
connectors
enterprise.
I
mean
that
we
connect
to
this
enterprise
connector.
All
of
them
are
for
free
for
use.
B
Of
course,
it's
for
the
azure
blob
service,
where
you
can
send
your
data
to
a
block
server
to
a
blob
or
on
a
pen
block
in
azure.
Also,
we
have
the
new
integration
for
graph
analogy
and
also
with
amazon.
We
just
come
together
with
a
connector
for
amazon,
s3
and
amazon
firewalls.
This
is
not
new.
At
least
loki
and
amazon
connectors
already
existed
in
different
image
images
provided
by
grafana
and
by
aws,
but
they
implemented
the
connectors
in
golem.
B
B
If
you
know
about
filtering
and
reach
your
metadata,
you
will
know
that
also
it's
not
about
enrichment.
You
also
can
take
some
logical
decisions
like
filter
your
data
out
or
take
some
action
or
trigger
some
alerts
and
now-
and
we
are
going
step
forward
with
that-
and
we
are
trying
to
bring
all
these
machine
learning
all
these
capabilities
capabilities
for
analytics
inside
the
fluence
pipeline.
B
As
of
last
month,
in
october,
we
hit
170
million
deployments
for
this
year.
I
think
we
are
going
to
close
200
million
in
the
whole
year,
so
this
means
a
huge
traction
in
the
project
so
and
these
kind
of
deployments
are
happening
every
day
and
like-
and
this
is
like
crazy-
these
are
crazy
numbers.
But
if
we
look
carefully,
this
is
thanks
to
the
help
of
every
enterprise
company
that
it's
a
a
cloud
provider
or
a
service
provider.
B
Actually
from
b8,
it's
been
supporting
the
three
major
cloud
providers
like
aws,
google,
cloud,
microsoft
and
others
like
digital
ocean
and
bicycle
cloud,
and
so
on.
So
as
a
project,
it's
really
important
that
we
keep
all
this
synergy
across
user
end
users,
enterprise,
the
companies
and
also
the
developers.
So
we
try
to
have
all
this
kind
of
furniture
together,
because
the
feedback
that
you
provide
also,
if
you're
watching
this
this
session
live
on
kubecon,
is
really
important.
B
I
would
say
that
78
of
the
features
that
you
see
from
a
bit
is
because
of
the
data
from
users,
like
you
also
we're
looking
for
how
to
get
better
at
metrics.
So
we
are
working
together
with
the
open,
telemetry
team
to
try
to
integrate
together
and
integrate
a
better
way
for
them
to
manage
loading
through
fluid
more
about
this
news,
you
will
get
a
next
year,
so
this
is
just
a
sneak
peek
about
the
work
that
we
are
doing
together.