►
From YouTube: Lightning Talk: Fluentd at Scale; Keys to successful Logging Syndication - Fred Moyer, Zendesk
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: Fluentd at Scale; Keys to successful Logging Syndication - Fred Moyer, Zendesk
This lighting talk will show how Zendesk uses Fluentd to manage a firehose of systems and application logging telemetry to multiple consumers. It will go into detail on configuration and operational details about how they syndicate logs to different systems for stakeholders such as security engineers, application developers, and infrastructure teams. If you find yourself dealing with an increasing firehose of logging telemetry with multiple teams consuming that telemetry, this talk is for you.
A
Hey
folks
welcome
to
fluentd
at
scale
keys
to
successful
logging
syndication.
This
lightning
talk
is
being
given
at
fluentcon
2021..
Let's
get
started,
I'm
fred
moyer
and
I
work
at
zendesk
providers
of
awesome
customer
service
software
at
zendesk.
Reliability
is
feature
number
zero.
If
we
can't
serve
our
customers,
they
can't
serve
their
customers
at
zendesk.
A
I
spend
a
lot
of
time
working
our
tour
working
on
our
telemetry
fire
hose
of
metrics
logs
and
traces
today,
I'll
be
talking
about
how
we
use
fluentd
to
handle
the
ever
increasing
volume
of
logs
that
our
distributed
microservices
generate
I've
been
doing
production
web
applications
for
a
while.
So
part
of
my
reflections
are
from
a
developer
perspective
and
part
from
an
opera
operational
perspective.
A
A
In
addition
to
our
you
know,
historic
support,
ticketing
service
app,
as
well
as
adding
other
products
through
acquisitions,
zendesk
moved
out
of
co-located
data
centers
starting
around
five
years
ago
and
moved
into
aws
services
on
amazon.
Nearly
all
services
now
are
delivered
from
what
we
call
a
pod,
which
stands
for
point
of
delivery.
A
Applications
are
delivered
via
kubernetes
clusters
across
multiple
availability
zones
and
regions.
Our
usage
of
kubernetes
has
grown
significantly
over
the
past
several
years,
which
has
resulted
in
an
equally
significant
increase
in
the
logging
volume
that
we
have
to
handle
logs
are
written
to
standard
out
on
docker
containers
via
the
docker
json
logging
plugin.
A
These
logs
are
read
from
the
file
system
by
fluentd,
which
is
deployed
in
the
pods.
As
a
kubernetes
daemon
said,
the
log
records
are
tagged
with
metadata,
such
as
the
kubernetes
cluster
details,
pod
id
and
other
metadata
through
a
number
of
fluentd
filter
plugins,
as
we
have
a
number
of
log
consumer
stakeholders,
we'll
include
as
much
metadata
as
possible
on
each
log
entry
so
that
end
users
can
take
advantage
of
those
in
any
of
the
monitoring
tools
we
use,
which
are
listed
on
the
right.
A
Finally,
a
set
of
matcher
plugins
sends
the
log
entry
to
one
or
more
output
destinations.
Currently,
those
destinations
are
datadog
kafka.
We
have
a
work
in
progress
to
add
s3
as
an
additional
output
destination
and
the
fluentd
output
plugins
made
the
actual
mechanics
of
sending
the
logs
to
their
destinations
relatively
easy
and
painless.
A
Zendesk
has
hundreds
of
services
generating
logs
which
deliver
the
products
shown
in
the
previous
slide.
Those
are
distributed
across
multiple
aws
regions
to
serve
customers
in
the
americas,
asia,
latin
america
and
europe,
middle
east
and
asia.
To
manage
these
business
rules,
there
are
multiple
engineers
contributing
to
the
configuration
and
operation
of
fluentd
like
any
large
organization.
A
The
business
rules
implemented
in
fluent
deconfiguration
can
get
a
bit
complex
and
not
something
that
one
can
easily
fit
into
a
mental
model
in
one's
head.
Anyone
who's
worked
on
a
high
skill.
Web
service
will
understand
this
now
to
up
the
stakes
even
more.
We
require
a
very
high
level
of
reliability.
A
Achieving
three
and
a
half
nines
at
startup
level.
Traffic
levels
might
result
in
a
few
hundred
lines
of
lost
telemetry,
but
at
much
higher
levels
of
traffic,
whole
swaths
of
telemetry
are
at
risk.
If
a
deployment
fails,
there
are
multiple
teams
who
are
stakeholders
of
logging
telemetry,
any
number
of
feature
teams,
security
teams,
customer
advocates
who
use
logs
very
heavily
and
other
teams.
A
A
A
Our
initial
fluency
deployment
was
four
years
ago
and
at
that
time,
fluent
bit
didn't
have
the
production
maturity
that
it
does
now.
There
have
been
a
number
of
performance
increases
in
fluent
bit
that
we've
learned
about
from
the
klopsha
folks.
Additionally,
zendesk
engineers
have
a
significant
amount
of
experience
in
ruby,
which
made
the
fluentd
implementation
appealing.
The
number
of
output
plugins
available
was
also
a
big
factor
being
able
to
get
things
up
and
running
without
having
to
write
a
lot
of
custom
code
was
a
very
big
draw.
A
All
that
being
said,
as
our
traffic
levels
have
grown
over
the
past
four
years,
we
have
revisited
this
decision
and
may
look
at
transitioning
to
fluent
bit.
The
business
rules
that
we
have
implemented
in
fluentd
are
an
area
of
consideration
that
we
would
have
to
ensure
compatibility
with
a
migration
of
fluent
bit
because
of
our
large
footprint.
A
A
change
like
this
would
take
time
and
the
input
of
many
stakeholders,
but
one
that
we
may
pursue
since
our
log
delivery
rates
continue
to
increase,
as
the
business
grows,
our
svp
of
engineering
tweeted
that
it
takes
six
months
to
make
a
major
infrastructure
change
in
a
company
as
large
as
zendesk.
I
found
that
to
be
true
across
the
small
number
of
large
companies.
I've
worked
at.
A
A
Another
trade-off
that
we
have
made
is
choosing
to
implement
simple
business
logic
in
our
fluentd
plugins.
Instead
of
complex
logic,
the
json
logs
produced
by
our
services
contain
a
coordinate
copia
of
rich
metadata
and
the
ruby-based
plug-in
architecture
makes
it
possible
to
do
a
number
of
complex
and
potentially
valuable
transformations
in
fluentd.
A
A
Essentially,
we've
chosen
to
implement
fluentd
as
a
router
and
embed
business,
specific
logic
in
the
monitoring
back-ends.
We
feel
that
this
has
paid
off.
There
are
enough
routing
rules
that
we've
had
to
implement
as
a
result
of
the
size
of
our
systems,
but
trying
to
add
complicated
business
logic
in
fluent
d
would
have
significantly
increased
the
risk
of
deployments
and
reliability.
It's
a
bit
of
the
unix
philosophy
use
small
tools
that
have
a
dedicated
purpose.
As
you
scale
out,
the
usage
of
those
tools
will
accrue
complexity
of
their
own
over
time.
A
A
There
are
a
number
of
settings
in
fluentd
which
allow
one
to
tune
the
flush
behavior
of
fluent
d.
You
may
want
to
adjust
these
settings
based
on
how
your
own
environment
behaves.
Some
of
the
changes
to
these
settings
that
we
have
found
helpful
was
to
decrease
the
flush
interval
to
keep
buffers
from
growing
too
large,
as
well
as
increasing
the
number
of
flush
threads.
A
We
found
that
as
the
buffers
would
grow,
the
memory
usage
of
fluid
d
would
increase,
so
more
frequent
flushes
would
keep
memory
usage
under
control.
Now
these
settings
were
specific
to
each
output,
plug-in
buffer
flushes
to
data
og
behave
differently
than
buffer
flushes
to
kafka
and
similar.
They
were
different
for
splunk
tuning.
These
settings
for
best
performance
is
a
bit
of
an
art
form,
but
we
were
lucky
to
have
some
advice
from
the
folks
at
klipsch
on
this.
The
best
values
for
these
settings
will
change
over
time
as
your
traffic
levels
change.