►
From YouTube: Lightning Talk: Data Flow Control in Cluster Logging Pipeline - Pranjal Gupta & Eran Raichstein, IBM
Description
Lightning Talk: Data Flow Control in Cluster Logging Pipeline - Pranjal Gupta & Eran Raichstein, IBM
Logging pipelines are crucial in ensuring container logs are reliably collected and routed to persistent storage. Logs generated by workloads (container processes) are written to files by Container Monitor processes (e.g. Conmon). In production environments, as Fluentd deals with a massive volume of logs, the log generation rate often exceeds the rate of log collection, which causes log loss. There is a need to prioritise application logs so that administrators can collect logs from high priority workloads in a controlled manner. In this talk, we introduce a new feature in the in_tail input plugin, which uses group rules to rate limit log collection. We share exciting insights from our systematic study about log loss on Fluentd plugins using our open-source benchmarking framework. We also present a Log Flow Control framework that allows users to define and enforce log rate limit policies to control log loss predictably.
A
So
hi
everyone,
sorry
for
the
small
delay.
We
had
some
technical
problems
and
before
talking,
I
just
want
to
say
thank
you
for
the
fluent
guys
in
the
community
that
are
hosting
us
here.
I'm
and
pranjel
gupta
we're
from
ibm
research,
and
we
are
working
very
strongly
with
the
reddit
team.
So
there
is
a
ibm
research
and
relative
are
working
together.
A
The
area
that
we
are
focused
on
is
observability
and
and
observability
stack
of
openshift
and
prangel
will
talk
about
it
and
elaborate
a
lot
more.
The
area
that
we
are
trying
to
promote
is
treating
logs
to
be
equivalent
citizenship
like
to
the
rest
of
the
resources
in
in
clouds
and
in
kubernetes
in
openshift.
So
just
like
cpu
and
memory,
which
are
managed
and
controlled
resources
in
such
distributed
environment.
We're
trying
to
get
to
the
situation
that
logs
are
treated
in
the
same
way
so
that
we
can
control
them.
A
We
can
control
the
amount
of
logs
that
are
being
generated
and
the
amount
of
log
that
is
being
collected
using
fluency
and
and
the
logging
stack
just
that
that
this
is
managed-
and
these
are
those
are
not
like
free
resources.
In
the
system-
and
I
think
that
eduardo
touched
that
in
the
beginning,
when
he
was
saying
that
there
are
too
many
logs
in
the
world
and
everyone
is
sending
logs
out
so
this
is
exactly
the
situation
that
we're
trying
to
handle
over
here,
making
sure
that
everything
is
under
control.
A
B
Thanks
around
for
the
introduction,
hi
everybody
so
first,
I
would
like
to
introduce
like
what
is
openshift.
So
openshift
is
a
flagship
product
as
a
service
platform
for
from
red
hat
built
on
top
of
kubernetes,
and
it
allows
you
to
deploy
and
manage
your
containers
in
a
easier
way
than
using
plain
kubernetes
environment.
B
B
B
So
on
this
slide,
this
is
a
like
very
high
level
view
of
how
our
logging
pipeline
looks
so
like
logs
from
containers
like
in
the
form
of
in
standard
out
and
standard
err.
Streams
are
written,
two
log
files
on
the
disk
by
container
runtime
interface
and
logs
from
these
files,
which
are
stored
in
var,
lock,
containers
are
read
from
fluently,
normalized
and
then
sent
to
persistent
storage
like
elastic
searches,
log
affluent
forward.
B
So
in
situations
where
you
don't
have
control
of
on
the
amount
of
logs
being
collected
right
where,
like
lot
of
logs,
are
being
generated
from
each
application
and
you
want
to
troubleshoot,
you
want
to
debug
and
you
don't
know
what
is
happening.
So
in
those
situations
you
need
to
have
a
very
good
state
of
what
is
happening
in
the
cluster.
B
B
So
this
means
that
when
fluenty
misses
log
rotation,
you
start
to
lose
logs
right
and
this
can
be
accounted
to
the
number
of
missed
rotations
into
the
size
of
each
file
of
log
log
file
right.
The
second
one
is
the
data
clogging.
So
when
you,
when
you
have
very
less
memory
or
cpu
resources
available
to
you,
fluently's
output,
buffer
buffer,
starts
to
get
like
overflow,
and
you
know
you,
you
tend
to
lose
logs
because
you
don't
know
what
to
do.
Right.
B
B
So,
given
these
scenarios
from
our
architecture
right,
we
have
come
up
with
a
motivation
so
during
like
worst
case
scenarios
when
you
want
to
debug
and
troubleshoot
what
is
happening
in
a
cluster,
you
want
to
prioritize
log
collection
at
the
input
level
so
that
you
don't
miss
on
important
logs
right
and
and
as
a
part
of
the
aggression
process.
You
want
to
make
sure
that
your
crucial
resources,
like
network
bandwidth
and
persistent
storage,
are
not
saturated.
Given
the
resource
contains
constraints.
B
B
So
before
moving
on
to
the
observations,
I
will
just
give
you
an
overview
of
what
is
our
experimental
setup.
So,
in
general
scenario,
you
have
two
groups
of
containers.
One
is
very
important
which
you
don't
want
to
miss
logs
from,
and
one
is
the
less
important
containers
which
are
chatty
which
are
noisy
and
it's.
It
is
okay.
B
If
you
lose
some
logs
from
those
containers
and
the
objective
is
we
want
to
preserve
logs
from
very
important
containers
so
that
you
can
troubleshoot
right,
which
is
very
important
for
you
as
a
developer
or
an
sre
to
come
to
a
stable
state,
and
the
approach
which
we
are
following
is
that
we
can?
We
are
saying
that
we
can
afford
to
lose
some
logs
from
less
important
containers
and
preserve
more
from
what
is
important
to
us
and
as
a
baseline,
we
are
using
one
of
the
existing
open
source
plugin,
which
is
called
the
throttle
plugin.
B
In
all
these
experiments,
we
have
two
graphs.
One
is
where
we
don't
have
throttle,
which
is
the
normal
situation,
and
one
is
where
we
have
throttle
applied
to
the
less
important
containers
so
upon
applying
throttle,
you
start
to
lose
log,
but
you
also
get
some
benefits
right.
So
there's
a
trade-off
in
what
you
choose.
B
B
B
So
when
you
have
different
like
when
you
have
a
large
buffer
size,
let's
say
1gb
or
the
reading,
like
the
amount
of
logs
read
from
each
file,
is
different
and
when
you
have
a
smaller
buffer
size,
you
see
that
the
peaks
are
of
equal
size.
That
means,
irrespective
of
what
is
the
generation
rate
of
your
log
you're,
reading
equal
amount
of
lines.
B
So
why
is
this
important
right
because
in
in
worst
case
scenarios
where
you
do
when
you
don't
know
what
to
do?
You
need
a
good
amount
of
logs
so
that
you
know
what
is
actually
happening
in
your
system.
You,
you
need
to
have
a
good,
clear
snapshot
of
your
entire
system,
so
you
need
to
have
some
information
from
all
your
all
of
your
pods.
B
So
here
comes
our
the
feature
which
is
called
group
based
throttling
in
entail,
so
you
can
form
groups
in
your.
You
can
define
user
groups
in
the
intel
plugin,
and
you
can
define
rules
for
assigning
each
workload
to
a
file
to
a
group.
Then
you
can.
You
can
read
limit
the
logs
being
collected
from
those
files,
and
this
feature
ensures
that
your
groups
are
bred
equally
and
rate
limiting
is
done
at
the
time
of
reading.
B
B
However,
this
intel
plugin
is
not
only
used
for
kubernetes
right,
so
we
have
made
this
grouping
pattern,
generic
so
that
you
can
use
for
other
files
as
well.
So
you
can
define
named
captures
in
your
group
pattern
and
then
you
can
specify
matching
key
file
or
hash
hash
table
in
your
match,
parameter
in
the
rule
directive.
B
This
can
allow
you
to
customize
and
generalize
your
grouping
rules
so
that
you
can
use
it
anywhere.
You
want
this
feature
will
will
be
available
in
1.15
release
version
of
fluendy,
which
will,
I
think,
will
be
released
in
may
end,
and
you
can
have
a
look
at
the
pr
and
it
has
not
been
merged
yet,
but
feel
free
to
look
at
it
and
make
some
reviews
if
you
want
now
coming
to
what
we
are
doing
as
part
of
red
hat
and
ibm
research.
B
Just
a
reminder.
This
was
this
is
the
api
which
we
are
using
to
configure
our
operator.
So
what
we
are
now
doing
is
we
are
defining
some
policies
through
which
we
can
control
different
components
of
our
cluster
logging
pipeline.
For
example,
we
can
simply
limit
the
rate
of
logs
being
sent
to
kafka
to
one
gigabit
per
second
to
avoid
saturating
network
link
because,
as
I
said,
persistent
storage
and
network
bandwidth
is
very
crucial.
B
Similarly,
you
can
control
certain
parts
from
from
a
namespace
with
certain
labels
and
you
can
define
per
container
limit,
which
is
a
rate
limit
per
file
or
for
rate
limit
for
the
entire
group,
or
you
can
simply
ignore
certain
pods
as
well.
That
means
you
don't
even
collect
from
those
pods.
In
this
way
you
are
saving
again
the
resource,
very
crucial,
cpu
resources
and
you're
concentrating
on
what
is
important.
B
So
how
does
this
api
look
when
we
apply
these
policies?
So
you
can
see
on
the
red
box.
Is
the
limit
reference
like
we
have
like?
We
are
trying
to
drop
or
drop
logs
if
the
maximum
records,
the
incoming
log
rate,
is
more
than
50
lines
per
second
per.
Second
sorry-
and
you
apply
this
rate
limit
to
an
input
application,
which
is
where
we
have
defined
custom
groups
there,
which
is
like
collect
all
lines
from
pods
which
have
named
less
important
and
are
from
namespace
log
stress.
C
A
So
in
in
kubernetes,
the
cpu
and
memory
and
resources
of
the
logging
stack,
like
any
other
set
of
applications,
is
also
limited.
So
we
don't
want
to
take
it
to
infinity
when
there
are
applications
that
emit
a
lot
of
logs
really
a
lot
of
logs.
It
does
make
sense
to
put
some
threshold
or
some
limit
to
the
amount
of
log
and
the
amount
of
resources
that
the
logging
stack
itself
is
taking
from
the
system,
because
it's
it
will
start
to
affect
the
other
application
that
you
have
on
the
cluster.
A
This
is
why
it
does
make
sense,
sometimes
when
it's
really
a
lot
of
logs
to
start
to
see
log
loss,
and
this
is
exactly
where
you
want
to
see
the
log
loss
on
containers
that
are
not
the
most
important
containers
that
you
have
in
the
system.
So
you
want
to
to
balance
the
effect,
and
this
is
exactly
what
we're
doing.