►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Logging Operator the Cloud Native Fluent Ecosystem - Sandor Guba, Cisco
The Kubernetes logging ecosystem does not support the native Kubernetes concepts. This is the reason behind the open-source project called Logging Operator. Moreover, the easy scaling of workloads results in a massive amount of logs produced. This can cause extremely huge bills on your analytics deployment (both SaaS and self-hosted). Some of the key features are namespace isolation, supporting label selectors, filtering, and using secrets for credentials. It is more than 3 years old and already deployed in production environments. Sandor will introduce the challenges of configuring a complex logging stack on Kubernetes and summarise the lessons learned of writing, supporting, and operating the Logging Operator.
A
Hello
folks,
thank
you
for
attending
on
my
presentation,
called
logging
operator,
the
cloud
native
fluent
ecosystem.
It's
not
a
big
secret.
We
will
talk
about
fluent
bits,
fluenty
and
kubernetes.
Just
a
couple
of
thoughts
about
me.
I
was
working
as
an
infrastructure
engineer
at
ustream
acquired
by
ibm
and
after
that
we
founded
bonsai
cloud
with
my
friends
and
it's
recently
acquired
by
cisco.
I
have
several
years
of
experience
in
the
observability
field.
We've
been
operating
kubernetes
cluster,
since
version
1.4,
and
I
have
personally
more
than
5
years
experience
with
the
fluent
ecosystem
at
bonsai
cloud.
A
So,
let's
start
at
the
beginning,
with
the
kubernetes
part,
kubernetes
doesn't
have
too
many
options
when
it
comes
to
logging,
the
container
runtime
stores
each
port's
standard
output
in
a
file
on
the
host
disk
when
a
user
uses
the
cube
ctel
logs
command.
The
cubelet
service
goes
to
this
well-known
path
and
reads
up
the
logs
from
that
file.
Simple.
As
that
and
that's
it.
There
is
nothing
much
you
can
change
about
that.
So
what
are
the
problems
with
this?
First
logs
are
stored
locally
on
a
host
file
system.
A
These
logs
eventually
get
rotated
or
they
refill
the
node
disk
and
second
be
honest.
The
cube
cattle
command
is
not
always
the
way
you
want
to
consume
your
logs.
So
eventually
you
want
to
ship
them
to
your
preferred
location
or
service.
Of
course,
this
solution
raise
several
questions
to
access
the
host
file
system.
You
need
privileged
containers.
A
A
But
if
you
realize
you
want
to
batch
your
logs
for
performance
or
other
reasons,
you
will
need
an
aggregator
later,
you
may
want
to
send
a
portion
of
logs
to
different
endpoints
as
well.
At
the
end
of
the
day,
you
end
up
hundreds
of
rows
of
configuration
and
a
big
bunch
of
them
are
copy,
pasted,
not
mentioned
managing
secrets
and
other
variables.
A
This
was
our
first
approach
as
well,
and
we
wanted
to
simplify
the
solution.
The
first
version
of
the
logging
operator
was
born.
The
goal
is
to
automate
configuration
generation
from
fluent
bit
fluently
and
handle
them
as
kubernetes
resources.
With
a
really
simple
custom
resource.
You
could
save
a
lot
of
manual
deployment
and
configuration.
A
It
was
watching
for
resource
changes
and
pulled
credentials
from
kubernetes
secrets.
The
first
version
had
some
strong
resolutions,
it
utilized
fluent
tags
for
routing
and
it
allowed
using
kubernetes
pods
app
labels
only
as
tag
even
with
those
restrictions.
It's
already
lifted
tremendous
amount
of
work
from
the
operator's
shoulder.
It
was
a
big
success
for
us
and
as
more
and
more
community
gathered
around
it
helped
us
identify
the
pain
points
of
this
solution.
A
So,
after
validating
the
use
cases
we
set
back
to
the
planning
table
and
drafted
the
second
version
of
the
logging
operator.
We
had
a
couple
of
strong
requirements
in
mind:
namespace
isolation,
kubernetes
label,
selectors
the
same
way
as
kubernetes
service
or
cube
catalog
command
works
and
multiple
flow
and
multiple
output
support.
A
So,
roughly
a
year
later,
in
september
2019,
we
came
out
with
the
new
custom
resources,
a
new
core
model
to
be
more
kubernetes
like,
let's
see
how
this
works
in
the
logging
pipeline
note
that
the
following
principles
are
still
the
base
foundation
of
version
3
as
well.
We
collect
all
logs
with
fluent
bit
our
labor
router
plugin
identifies
log
flows
based
on
kubernetes
metadata.
A
A
A
You
can
define
local
output
for
outputs
in
a
namespace
or
global
outputs
for
cluster
output
resources.
It
is
possible
to
define
more
than
one
output
for
a
flow.
There
is
one
missing
piece
of
the
locking
pipeline.
We
need
to
define
the
outputs
as
well,
similarly
to
flow
and
class
airflow.
There
is
an
output
and
cluster
output.
A
When
you
configure
the
output
parameters,
you
can
inject
sensitive
data
from
kubernetes
secrets.
You
just
need
to
refer
to
them,
like
you
would
do
in
a
pod
definition.
After
you
apply
those
custom
resources.
There
is
one
more
step
the
operator
will
do.
It
will
create
a
configuration
checkpoint
for
the
system,
integrity,
only
open
a
successful
configuration
check.
Will
the
operator
swap
the
active
configuration
to
the
new
configuration.
A
This
check
runs
on
every
changes
on
the
custom
resources
we
arrived
at
the
last
station
of
the
logging
operator.
The
core
concepts
remain
the
same,
but
we
released
the
version
3.
On
march
2020,
we
added
several
commonly
driven
new
features
like
not
just
selecting
but
excluding
logs
from
a
flow
namespace
filter
for
cluster
flows
and
default
flow
for
the
leftover
data
that
haven't
matched
any
other
logging
flows.
A
A
A
A
A
After
that
one,
I
will
know
that
I
set
my
aws
credentials
in
my
environment
and
create
a
kubernetes
secret
from
them
with
that
secret.
It
will
create
my
output.
After
that
we
have
to
create
a
clusterful
resource.
The
cluster
flow
resource
needed
cluster
output.
We
chose
the
s3
output
that
we
just
created
and
if
we
accept
the
template,
it's
just
ready
to
go
and
if
you
apply
to
the
cluster
it
stores
the
locks.
A
A
A
To
achieve
this
first,
we
have
to
create
an
output.
For
this
case,
we
will
create
a
null
output.
This
is
a
special
case
because
it
doesn't
really
transfer
the
logs
anywhere
instead
just
drops
them,
but
it's
really
useful.
If
you
have
a
filter
that
produce
output
itself,
and
you
don't
really
need
your
logs,
it
also
means
that
we
don't
have
to
specify
any
parameter
for
this
output.
A
A
A
As
we
talked
about
it's
an
nginx
access,
log
and
fluentd
has
a
built-in
parser
for
nginx,
so
we
can
use
the
parse
type
nginx
for
this
kind
of
resource.
We
have
two
more
important
attributes
to
add.
The
remove
key
name
field
is
removing
the
original
message
from
the
log
and
the
reserve
data
to
means
that
we
want
to
keep
the
kubernetes
metadata
next
to
our
log.
A
A
A
A
A
So
we
have
our
metrics
in
place,
but
it's
more
important
that
we
have
these
metrics
in
parameters
as
well.
We
can
open
up
graphana,
go
to
the
explorer
menu
and
just
put
it
in
the
metric
name
and
hit
search
and
voila.
We
have
all
the
metrics.
We
can
use
the
read
function
and
check
how
the
log
generator
produces
the
random
status
codes.
A
We
have
time
for
one
more
example
as
well,
so
let's
create
a
namespace
called
infra
and
we
create
a
new
resource
called
host.
Taylor
hosteller
is
a
custom
resource
of
the
loggings
extension
another
great
tool
embedded
in
one
eye.
With
this
we
can
create
pods
that
can
tell
logs
from
the
host
file
system.
A
Let's
call
this
resource
a
system
decouplet
and
put
it
in
the
info.
Namespace
we
just
created
this
resource
will
be
a
system
detailer
that
tells
the
journal
log
of
systemd.
We
only
need
to
specify
a
couple
of
attributes
like
the
disabled
force,
the
max
entries,
let's
say,
100
the
name
of
this
tailor
kubelet.
This
will
be
the
container
name
and
a
systemd
filter
that
narrows
down
what
kind
of
logs
we
want
to
tear
from
the
systemd.
A
We
will
name
this
loki
because
it
will
be
a
lucky
output
and
we
will
put
this
into
the
infernal
space
for
every
output.
We
can
define
the
buffering
mechanism
as
a
lucky
output.
We
will
define
a
time.
Key
output
means
that
every
10
seconds
we
will
flash
the
messages
we
want
to
use
the
utc
time
keys
in
the
messages,
and
we
use
time
cuevate
parameter
2
seconds
to
wait
and
last
but
not
least,
we
set
the
configure
kubernetes
labels
through
that
will
attach
the
kubernetes
labels
to
lucky
metadata.
A
A
A
A
A
When
we
check
the
configuration
of
the
flow.
We
can
see
that,
yes,
we
are
triple
this
output,
but
we
don't
need
no
fancy
ui
to
check
the
problem.
You
can
use
the
command
line
tools
as
well.
The
cubecut
will
get
logging
or
command
we
print
out.
All
the
resources
and
the
resources
with
active
force
are
the
problematic
ones.
A
We
have
to
check
the
resource
status
field
for
it
we
can
write
a
simple
one:
either
cube
cattle
get
flow,
hyphen,
infra
q,
blood,
hyphen,
o
json
path
and
check
the
dot
status,
dot
problems
field
for
them
it
will
print
out
that
we
didn't
specify
the
proper
output
just
for
fun.
I
will
fix
this
issue
on
the
ui,
because
the
ui
and
the
cli
tool
works
from
the
same
resources
as
well.
A
So
last
but
not
least,
talk
about
the
future
of
the
logging
operator.
We
have
a
couple
of
things
in
our
mind:
advanced
routing,
based
on
richer
metadata
or
log
content,
apply
airbag
rules
to
different
lock
streams
and
provide
a
logging
api
to
have
a
unified
experience,
enhanced
with
authentication
and
authorization
to
be
easier
to
use.
A
Logging
operator,
as
a
component
in
a
bigger
system-
and
we
are
watching
several
interesting
projects
in
the
cloud
native
landscape,
open,
telemetry
collector,
which
speaks
the
fluent
protocol
and
tremor,
which
is
an
interesting
approach
for
collecting
and
transporting
logs
as
well.
Thank
you
for
listening
to
my
presentation.
Have
a
nice
day.